diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000000000000000000000000000000000000..cafba8b819efd1f756f3e71ae2e353f6b65cc6be --- /dev/null +++ b/.gitattributes @@ -0,0 +1,36 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +assets/training_progress.png filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..d31924d45ab7f56312ae7e094e373e44a1816d0f --- /dev/null +++ b/README.md @@ -0,0 +1,210 @@ +--- +license: apache-2.0 +language: +- en +datasets: +- openbmb/Ultra-FineWeb-L3 +- wikimedia/wikipedia +- HuggingFaceTB/finemath +- HuggingFaceTB/smollm-corpus +- OpenCoder-LLM/opc-annealing-corpus +tags: +- diffusion-language-model +- masked-diffusion +- parallel-decoding +- diloco +- text-generation +pipeline_tag: text-generation +--- + +

DiffuRefill-1B

+ +# DiffuRefill-1B + +**Status: pretraining, about half way.** Step ~100k of 200k, ~46B effective +training tokens, merge round 290. The weights here are a live pretraining +checkpoint: fluent English, facts still unreliable, not instruction-tuned. Use +`global_ema.pt`; it is the best checkpoint at every round. + +A 1.08B masked-diffusion language model decoded with **confidence-targeted +re-infilling**: draft the whole sequence in a handful of parallel denoising +passes, then spend a few more passes rewriting only the tokens the model is +least sure about. An autoregressive model needs one pass per token; this one +needs ~20 passes for a whole sequence, whatever its length. + +| | | +|---|---| +| parameters | 1.08B (embedding tied to the output head) | +| architecture | 18 blocks, dim 2048, 16 heads, SwiGLU 5632, RMSNorm, RoPE, **bidirectional** attention | +| context | 2048 tokens | +| tokenizer | `openbmb/MiniCPM4-0.5B` (73,440 tokens; vocab padded to 73,760 for MASK and PAD) | +| objective | absorbing masked diffusion, loss on masked positions only | +| training | DiLoCo across 3-5 single-GPU boxes, synchronised through this repository | +| progress | step ~100k / 200k, ~46B effective tokens | + +## Training progress + +![training progress](assets/training_progress.png) + +Tracked every round on fixed, deterministic batches, so the change between two +rounds carries no sampling noise: + +| measure | what it is | latest | +|---|---|---| +| held-out CE (EMA) | masked CE at t = 0.3 / 0.6 / 0.9 on a fixed Ultra-FineWeb slice no worker trains on | **2.939** (round 287) | +| held-out CE (global) | the same for the latest merged global | 2.957 | +| fact probe, mc | 1,050 PopQA facts written as plain sentences with the answer masked; the gold answer must beat three same-length distractors (chance 0.25) | **0.560** (was 0.540 before the data mix) | +| fact probe, answer NLL | per-token NLL of the masked gold answer | **6.08** (was 6.55) | +| fact probe, exact match (EMA) | the answer decoded token by token and compared with PopQA's accepted answers | 0.083 (was 0.075) | +| decoding accuracy | confidence decoding of 50% masked held-out text in 8 passes: token / adjacent-pair accuracy | 0.653 / 0.453 (round 275) | + +The guard-probe curve at the top is measured on an Ultra-FineWeb slice. When +the fact-dense sources were added at step ~96.8k, Ultra-FineWeb fell from 100% +to 56% of the documents, and that probe rose by design while the fact probe +improved quickly. From here the fact probe and the held-out CE are the numbers +to watch. No public benchmark scores are reported yet: at this stage they would +sit near chance and say little. + +## How it is trained + +**Fleet.** Ephemeral single-GPU boxes (RTX PRO 6000 Blackwell, 96 GB) with no +interconnect, joined DiLoCo-style through this repository. Each worker trains 150 +local AdamW steps, pushes `ckpt_wNN.pt`, and adopts the merged `global_b.pt`. + +- **Merging.** The merger averages the round's checkpoints (a plain mean, folded + into a running fp32 sum as they arrive) and waits at most 150 s past the first + arrival. +- **Phase.** Workers pick up a new global as soon as it appears and take its step + counter, so the fleet starts each interval together. +- **EMA.** An exponential average of the globals (β 0.9) is published as + `global_ema.pt`. + +**Objective.** +- `t ~ U(0.15, 1)` per row, with positions masked independently. +- Half of the rows keep a random-length clean prefix, which is the shape of + prompted generation. +- Attention stays inside each packed document. +- On a quarter of the rows, names and numbers are masked as whole spans + (salient span masking). Masked token by token, a two-token name leaks through + its visible half; masked whole, it has to be recalled. + +**Optimiser.** AdamW (0.9, 0.95), weight decay 0.1, clip 1.0, peak LR 1.5e-4, +cosine to 10% over 200k steps. Each worker step covers 64 × 2048 = 131,072 +tokens. The blocks are `torch.compile`d. + +**Data** (by document, from step ~96.8k): + +| source | share | +|---|---| +| [Ultra-FineWeb-L3](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3) Multi-Style + QA synthetic | 56% | +| Wikipedia (en) | 15% | +| FineMath 4+ | 8% | +| Cosmopedia v2 | 8% | +| OpenCoder annealing: algorithmic corpus + code snippets | 8% | +| Wikipedia leads restated as textbook / lecture / museum-guide prose (own corpus, filtered for copying and for numbers not in the source) | 5% | + +Before step ~96.8k the model saw Ultra-FineWeb only. + +### What the run taught + +Measured on this model or on its 37.9M-parameter stand, with fixed probes: + +| change | result | +|---|---| +| peak LR 3e-4 → 1.5e-4 | Local drift over one interval fell from 0.076 to 0.013 CE. The merged model improved from 4.336 to 4.288. | +| keeping local drift across a merge (`θ += global − pushed`) | Worse in 60 of 63 adoptions, +0.079 CE on average. The run uses plain replacement. | +| EMA of the globals | Beats the latest global every round. Seeding training from it gave the gain back. | +| outer Nesterov (μ 0.5) | Raised held-out CE for three rounds. Reverted. | +| Muon switched on mid-run | Loss rose from 3.16 to 3.32. It was a 2× win from scratch on the stand, but not as a switch on a running model. | +| span masking, PMI units, stratified t, complementary masks | Each lost to plain independent masking on the stand. Span masking was removed at step ~91.9k. | +| fact-dense data mix + salient span masking | Fact-probe answer NLL 6.55 → 6.08 within about four hours. | +| `torch.compile` per block | 17% faster steps, less memory. | +| FP8 matmuls (torchao, rowwise or tensorwise) | No speed-up on this card. Not used. | + +## The method + +``` +1. K denoising steps -> a full draft + a confidence value per position +2. repeat R times: + take the p% least confident generated positions + re-mask them + one forward pass -> new tokens, chosen with both sides in view + refresh their confidence +``` + +Cost is `K + R` forward passes. One refill pass fixes any number of positions at +once, because the model processes the whole sequence in parallel. That is what +makes targeted repair cheaper than more denoising. + +### Decoding configuration + +The draft and the refill deliberately use different sampling rules. The draft +decides what the text is about and needs freedom. The refill touches a fifth of +the tokens and needs discipline. Sharing one setting between them trades every +metric straight against another. + +| stage | settings | +|---|---| +| draft | `K=12`, temperature `2.2 -> 0.9` annealed, Gumbel noise `2.5` on the unmask order, no truncation | +| refill | `R=8`, `p=20%`, temperature `0.8`, `min_p=0.10` | +| both | neighbour ban: subtract `6.0` from the logit of the token already standing to the left or right; never commit two positions closer than 2 in one pass | + +- **Gumbel unmask order.** Strict top-k confidence reveals the easy positions + first, such as articles and punctuation. Content words are left for last, when + the context is already frozen. Annealed Gumbel noise on the log confidences + breaks that order. It was the largest single effect in the pilot. +- **Neighbour ban and commit gap.** Two adjacent positions sampled + independently both pick the locally likely token. That is how parallel + decoding produces repeats, and these two rules target exactly that. +- **Split-stage sampling.** A hot, free draft followed by a cool, truncated + refill. + +## Pilot results (TinyStories, 37.9M parameters) + +Two models of identical architecture were trained from scratch on the same +corpus: one autoregressive, one masked-diffusion. An **independent judge** +(`roneneldan/TinyStories-33M`, related to neither) scored the NLL of each +model's generated continuation. + +| | calls | NLL ↓ | doubled tokens ↓ | distinct-2 ↑ | +|---|---|---|---|---| +| real text | — | 1.15 | 1.38% | 0.478 | +| autoregressive | 128 | 3.08 | 1.64% | 0.466 | +| diffusion + refill, tuned | **20** | **2.87** | **0.30%** | 0.459 | + +The diffusion model matched autoregression in quality at a sixth of the passes. +It produced five times fewer doubled tokens than the autoregressive baseline, +below the rate in the real corpus. Diversity came out at parity, slightly under +autoregression. + +**What did not work in the pilot:** +- **Autoregressive repair of the draft.** It was worse than not repairing at all. + An AR model scoring position `i` cannot see `i+1`, so it duplicates the + neighbour it is not looking at. +- **Self-repair training on the model's own drafts.** It was harmful at 60–100% + corruption and neutral at 10–35%. + +## Honest limits + +- **Unfinished.** Half of the planned steps remain, and facts are unreliable. + Expect confident, fluent mistakes. +- **The pilot evidence is small.** 37.9M parameters on TinyStories does not show + that the decoding results transfer to 1B on natural text. +- **The judge is an AR model.** A low NLL under it rewards predictable text as + well as good text. +- **Diffusion costs more to train.** It learns only from masked positions, so it + reads more tokens than an AR model for the same quality. The saving is entirely + at inference. + +## Files + +| file | contents | +|---|---| +| `global_ema.pt` | EMA of the merged globals; the checkpoint to use | +| `global_b.pt` | the latest merged global | +| `ckpt_wNN.pt`, `log_wNN.txt` | per-worker checkpoints and logs, overwritten every interval | +| `config.json` | the exact architecture, objective, data mix and training setup | +| `pilots/` | held-out and fact-probe logs, stand pilot results | + +Checkpoints are `torch.save` dicts with a `model` state dict for the `Net` class +described in `config.json`. diff --git a/anneal/global_r760.pt b/anneal/global_r760.pt new file mode 100644 index 0000000000000000000000000000000000000000..58cc7ccf4e5b8e98b54ea998f148e2636957deaa --- /dev/null +++ b/anneal/global_r760.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ba83d63aa733cb18980b74df091af2236ca7cd1acda64ed97fe3d1b0af2a413d +size 2454120580 diff --git a/anneal/global_r780.pt b/anneal/global_r780.pt new file mode 100644 index 0000000000000000000000000000000000000000..c2adb4041f3b286d093e6fb2544a359165fa5bd4 --- /dev/null +++ b/anneal/global_r780.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e0d7b407dfe6ead33c994a750237a6620d18768fd096c0425f34d4a4e02c7767 +size 2454120580 diff --git a/anneal/global_r800.pt b/anneal/global_r800.pt new file mode 100644 index 0000000000000000000000000000000000000000..a613b914a1a3b3bf17f1ec7dda216cefca114e7e --- /dev/null +++ b/anneal/global_r800.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e8c9744f854c867a570f9882a395bc8efd2aacb6c738926cb6a15867cbd06c0a +size 2454120580 diff --git a/anneal/global_r820.pt b/anneal/global_r820.pt new file mode 100644 index 0000000000000000000000000000000000000000..d94aae6f70149122d3a50b5858b0b152ed7fdf12 --- /dev/null +++ b/anneal/global_r820.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b0c223a10325922c3b4279d378f9e2d75779ae35df9f1f48df8f8bf02590ffca +size 2454120580 diff --git a/anneal/global_r840.pt b/anneal/global_r840.pt new file mode 100644 index 0000000000000000000000000000000000000000..41b0df809f3ccc67162f2e2c41d9c06ba19336a6 --- /dev/null +++ b/anneal/global_r840.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7c30bf93abe59990fe462202f81315f0c1b0d7e54fec98034c1ca723590ce1fd +size 2454120580 diff --git a/anneal/global_r860.pt b/anneal/global_r860.pt new file mode 100644 index 0000000000000000000000000000000000000000..e937194d0bc4b2b92fb96352f28d99f093e5d14a --- /dev/null +++ b/anneal/global_r860.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9708b087eed172de8e62d0292cb6a13ebfdb7257646609647a3fffc847b8cc68 +size 2454120580 diff --git a/anneal/global_r880.pt b/anneal/global_r880.pt new file mode 100644 index 0000000000000000000000000000000000000000..c96d4cd85787d1ed174c61db0dfe31b7826ae53f --- /dev/null +++ b/anneal/global_r880.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:80f072ce83c9a404cf77f594d175c7d46985c5a4712d7d0baf62c309a2959e83 +size 2454120580 diff --git a/assets/avatar.svg b/assets/avatar.svg new file mode 100644 index 0000000000000000000000000000000000000000..4bcc83a717f424a3c8602399285e50a51b9ce8e6 --- /dev/null +++ b/assets/avatar.svg @@ -0,0 +1,53 @@ + + DiffuRefill-1B + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + DiffuRefill + 1B · MASKED DIFFUSION + diff --git a/assets/training_progress.png b/assets/training_progress.png new file mode 100644 index 0000000000000000000000000000000000000000..5bced14397a79ad8510623a10ad606733360ab28 --- /dev/null +++ b/assets/training_progress.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9326aeb6093da9acaa898722d0f9feef1f1764823745a1d809635b055a3c0dec +size 152768 diff --git a/ckpt_w0.pt b/ckpt_w0.pt new file mode 100644 index 0000000000000000000000000000000000000000..962f7c2b1477c3ebaadea93eb7b8145e1bb8a1d2 --- /dev/null +++ b/ckpt_w0.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bc48c834e124a9bf4cc3bec73cb40a3c8cd85b52f09d128a4c563eac0579b468 +size 2152005695 diff --git a/ckpt_w1.pt b/ckpt_w1.pt new file mode 100644 index 0000000000000000000000000000000000000000..c4d4c15c1823c4d093abea9ea4c71f10693ee400 --- /dev/null +++ b/ckpt_w1.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c809ec752ae23731aa0609bbab96b6a33855607ce7e294d7faed40c38f2d02e5 +size 2152005695 diff --git a/ckpt_w10.pt b/ckpt_w10.pt new file mode 100644 index 0000000000000000000000000000000000000000..75149843053284facac2ebdddb149dd9a66152b7 --- /dev/null +++ b/ckpt_w10.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e59d70894c26651aba1ffe32ee12ec0aac14432c2d1c215a1c56475f20754c87 +size 2152006469 diff --git a/ckpt_w100.pt b/ckpt_w100.pt new file mode 100644 index 0000000000000000000000000000000000000000..a3b22fa37e9abae81451c9173dc2c3c8e9761019 --- /dev/null +++ b/ckpt_w100.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7d15f2c6c7893a4e6335dfc59f82baefb2a3995e09f255ac339c908c512119b4 +size 2152006603 diff --git a/ckpt_w101.pt b/ckpt_w101.pt new file mode 100644 index 0000000000000000000000000000000000000000..a7f9f53a2200259c7958ff0e73411e7a5ea7b023 --- /dev/null +++ b/ckpt_w101.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2ccdb58193a6c600c2c93a28119b7909ebcc276aa6ee319537a81247e7af5174 +size 2152006603 diff --git a/ckpt_w102.pt b/ckpt_w102.pt new file mode 100644 index 0000000000000000000000000000000000000000..45a26a5b02133ae9d91316f43f1a045578956c20 --- /dev/null +++ b/ckpt_w102.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8b7550fa3fc11cdfda6af123480e4d17210789d495e0143d648a56816e07b442 +size 2152006603 diff --git a/ckpt_w103.pt b/ckpt_w103.pt new file mode 100644 index 0000000000000000000000000000000000000000..39a7f7a1b1edc61cf6ae4a5a13ff85948f4038d0 --- /dev/null +++ b/ckpt_w103.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a66cdfc8798387d530cb3d67a5e2460333e65b3f72afd999e7bb42bf1af85bb1 +size 2152006603 diff --git a/ckpt_w106.pt b/ckpt_w106.pt new file mode 100644 index 0000000000000000000000000000000000000000..d5293ee4415568a3d019af42ec3d4da70a8dfee7 --- /dev/null +++ b/ckpt_w106.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9bab673c873d29d747d950cba9c1d0c2edf4020086fb19e3005e733a88147b2e +size 2152006603 diff --git a/ckpt_w107.pt b/ckpt_w107.pt new file mode 100644 index 0000000000000000000000000000000000000000..7a83ba4c24572064d86899dcc4979bec863c5a81 --- /dev/null +++ b/ckpt_w107.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f9c3a31047305c6c986bcf34b26dd05b9ab7f2a70add2a8fd3a64396d5cc6ac3 +size 2152006603 diff --git a/ckpt_w109.pt b/ckpt_w109.pt new file mode 100644 index 0000000000000000000000000000000000000000..581f9d56294df58b1ac7b49a33eb2f21d8bb42ac --- /dev/null +++ b/ckpt_w109.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1b7062b2d1ba0ce7d0364f77c4c411d217dd243784ac52d05ce81e533ac48d4b +size 2152006603 diff --git a/ckpt_w11.pt b/ckpt_w11.pt new file mode 100644 index 0000000000000000000000000000000000000000..4ce282f4e98eecd620c04ee53db39640ea8bd530 --- /dev/null +++ b/ckpt_w11.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9252ba8b8afd5685947825afb17da1fbcaad49b32b8e3403a663a8b2cf81b914 +size 2152006469 diff --git a/ckpt_w110.pt b/ckpt_w110.pt new file mode 100644 index 0000000000000000000000000000000000000000..03fd5f4f1a5c87a0e23013175f185a4f3378064a --- /dev/null +++ b/ckpt_w110.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a7e9f3ee84d6bc1442d9edca928d00d01f485b4af72a25fa0e68a39c7d13696d +size 2152006603 diff --git a/ckpt_w111.pt b/ckpt_w111.pt new file mode 100644 index 0000000000000000000000000000000000000000..931ffc1637a8a30da38734947220353a3858499f --- /dev/null +++ b/ckpt_w111.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f26570a83fe5cdf7a2bf40894e87a0807ffb6342974565210d2d666384ad9586 +size 2152006603 diff --git a/ckpt_w112.pt b/ckpt_w112.pt new file mode 100644 index 0000000000000000000000000000000000000000..f88b07cde5e59904937ea75c27b045bcaf33c7a6 --- /dev/null +++ b/ckpt_w112.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:61582fac4259bdaf5e9ebd6974a32640d056f67efa96c32cd0cd8dcfc71e6993 +size 2152006603 diff --git a/ckpt_w113.pt b/ckpt_w113.pt new file mode 100644 index 0000000000000000000000000000000000000000..7085017128e72957303f7594a1df02ba515e78af --- /dev/null +++ b/ckpt_w113.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a1b12731d764da327e192f1f69d585a2d1e4b69b02bb0d59bb6a41cbaa5654e5 +size 2152006603 diff --git a/ckpt_w115.pt b/ckpt_w115.pt new file mode 100644 index 0000000000000000000000000000000000000000..4c5a5051ed9a5a1542e3ce9530e040ba26b9323f --- /dev/null +++ b/ckpt_w115.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:765c68e8bab49178a57d19075dd891493d8c3a7c27d4594c3cd8a3ec8f63a9d8 +size 2152006603 diff --git a/ckpt_w116.pt b/ckpt_w116.pt new file mode 100644 index 0000000000000000000000000000000000000000..095e90a828811f1abcc8457c9ae8e8edfa2eadbd --- /dev/null +++ b/ckpt_w116.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9beacee12952287e8a7b0227945e490cb3e847ca6f686c2d070c39674bc782ac +size 2152006603 diff --git a/ckpt_w117.pt b/ckpt_w117.pt new file mode 100644 index 0000000000000000000000000000000000000000..e77709524b6cc9a3558e66a7d0e9a9b473f903f1 --- /dev/null +++ b/ckpt_w117.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a6e2929f9193258d937068138920023d8c8a208053e2c0b14825e0c65ae7adf0 +size 2152006603 diff --git a/ckpt_w118.pt b/ckpt_w118.pt new file mode 100644 index 0000000000000000000000000000000000000000..aa8298da333fcfef0caea488f7e89cce42de87c9 --- /dev/null +++ b/ckpt_w118.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:82f4bfc82f423d8cad5b74c1b72efd8cdb16b10f6f3f791c0e8700228deefd95 +size 2152006603 diff --git a/ckpt_w119.pt b/ckpt_w119.pt new file mode 100644 index 0000000000000000000000000000000000000000..f418f2045c8369db9e6000849d5dde89c3cedf65 --- /dev/null +++ b/ckpt_w119.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5fecfd8bdc12f9f814eeac07bc4cf8eac9f04fae3f6b3885e0c850aba7976163 +size 2152006603 diff --git a/ckpt_w120.pt b/ckpt_w120.pt new file mode 100644 index 0000000000000000000000000000000000000000..ccc478efccf7cd0426b206a1129b3792069cec3a --- /dev/null +++ b/ckpt_w120.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bbcc39aecbbb6672b7e8103786d61e3cdcaec71db37e1074d44b6f32ec70898d +size 2152006603 diff --git a/ckpt_w121.pt b/ckpt_w121.pt new file mode 100644 index 0000000000000000000000000000000000000000..185d8d1f035c4fff4a5938cce0376b10763b4abf --- /dev/null +++ b/ckpt_w121.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5e87ab48d00dedc65429fc69295da855558c666035df361930c1cdaa68f4f9be +size 2152006603 diff --git a/ckpt_w122.pt b/ckpt_w122.pt new file mode 100644 index 0000000000000000000000000000000000000000..c8f9ef5c224e71e99c4a4a48ca6e12bbd458d831 --- /dev/null +++ b/ckpt_w122.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d44f7a9478248eb8fff728706b3715935473808a97de2bee23486076ff1b56b0 +size 2152006603 diff --git a/ckpt_w123.pt b/ckpt_w123.pt new file mode 100644 index 0000000000000000000000000000000000000000..75263d816c7932cb60537db5422e749ff411a797 --- /dev/null +++ b/ckpt_w123.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:eeb0f2d93b6016f9ebbce149b44f394a16c334c2611d915a8186d04a4841b3bc +size 2152006603 diff --git a/ckpt_w124.pt b/ckpt_w124.pt new file mode 100644 index 0000000000000000000000000000000000000000..9a478229867d06fb1a4e69beab9d5e47d4d1ba3e --- /dev/null +++ b/ckpt_w124.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:de7d8fdd1813ddb3b2cb1b13d1db17a575b9233cd05ae5741461bb010d194b56 +size 2152006603 diff --git a/ckpt_w125.pt b/ckpt_w125.pt new file mode 100644 index 0000000000000000000000000000000000000000..43f9ce99bc8306f8b8ce4aba069f4a08c4be269c --- /dev/null +++ b/ckpt_w125.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3a58b3f3afa0a833aaf7ba054d207d7ddcbb2ef1b9409776b9590d6082f70d7d +size 2152006603 diff --git a/ckpt_w127.pt b/ckpt_w127.pt new file mode 100644 index 0000000000000000000000000000000000000000..c5ff99da3252d7e7754b728ef2e74e072bec6fe8 --- /dev/null +++ b/ckpt_w127.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fd8ef09d4ef09f6237d5150d0d67dc959cb6c8de64aa3a71d552d186a5a03ef0 +size 2152006603 diff --git a/ckpt_w128.pt b/ckpt_w128.pt new file mode 100644 index 0000000000000000000000000000000000000000..85f264917b766257ac71bccf33f9157be0098b30 --- /dev/null +++ b/ckpt_w128.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a5cec99950bdba3ea7baaf79df9e4ba252542842952b6462411c24556333101a +size 2152006603 diff --git a/ckpt_w129.pt b/ckpt_w129.pt new file mode 100644 index 0000000000000000000000000000000000000000..2e56b9e49e0aa3cc8665d3134f33b99d972e02f6 --- /dev/null +++ b/ckpt_w129.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:444508900c9ab5c7eb77fe055db525b63f8c40e4813b4622facf7ec9124b8031 +size 2152006603 diff --git a/ckpt_w13.pt b/ckpt_w13.pt new file mode 100644 index 0000000000000000000000000000000000000000..361faf20650eb2afb25e4cdb8bbb8eae0dc3fa17 --- /dev/null +++ b/ckpt_w13.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cb2c39cb7f4652f5c3b6d3ad5dfa639a297b98f742196bb9b94044eb3045118e +size 2152006469 diff --git a/ckpt_w130.pt b/ckpt_w130.pt new file mode 100644 index 0000000000000000000000000000000000000000..a271fcea88f23469f59893bad59ff095e856bd4e --- /dev/null +++ b/ckpt_w130.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9a34bf6ef0e9f64dbc44142fe396ed0ff33fbe7c28e55ace962d7215de9fa281 +size 2152006603 diff --git a/ckpt_w131.pt b/ckpt_w131.pt new file mode 100644 index 0000000000000000000000000000000000000000..919dde77ada03585d0642d7660f8a39daf2afb24 --- /dev/null +++ b/ckpt_w131.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:25148fc5d2d42fce0a0a7af51787b6bbff017b0c7cf57b1a7e06f74cc126ede8 +size 2152006603 diff --git a/ckpt_w132.pt b/ckpt_w132.pt new file mode 100644 index 0000000000000000000000000000000000000000..9311ac63340df095c08d8dc075ecab00255af574 --- /dev/null +++ b/ckpt_w132.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d31c9a0acf571e41920ad3a1a3869713c999c212ca46e06b70832e2dddba96f4 +size 2152006603 diff --git a/ckpt_w133.pt b/ckpt_w133.pt new file mode 100644 index 0000000000000000000000000000000000000000..c8292efc8a5944918210af853fb461e7a9c6244c --- /dev/null +++ b/ckpt_w133.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:adca03cb6b01f40b38da58062061b3ac07299f95f644ac4c244533c34f3b09e1 +size 2152006603 diff --git a/ckpt_w134.pt b/ckpt_w134.pt new file mode 100644 index 0000000000000000000000000000000000000000..ae4da389f952cbe932f4f1d87f12e56230f2fc2c --- /dev/null +++ b/ckpt_w134.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1e97da28cf795c3781b9c2d3d402a8775327ca7115d1df629324b3a795de2cef +size 2152006603 diff --git a/ckpt_w135.pt b/ckpt_w135.pt new file mode 100644 index 0000000000000000000000000000000000000000..a8af6f35e155664f080df22c10863f3f79e06440 --- /dev/null +++ b/ckpt_w135.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e60974c45e2f5bed823a7bf66e20b7718ed7799102c920d48a63c938d4ee8962 +size 2152006603 diff --git a/ckpt_w136.pt b/ckpt_w136.pt new file mode 100644 index 0000000000000000000000000000000000000000..3f84e09ea4a7dfebcf50231bcbb38dd4ed7321e7 --- /dev/null +++ b/ckpt_w136.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ef12eb1910308641801d7cbfd0e80ee944e34f5deb13db5f287f2383975d5e18 +size 2152006603 diff --git a/ckpt_w137.pt b/ckpt_w137.pt new file mode 100644 index 0000000000000000000000000000000000000000..cbf64c8e39b6d641f7d2238eb94f505ab751c578 --- /dev/null +++ b/ckpt_w137.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6bca39e573dde50987708eac46f48dfeff809a05ff963bb3b1388ba01af73970 +size 2152006603 diff --git a/ckpt_w138.pt b/ckpt_w138.pt new file mode 100644 index 0000000000000000000000000000000000000000..c635085178555184bb113b94e0259cfcb8520a53 --- /dev/null +++ b/ckpt_w138.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3870f437161cfa6dd79d0eaf56a4e0c8bdbe63e553f91fe9967aaf6ae7134d1b +size 2152006603 diff --git a/ckpt_w139.pt b/ckpt_w139.pt new file mode 100644 index 0000000000000000000000000000000000000000..bbb061ded1ea80de67cc1af572a102431e7f647d --- /dev/null +++ b/ckpt_w139.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f580fb68977a0e8d02b2e182fe3742acb1164ea42d209ff61ab9f4e8ee12489d +size 2152006603 diff --git a/ckpt_w140.pt b/ckpt_w140.pt new file mode 100644 index 0000000000000000000000000000000000000000..882fdb2c5696dd168b6fa0babcf947e19a8f0205 --- /dev/null +++ b/ckpt_w140.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:de6203b609d78b91cf0865281f2fdfc7937ed24b7f3851aa7b60938629d15674 +size 2152006603 diff --git a/ckpt_w141.pt b/ckpt_w141.pt new file mode 100644 index 0000000000000000000000000000000000000000..8069824a595ffa9c894b9f14f3d5b3804fd5bfbb --- /dev/null +++ b/ckpt_w141.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:87c0623b760b7168d80a609bfb1b69339929c9778e77fa344a6645aed5f42ab5 +size 2152006603 diff --git a/ckpt_w143.pt b/ckpt_w143.pt new file mode 100644 index 0000000000000000000000000000000000000000..7d8e7ed42e0e48bf56077309a83bae48f9d7323f --- /dev/null +++ b/ckpt_w143.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0f9ab32337946540df1a01ef871fc5910e41534a55d6d5e043ac688d54c34ff1 +size 2152006603 diff --git a/ckpt_w146.pt b/ckpt_w146.pt new file mode 100644 index 0000000000000000000000000000000000000000..b3a9a995dce74665aec0ecf7fff857a92e892a0f --- /dev/null +++ b/ckpt_w146.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:85117a1e971bfb916108507e7894a49739a209037f11114d818ef8f93f43682b +size 2152006603 diff --git a/ckpt_w147.pt b/ckpt_w147.pt new file mode 100644 index 0000000000000000000000000000000000000000..151ab86d67ad04a916c2218735ced411a3d2dc2f --- /dev/null +++ b/ckpt_w147.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d57def9826ae4fc029e54e6da8d35440b97385780316002893ce43f4e5a2f6b5 +size 2152006603 diff --git a/ckpt_w148.pt b/ckpt_w148.pt new file mode 100644 index 0000000000000000000000000000000000000000..b8701bcba13606c4d7ff468cf4a7389e37b8bd19 --- /dev/null +++ b/ckpt_w148.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:810ffcf43800211e7a5604d6f9025b45b79aca15fd9b8f1d908613337364834f +size 2152006603 diff --git a/ckpt_w149.pt b/ckpt_w149.pt new file mode 100644 index 0000000000000000000000000000000000000000..9019aa8e64311a2b077ae6dd28020fd9c8be78e8 --- /dev/null +++ b/ckpt_w149.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6fbeca6f0f3031feb80a77bcf4c77cb1f07a78fc69b5192cbed8864ec27fb68e +size 2152006603 diff --git a/ckpt_w150.pt b/ckpt_w150.pt new file mode 100644 index 0000000000000000000000000000000000000000..e703128c455f18f85f0c743d17ffbee4d7cb0e6d --- /dev/null +++ b/ckpt_w150.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6462b4ff19c4d867049e5f4ee874719a2a4d3004242e68a35f7815087cbbefed +size 2152006603 diff --git a/ckpt_w151.pt b/ckpt_w151.pt new file mode 100644 index 0000000000000000000000000000000000000000..ce60f5e91298df43b29c0752305a8560b585f090 --- /dev/null +++ b/ckpt_w151.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e1ab9f7ffbbe87b7c870727ff7a038bdfb33d766e55063e485f63dd17d82140d +size 2152006603 diff --git a/ckpt_w152.pt b/ckpt_w152.pt new file mode 100644 index 0000000000000000000000000000000000000000..e24cd3391a523c89b1b1a38765dcf14fe8d85f69 --- /dev/null +++ b/ckpt_w152.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f0177d6d298871c9b4d423d95719218f307527e835fe4ac227e24487aead5047 +size 2152006603 diff --git a/ckpt_w153.pt b/ckpt_w153.pt new file mode 100644 index 0000000000000000000000000000000000000000..84713c75704476260ac608b1a6c0d4fb54bc33f2 --- /dev/null +++ b/ckpt_w153.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c1c34224d54301fb85d7dfb1d5176719f4975738575c1d2b306dab411801bbb0 +size 2152006603 diff --git a/ckpt_w154.pt b/ckpt_w154.pt new file mode 100644 index 0000000000000000000000000000000000000000..75dd7385f5204bb92c2d1cb1b1c2ffa7eef4afac --- /dev/null +++ b/ckpt_w154.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a401df072f8cca148c2e66c6f6c3643d08bc9e8d42c5e12ca533c71befe14cd3 +size 2152006603 diff --git a/ckpt_w155.pt b/ckpt_w155.pt new file mode 100644 index 0000000000000000000000000000000000000000..a73f34afcb0dae47cd5dbdaf83eeb0d083e36ae8 --- /dev/null +++ b/ckpt_w155.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:32ec0df1db97d4bfae5f4272bb09d89adba9abac4371e6676de4cbd9c032fc9d +size 2152006603 diff --git a/ckpt_w156.pt b/ckpt_w156.pt new file mode 100644 index 0000000000000000000000000000000000000000..d613662af3e916f5c845695552c026385381e41e --- /dev/null +++ b/ckpt_w156.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e1ed512bda74e3f98aa58f9a5cb99923bea30093a7ccca4c1ac3ef0f52890e25 +size 2152006603 diff --git a/ckpt_w157.pt b/ckpt_w157.pt new file mode 100644 index 0000000000000000000000000000000000000000..984f11982c8c48bf7a30129d0a020f6a6c3dc4e3 --- /dev/null +++ b/ckpt_w157.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:dfecfdfae315d4602cc8ba7802f0455675a55e80d5b7049560d8f233b8b8b124 +size 2152006603 diff --git a/ckpt_w158.pt b/ckpt_w158.pt new file mode 100644 index 0000000000000000000000000000000000000000..42d4f41505bf53339ea7f5c22145b8c5198f7bb9 --- /dev/null +++ b/ckpt_w158.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:18f16eade12969062d28377c145d76bcfe5fd152d14a7c5977aa3711cc3c7b05 +size 2152006603 diff --git a/ckpt_w159.pt b/ckpt_w159.pt new file mode 100644 index 0000000000000000000000000000000000000000..1556be881ab8a2af82cca4d05a79ab8409db7d20 --- /dev/null +++ b/ckpt_w159.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c349d2e7685618521eb5603d5bc7b8f7c9d1491f34cb8ef3d5cc6de9856c1b78 +size 2152006603 diff --git a/ckpt_w160.pt b/ckpt_w160.pt new file mode 100644 index 0000000000000000000000000000000000000000..fbde276a2853c7d198f22f0952c3636e92bc158a --- /dev/null +++ b/ckpt_w160.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:83abff9b2b9e822280345365ba3a613bf7875c4a13c4603a6ca58fe7e1993046 +size 2152006603 diff --git a/ckpt_w161.pt b/ckpt_w161.pt new file mode 100644 index 0000000000000000000000000000000000000000..b71154f59ae14282dce98b7a850050befc712cc6 --- /dev/null +++ b/ckpt_w161.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c76981e1c85f3467b568ce49705b91c57da2f740305ad52baec11fcb92cee43a +size 2152006603 diff --git a/ckpt_w162.pt b/ckpt_w162.pt new file mode 100644 index 0000000000000000000000000000000000000000..282a43ef791814a91c2c83104cde75bd51cb70b4 --- /dev/null +++ b/ckpt_w162.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2bc24fa5f83618c8a79c376f03342268aee44b76d889e1abb91554eacae1bce7 +size 2152006603 diff --git a/ckpt_w163.pt b/ckpt_w163.pt new file mode 100644 index 0000000000000000000000000000000000000000..b6601fabd42d38870c8a3dbb679ab22f7b87e264 --- /dev/null +++ b/ckpt_w163.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bf1d4d1333fa92f7dcddeeed70037c55d9ca8cf047a439a95815ed710c5c5565 +size 2152006603 diff --git a/ckpt_w164.pt b/ckpt_w164.pt new file mode 100644 index 0000000000000000000000000000000000000000..55485a3ee59816a0e57b83b84e5c80838e792d6c --- /dev/null +++ b/ckpt_w164.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ca5bf725e916ccd3bad7b297accffe2f3cc313cf00086ae090c865f2b7b27bc4 +size 2152006603 diff --git a/ckpt_w165.pt b/ckpt_w165.pt new file mode 100644 index 0000000000000000000000000000000000000000..a27d78d2b2a61f2a8fb950385683596014419fad --- /dev/null +++ b/ckpt_w165.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c1677b8a78cf36fb0a65ac85c1b577d1d4ead3addb228a167bd2113d279ea67a +size 2152006603 diff --git a/ckpt_w166.pt b/ckpt_w166.pt new file mode 100644 index 0000000000000000000000000000000000000000..0da408da50495c119a9593f5a7a0d63a0b756c76 --- /dev/null +++ b/ckpt_w166.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:56958757c73bded362e8e221ad78dafba4b41bbffc29f159c1c02772b9f18d30 +size 2152006603 diff --git a/ckpt_w167.pt b/ckpt_w167.pt new file mode 100644 index 0000000000000000000000000000000000000000..851b317b5adf8c729a61c898e850e5baca8988f6 --- /dev/null +++ b/ckpt_w167.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:dec0ff0296c6f4d286fea3444c12b4818cd27715bcfcc444fe2485076c140484 +size 2152006603 diff --git a/ckpt_w168.pt b/ckpt_w168.pt new file mode 100644 index 0000000000000000000000000000000000000000..96f8d066d4ab6a3f7443f98b64bb7f8067232046 --- /dev/null +++ b/ckpt_w168.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:dbdd61b2f615f75b3abac874144b4cdd1f2489205690eeba9c32c8941bf023f8 +size 2152006603 diff --git a/ckpt_w169.pt b/ckpt_w169.pt new file mode 100644 index 0000000000000000000000000000000000000000..e293cea4cd8c1f9dc9650f7ca63bed0162caebdd --- /dev/null +++ b/ckpt_w169.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:44c0dc6a318b0dcd8a4e249d488f40426ec1ec853d7bb4b33aec195c6d0ec1e7 +size 2152006603 diff --git a/ckpt_w170.pt b/ckpt_w170.pt new file mode 100644 index 0000000000000000000000000000000000000000..284307a37227c93c9794f1ef1dfb6ba5e555c542 --- /dev/null +++ b/ckpt_w170.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0dd23c29cc94c0e4d91f0aff81c48ce1e60ac43ae1a4339e47092edb8caad21b +size 2152006603 diff --git a/ckpt_w171.pt b/ckpt_w171.pt new file mode 100644 index 0000000000000000000000000000000000000000..fe50a8666688adddccbe6f904e552fea9fde56c0 --- /dev/null +++ b/ckpt_w171.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f2f90504f409191975b76ae98243e91bed3268530a657fc0861d79e99918fc5e +size 2152006603 diff --git a/ckpt_w172.pt b/ckpt_w172.pt new file mode 100644 index 0000000000000000000000000000000000000000..fa085350263c15949f38f8e218a710feacbafd1f --- /dev/null +++ b/ckpt_w172.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:799f8e86d334cbd98688844dbc7fcd4a4222350f0c6ef793b887860ee6b2891b +size 2152006603 diff --git a/ckpt_w173.pt b/ckpt_w173.pt new file mode 100644 index 0000000000000000000000000000000000000000..1930277bf7aa6fe00bbd55142f539ce68b216d8c --- /dev/null +++ b/ckpt_w173.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3b2f80bf10ef289f97e1ff6da4c4a5877f2e23beadd659a8b971d24ebc821d42 +size 2152006603 diff --git a/ckpt_w174.pt b/ckpt_w174.pt new file mode 100644 index 0000000000000000000000000000000000000000..354da257ad3267f7b8dc1a367ce7c97cc6fd4fd5 --- /dev/null +++ b/ckpt_w174.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:af263bf85b47c2df5e367dbfcf0aa45fd2ab52fd866b96c240e7f362d485aa6a +size 2152006603 diff --git a/ckpt_w175.pt b/ckpt_w175.pt new file mode 100644 index 0000000000000000000000000000000000000000..612fba78dd8774c5aca336893bdef082d297fb6f --- /dev/null +++ b/ckpt_w175.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7db57a355359d421224c1ab9c80c6bf78d4eade36643ac1a99cfcc212e978e43 +size 2152006603 diff --git a/ckpt_w176.pt b/ckpt_w176.pt new file mode 100644 index 0000000000000000000000000000000000000000..f5385af28a1fe8d93f06b543b1855239133d0913 --- /dev/null +++ b/ckpt_w176.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:95ab09f7fc899074e114a47b84a7800b0ce145b1bab97d1d64c2fed7387e1a8e +size 2152006603 diff --git a/ckpt_w177.pt b/ckpt_w177.pt new file mode 100644 index 0000000000000000000000000000000000000000..a858d8a21da5a2beb0697e1ff5a52dd6eb7ab0b0 --- /dev/null +++ b/ckpt_w177.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9aec58c1600079a48983e821ea99dd064e67b0f38c771d9d3d8fc79c75b652b3 +size 2152006603 diff --git a/ckpt_w178.pt b/ckpt_w178.pt new file mode 100644 index 0000000000000000000000000000000000000000..8125d81f6dfcf08144c2b62ba405fc05772bdc3b --- /dev/null +++ b/ckpt_w178.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:790c4d4dcba2f7ad736554cf3380c8ae36bd1f7544fbb4f5b126b661555b7128 +size 2152006603 diff --git a/ckpt_w179.pt b/ckpt_w179.pt new file mode 100644 index 0000000000000000000000000000000000000000..a29b5e2b13c9a1205166eca732375cd30c3a5b4d --- /dev/null +++ b/ckpt_w179.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b759c0b4fb886486a33f8436864668ac61451f66b6d28c1c91c5103f48dfc680 +size 2152006603 diff --git a/ckpt_w180.pt b/ckpt_w180.pt new file mode 100644 index 0000000000000000000000000000000000000000..8fd1f7a608a62012a88ca44c14d2705786a1637a --- /dev/null +++ b/ckpt_w180.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7f37c1ed5f8eeb437ebeee2a61dd454ed5de9e3d80c3a9826428709d4049bf46 +size 2152006603 diff --git a/ckpt_w181.pt b/ckpt_w181.pt new file mode 100644 index 0000000000000000000000000000000000000000..d62d27a2c6cd7a5a9e6ecaf90a513e2101ff79e1 --- /dev/null +++ b/ckpt_w181.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e5cb678418c341d6894399b80a93ee253054be57a41d6e5d5527a1cabb6299ca +size 2152006603 diff --git a/ckpt_w182.pt b/ckpt_w182.pt new file mode 100644 index 0000000000000000000000000000000000000000..0f985e623c4445c4802ec423bb01f8615e6c3454 --- /dev/null +++ b/ckpt_w182.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:475decc5d12a14f293bfe88d5e98d253da32233407c5eeeff70cda31a930be64 +size 2152006603 diff --git a/ckpt_w183.pt b/ckpt_w183.pt new file mode 100644 index 0000000000000000000000000000000000000000..9e9c00694cc422556b0f5546aa2c5a14a50303fc --- /dev/null +++ b/ckpt_w183.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:40a18ce6ba08d5ea0e20e77ad95af88202b08aa829d7f848f6d01fee89410d0d +size 2152006603 diff --git a/ckpt_w184.pt b/ckpt_w184.pt new file mode 100644 index 0000000000000000000000000000000000000000..a2559c30d6ac033b4ae50faf26c5c5ff958a396f --- /dev/null +++ b/ckpt_w184.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bdb5a1c8f5b64af5e8027fc41afba12a94b1231e4b94de237d80a5381747bebb +size 2152006603 diff --git a/ckpt_w185.pt b/ckpt_w185.pt new file mode 100644 index 0000000000000000000000000000000000000000..48e386f545e8a3cf339622c1460958a8282635d9 --- /dev/null +++ b/ckpt_w185.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:75729806d26dba6a49ef9df7994e31a157796039e0cae7bb43d969299493af1c +size 2152006603 diff --git a/ckpt_w186.pt b/ckpt_w186.pt new file mode 100644 index 0000000000000000000000000000000000000000..c444bead73097d4ebbda2d32f3f9b33dbeed7def --- /dev/null +++ b/ckpt_w186.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ec8666fc9cc01aa51fa95b3ecbbadfe418688fb39aa5da03c878146c13588897 +size 2152006603 diff --git a/ckpt_w187.pt b/ckpt_w187.pt new file mode 100644 index 0000000000000000000000000000000000000000..122eeb7b77c458d249e477db1d3cab59cbc94217 --- /dev/null +++ b/ckpt_w187.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2cf9dcf2a8f68412f44efe59127d336907a4a216d7f53fcb5ae0730c980ac516 +size 2152006603 diff --git a/ckpt_w188.pt b/ckpt_w188.pt new file mode 100644 index 0000000000000000000000000000000000000000..878db99f0af2ca11498c64009e983d7b21839789 --- /dev/null +++ b/ckpt_w188.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fec7ac3a714ee141f0719c4b17679214f4737409f2627ad3944714fdbf51c3e0 +size 2152006603 diff --git a/ckpt_w189.pt b/ckpt_w189.pt new file mode 100644 index 0000000000000000000000000000000000000000..e84de136ab8045ce1f7d90cc6fd91285f48c8f1f --- /dev/null +++ b/ckpt_w189.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d4f4e0ecbe85b1c5be2ded76421471b13e45298076d67b7fa65ce322d4a80df9 +size 2152006603 diff --git a/ckpt_w190.pt b/ckpt_w190.pt new file mode 100644 index 0000000000000000000000000000000000000000..47aea417b778a71a2d68ec2c7e20ecb9b4b74f3a --- /dev/null +++ b/ckpt_w190.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1e97a9f63cc7e240f23a6c576a8681f357fb6ab25d91822efbd2c06ad0faeed6 +size 2152006603 diff --git a/ckpt_w191.pt b/ckpt_w191.pt new file mode 100644 index 0000000000000000000000000000000000000000..fe1103e22b62372279e2ff6e372b15aeab6a6a1a --- /dev/null +++ b/ckpt_w191.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:83a5fbdcce59787755fc6dd6808405e4cdf5829e46ba3e4e915510c37601ac32 +size 2152006603 diff --git a/ckpt_w192.pt b/ckpt_w192.pt new file mode 100644 index 0000000000000000000000000000000000000000..816c3a4c55b3d086750d6e4c56198a7a1fcaf0cb --- /dev/null +++ b/ckpt_w192.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:132df7db590d08c4f27bba9eaa424a76e543e2269fafcd2aa1727fc6969d3628 +size 2152006603 diff --git a/ckpt_w193.pt b/ckpt_w193.pt new file mode 100644 index 0000000000000000000000000000000000000000..cfd0991f6e2c53c89d57e7e332e42cc14d0c9445 --- /dev/null +++ b/ckpt_w193.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:823613d2457b3b1ff96f65f67fe270def70ab346c741c1d49b381e5ae8ff8280 +size 2152006603 diff --git a/ckpt_w194.pt b/ckpt_w194.pt new file mode 100644 index 0000000000000000000000000000000000000000..b4a2b8a3408f133f500643174df610a1016a9a97 --- /dev/null +++ b/ckpt_w194.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:874ff9996ff7460dc4622d12a3000ab5eb8121f3d7907129d7a812c53935368e +size 2152006603 diff --git a/ckpt_w195.pt b/ckpt_w195.pt new file mode 100644 index 0000000000000000000000000000000000000000..36f5f26b59a7156b06ceb76ebabf41cbc3e35335 --- /dev/null +++ b/ckpt_w195.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6a1373f6dff85ae37cacfe6fbf916553391c054b89fd598c84aa387fc1265b2c +size 2152006603 diff --git a/ckpt_w196.pt b/ckpt_w196.pt new file mode 100644 index 0000000000000000000000000000000000000000..9e2c980d4debbcc8091bec122e07f85a2c54a81c --- /dev/null +++ b/ckpt_w196.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6265893afe18cfd5e25eaf9099c7bbc5897e72accd03148c89c9e352e4d22cc4 +size 2152006603 diff --git a/ckpt_w197.pt b/ckpt_w197.pt new file mode 100644 index 0000000000000000000000000000000000000000..cab9a972d44dff9cfc1f746cc5fb57829ce6d854 --- /dev/null +++ b/ckpt_w197.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b306b2850c3d2c6ee7a7120ba10d38eefc6632a0193a5b2ec5aec6999fe1de6d +size 2152006603 diff --git a/ckpt_w198.pt b/ckpt_w198.pt new file mode 100644 index 0000000000000000000000000000000000000000..af1b7f7eb42ff27ed74e3630573937216672b853 --- /dev/null +++ b/ckpt_w198.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2eb6cf5ec4d7ff23c09dd5bdf66eb73210c0ea82249438af34649674d9be6eb9 +size 2152006603 diff --git a/ckpt_w199.pt b/ckpt_w199.pt new file mode 100644 index 0000000000000000000000000000000000000000..d595f906ab9e0676bb8271ab635e613e6078a20d --- /dev/null +++ b/ckpt_w199.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b5670e7c17085b04ad3c648ca53166adf8b3831d5d297aa6aa3ac900525d2337 +size 2152006603 diff --git a/ckpt_w2.pt b/ckpt_w2.pt new file mode 100644 index 0000000000000000000000000000000000000000..81c08144e9dcdf3978af7d383916428dc723f440 --- /dev/null +++ b/ckpt_w2.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ef8a5b8b4196f93353470457ed26e187f04e1f0790cb48c247fd1b2287f2ec21 +size 2152005695 diff --git a/ckpt_w200.pt b/ckpt_w200.pt new file mode 100644 index 0000000000000000000000000000000000000000..63b85d402e0374b8a88a2df67f36250fdef61762 --- /dev/null +++ b/ckpt_w200.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9fe761316afbf8c49616ebd8bc1ec41af8006a3f63b4aab7c0c32cc60f1f978f +size 2152006603 diff --git a/ckpt_w201.pt b/ckpt_w201.pt new file mode 100644 index 0000000000000000000000000000000000000000..e77efa52908bf2dd9fd5e7a6e00aab2eafd2e318 --- /dev/null +++ b/ckpt_w201.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b266bf410aa19860edfb153b0552c16515be414bdc2976121eb8234f84b463ea +size 2152006603 diff --git a/ckpt_w202.pt b/ckpt_w202.pt new file mode 100644 index 0000000000000000000000000000000000000000..fa1a3753740d29d2826c5171f08c310508655608 --- /dev/null +++ b/ckpt_w202.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3cd00dda69e3708d9740b41188cddb63d6e03a5193cdb3500c8e6dbe0b229069 +size 2152006603 diff --git a/ckpt_w203.pt b/ckpt_w203.pt new file mode 100644 index 0000000000000000000000000000000000000000..d65cd924090d3fe58bb2aad581ad19f3c9a5d7d9 --- /dev/null +++ b/ckpt_w203.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:35169c0c80305eaab1014cae2ab08240fe353871e34c1c48e038f217e92cbb00 +size 2152006603 diff --git a/ckpt_w204.pt b/ckpt_w204.pt new file mode 100644 index 0000000000000000000000000000000000000000..ef067ed9f512755410160a334fc7c9e68003607f --- /dev/null +++ b/ckpt_w204.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:31b80a49491c70f8146dbd7df580143605b93138a0521da7587125d1af872113 +size 2152006603 diff --git a/ckpt_w21.pt b/ckpt_w21.pt new file mode 100644 index 0000000000000000000000000000000000000000..cda3a2a9c80e545b03eb02af1c1b0e188a19539c --- /dev/null +++ b/ckpt_w21.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:64f8fca0cd7b30a449dd250a1ffedc7128b7fa55d10f5f43c8502d4540a76ff8 +size 2152006469 diff --git a/ckpt_w22.pt b/ckpt_w22.pt new file mode 100644 index 0000000000000000000000000000000000000000..d1b103f529a10b7b67aef6b1f051083e960f0eb3 --- /dev/null +++ b/ckpt_w22.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b154be2bfcc787866d464750df8dd1af9e31da3d19842e71813790bd013c0975 +size 2152006469 diff --git a/ckpt_w23.pt b/ckpt_w23.pt new file mode 100644 index 0000000000000000000000000000000000000000..d08395e41bdb656f6ea13a27db8ce24bb436f368 --- /dev/null +++ b/ckpt_w23.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:119ebe833e2cebafeae94db99f467a5de6be0b73a6e81bb93503676e2ae94d6f +size 2152006469 diff --git a/ckpt_w24.pt b/ckpt_w24.pt new file mode 100644 index 0000000000000000000000000000000000000000..e9f2a89097aa454bdb905c19347c0626229f3019 --- /dev/null +++ b/ckpt_w24.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9f5c6a2eb268c9f7d587baab0948e2f19b3eeef90d52a0222817cf7189f32bd9 +size 2152006469 diff --git a/ckpt_w25.pt b/ckpt_w25.pt new file mode 100644 index 0000000000000000000000000000000000000000..09f54547d4744f9ad288d14d44a98e62db621a45 --- /dev/null +++ b/ckpt_w25.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:27369acf9bee3682bf1753ee4eeb124f0b2ba0d0f7affffa460d98d187ef2c7e +size 2152006469 diff --git a/ckpt_w26.pt b/ckpt_w26.pt new file mode 100644 index 0000000000000000000000000000000000000000..2c0ed4e6a468dac899abe486b1b3851acf7ecc71 --- /dev/null +++ b/ckpt_w26.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e68de96aa60f0fa7300f76c18730e370dc93dc551c8576e1aa1d257f51a60acd +size 2152006469 diff --git a/ckpt_w3.pt b/ckpt_w3.pt new file mode 100644 index 0000000000000000000000000000000000000000..72a0d276fc26c0a314bca652d219df1323fcb789 --- /dev/null +++ b/ckpt_w3.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1eedc9136f658d511cb24e6aebd609692c2c60d4bfe321745cc4d1738d14bdd7 +size 2152005695 diff --git a/ckpt_w31.pt b/ckpt_w31.pt new file mode 100644 index 0000000000000000000000000000000000000000..d9e5954dc95aba7e3765f5105a34f8d0b7969ef4 --- /dev/null +++ b/ckpt_w31.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7faa5f7df91484e6f08b3fd1dee931c3242c7de97fa359bc3e2651a4b7cf3ea8 +size 2152006469 diff --git a/ckpt_w32.pt b/ckpt_w32.pt new file mode 100644 index 0000000000000000000000000000000000000000..b94145e69f716a26d1acfad24047dc68e826dda8 --- /dev/null +++ b/ckpt_w32.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:519a86d31688acde05dc815481ceaafb59d5ce7390540725a296ddc9ab78cbca +size 2152006469 diff --git a/ckpt_w33.pt b/ckpt_w33.pt new file mode 100644 index 0000000000000000000000000000000000000000..b8067b847b0ee62700927f3929211ee4958c4dc0 --- /dev/null +++ b/ckpt_w33.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:13dbd9545154870c24864a9bd946f96b64baca027a2f67d537bb428044c9b479 +size 2152006469 diff --git a/ckpt_w4.pt b/ckpt_w4.pt new file mode 100644 index 0000000000000000000000000000000000000000..9056c4aa5ba18debbf1efa9c0423f36fa88c8721 --- /dev/null +++ b/ckpt_w4.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:815cf00bc811451372a557eb4c266c04d5acef18af29cc48e6defc64fcedf0b4 +size 2152005695 diff --git a/ckpt_w41.pt b/ckpt_w41.pt new file mode 100644 index 0000000000000000000000000000000000000000..502ee84e832dc845f43aa8d0aaeb76d1cca2f6d4 --- /dev/null +++ b/ckpt_w41.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7418cb6b3922befa5ed5afdd5a0b9c649032ca97fa5712c8f6c9994e7f90509c +size 2152006469 diff --git a/ckpt_w42.pt b/ckpt_w42.pt new file mode 100644 index 0000000000000000000000000000000000000000..0e296cbbb3cf1efe8b5e3ad9c918e6b8a86b61c9 --- /dev/null +++ b/ckpt_w42.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:84e62ffa518632d397a0456b1a3366a0ca348b7954ac19f7f5b32d1bc86f8f05 +size 2152006469 diff --git a/ckpt_w43.pt b/ckpt_w43.pt new file mode 100644 index 0000000000000000000000000000000000000000..c23627ef71dcd848388e7803fa9373f0d9388590 --- /dev/null +++ b/ckpt_w43.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a0ad6e6c008a464e4a8136910eabea02a30237742472feb16121da5eb7453200 +size 2152006469 diff --git a/ckpt_w44.pt b/ckpt_w44.pt new file mode 100644 index 0000000000000000000000000000000000000000..6f26613256391f9bbe8fc49b7e5ed0bafd478278 --- /dev/null +++ b/ckpt_w44.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:105f613755b89dc16be21a268c2b21000f1eb59587c3cba458a95ae5c42c8abc +size 2152006469 diff --git a/ckpt_w45.pt b/ckpt_w45.pt new file mode 100644 index 0000000000000000000000000000000000000000..850b4b52593376a8d3919a7595748bd2cb86f866 --- /dev/null +++ b/ckpt_w45.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b4899c64d629b3709e6decbb0c55d28c816dc1dc70c05ffaf13637c880988340 +size 2152006469 diff --git a/ckpt_w5.pt b/ckpt_w5.pt new file mode 100644 index 0000000000000000000000000000000000000000..df72ac31af73fbaf29cf4d437aa59e614a5dfcf8 --- /dev/null +++ b/ckpt_w5.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a47f838fa4d4a912eb3d51dccb813dc36bedf43b0acd3361805f472f8938c06c +size 2152005695 diff --git a/ckpt_w51.pt b/ckpt_w51.pt new file mode 100644 index 0000000000000000000000000000000000000000..1d448542b66f431a9e65478bd369301d63f064bf --- /dev/null +++ b/ckpt_w51.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8e0004d2bb0fe9034af774bf6abc4cb39efdb4f1a5af1f1b377ee1f375bb3748 +size 2152006469 diff --git a/ckpt_w52.pt b/ckpt_w52.pt new file mode 100644 index 0000000000000000000000000000000000000000..9c31fa8106f479726b723ec84a44e8e0c7295e68 --- /dev/null +++ b/ckpt_w52.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bf6ea265bebe13c825443c2214ef813e7252b41224bccbd2c435d9a69d2de8be +size 2152006469 diff --git a/ckpt_w53.pt b/ckpt_w53.pt new file mode 100644 index 0000000000000000000000000000000000000000..1edd2e46b5c03568d3ddcfabe03a9af22566d538 --- /dev/null +++ b/ckpt_w53.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0e67c58175662113f7e48e3a8d9147f001e56928cad19a67b74e486ccc3852f8 +size 2152006469 diff --git a/ckpt_w54.pt b/ckpt_w54.pt new file mode 100644 index 0000000000000000000000000000000000000000..83ee74e431060d2501308a2d01a4426e4f5618d9 --- /dev/null +++ b/ckpt_w54.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a9a127e624421551797c00bbaaad2ea27d803d8894e17e07318b0a1bc06e28b0 +size 2152006469 diff --git a/ckpt_w55.pt b/ckpt_w55.pt new file mode 100644 index 0000000000000000000000000000000000000000..e2d9f49d4c62c92a7fcc03c649f3d42a8bfedc4f --- /dev/null +++ b/ckpt_w55.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c7bae9091b72755d602af402827416697b2e092b37dd248091cc8a669c8c31d1 +size 2152006469 diff --git a/ckpt_w56.pt b/ckpt_w56.pt new file mode 100644 index 0000000000000000000000000000000000000000..08662212cedd1c17c121922d2e8f501ec457103a --- /dev/null +++ b/ckpt_w56.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:947eadad1c686d1b01c47422c5ba52c635da645963ab94187efa5bc2e8cd5dc0 +size 2152006469 diff --git a/ckpt_w57.pt b/ckpt_w57.pt new file mode 100644 index 0000000000000000000000000000000000000000..a19b8728624890c22bedc1a4ca6a3aed17a71bcd --- /dev/null +++ b/ckpt_w57.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:56ac63ae2ed70df0d0f02df4596de09737fb228bfa352762e29f19b9128bc975 +size 2152006469 diff --git a/ckpt_w58.pt b/ckpt_w58.pt new file mode 100644 index 0000000000000000000000000000000000000000..d9be03060ddfeb3a43c2f67fdbdfb721b1488bcc --- /dev/null +++ b/ckpt_w58.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6ba033681a989d8203eb908ce99e21cf2f4e54de82dec9bf6c2b88b0678e0204 +size 2152006469 diff --git a/ckpt_w59.pt b/ckpt_w59.pt new file mode 100644 index 0000000000000000000000000000000000000000..c13a2eae9155d0a9ea8dfd4899cd5abf8614a43a --- /dev/null +++ b/ckpt_w59.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:38e2f923fab8281aed075fe62a18ebc60f712bdf7605fb102bc6db53e9e2b242 +size 2152006469 diff --git a/ckpt_w6.pt b/ckpt_w6.pt new file mode 100644 index 0000000000000000000000000000000000000000..163ef5a2da092a5e19fde4da9fcf65d6f6156902 --- /dev/null +++ b/ckpt_w6.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:83e7f748317183f48e44f51d436210acbc033c974021f420642cf93fcaf4a745 +size 2152005695 diff --git a/ckpt_w60.pt b/ckpt_w60.pt new file mode 100644 index 0000000000000000000000000000000000000000..472d51d9605df6c1e76f1281ad39cb17338fe521 --- /dev/null +++ b/ckpt_w60.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b756403f03baf1c7a0a02789fea349746fc506093859299f21853be88ff89655 +size 2152006469 diff --git a/ckpt_w63.pt b/ckpt_w63.pt new file mode 100644 index 0000000000000000000000000000000000000000..603d19755b0de0cf16326e6e849cc825c3122b53 --- /dev/null +++ b/ckpt_w63.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c3fdb6134712080ff4e3abdf152c968ee88d2739a22674c01858f7a679a0d4a9 +size 2152006469 diff --git a/ckpt_w64.pt b/ckpt_w64.pt new file mode 100644 index 0000000000000000000000000000000000000000..3e5c16b6a49cbd1de8910e5b2fd89bc6988527b4 --- /dev/null +++ b/ckpt_w64.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b15f2eac156c398b6615b35d532f001e955654c6c3e17413a4da81272db18e0a +size 2152006469 diff --git a/ckpt_w65.pt b/ckpt_w65.pt new file mode 100644 index 0000000000000000000000000000000000000000..e95cfdd15d2d0839ee90a0e8c06ac28034aab921 --- /dev/null +++ b/ckpt_w65.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:21559e41b234ecb3f06d876515548839acc13af4ece3576e3bcf6d245a791d9a +size 2152006469 diff --git a/ckpt_w66.pt b/ckpt_w66.pt new file mode 100644 index 0000000000000000000000000000000000000000..f05b60f44a180bb0e91d4be2a8fccc5ebeecc594 --- /dev/null +++ b/ckpt_w66.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b97b5994d0f3c26b51c465ce0e5ea7a83cb0cb70ed38a6d3eb2a5cba1b2d1fc2 +size 2152006469 diff --git a/ckpt_w67.pt b/ckpt_w67.pt new file mode 100644 index 0000000000000000000000000000000000000000..aa7adb926c860b7dc37881e2e078f088226b8b23 --- /dev/null +++ b/ckpt_w67.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:59e6516fe8bd55fb53c599ff017150020eb893899b522543fe1a22da1d616676 +size 2152006469 diff --git a/ckpt_w68.pt b/ckpt_w68.pt new file mode 100644 index 0000000000000000000000000000000000000000..e96cc28a89f2a3a75b541072a4d66fb10430b96a --- /dev/null +++ b/ckpt_w68.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:07a39c6179c3d9ad10b395e2d2b1d0346ad3c14043cd523cdd5a7349f30411e1 +size 2152006469 diff --git a/ckpt_w7.pt b/ckpt_w7.pt new file mode 100644 index 0000000000000000000000000000000000000000..d9a88d2c46bb21def610c9804e8e41e0409135df --- /dev/null +++ b/ckpt_w7.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8515bb365d5eb7024630ad5982eb796966c832364761349db27ceaa638b05724 +size 2152005695 diff --git a/ckpt_w71.pt b/ckpt_w71.pt new file mode 100644 index 0000000000000000000000000000000000000000..963ff2275f174fc54db700d062f325ad06786aa8 --- /dev/null +++ b/ckpt_w71.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:09fd98537d49555f44e6d07388d9bb7f776f8a5887345015703cfd9dddb32837 +size 2152006469 diff --git a/ckpt_w72.pt b/ckpt_w72.pt new file mode 100644 index 0000000000000000000000000000000000000000..0d19daeed3353e4e1f74f57515ac45de12689fb1 --- /dev/null +++ b/ckpt_w72.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:299eae14db4046618d6088034b79f143957f0d89b7876675323d1795846a317b +size 2152006469 diff --git a/ckpt_w74.pt b/ckpt_w74.pt new file mode 100644 index 0000000000000000000000000000000000000000..39f41b564f7c9eacbaa99a51844d2b311abaaab0 --- /dev/null +++ b/ckpt_w74.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:456a595a5711935422cc549f04eb25b04d79b5cdadae6297a40c05bc9636c65a +size 2152006469 diff --git a/ckpt_w75.pt b/ckpt_w75.pt new file mode 100644 index 0000000000000000000000000000000000000000..364e2da47cbfb2da83815d64d6933848287868d7 --- /dev/null +++ b/ckpt_w75.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ecc74e4a27a96ab17ae20218331b7628108b712bd6442083a29573f1f0e5ed0e +size 2152006469 diff --git a/ckpt_w76.pt b/ckpt_w76.pt new file mode 100644 index 0000000000000000000000000000000000000000..0b134fc9ac593691aa24923a511450a36af987c1 --- /dev/null +++ b/ckpt_w76.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:301e9682a7a7fa665dab533524ad2af3944911818d025bf6cdb0a4e4b73e206a +size 2152006469 diff --git a/ckpt_w77.pt b/ckpt_w77.pt new file mode 100644 index 0000000000000000000000000000000000000000..f7b87c9d02a03cfe8da26b5cd3f193879409e8ff --- /dev/null +++ b/ckpt_w77.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ce5f35cea4018d5e2f0926c1d604ba7f76c0882900ef19c388eaa65c91e21e4c +size 2152006469 diff --git a/ckpt_w78.pt b/ckpt_w78.pt new file mode 100644 index 0000000000000000000000000000000000000000..0aca4b1a20d2a122e3fbf4d12e7faaf41f43790a --- /dev/null +++ b/ckpt_w78.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c10e0326a2c0355fdb9c1eb05035058a2bac2420195e1c61fd88f7d6e2eefd17 +size 2152006469 diff --git a/ckpt_w79.pt b/ckpt_w79.pt new file mode 100644 index 0000000000000000000000000000000000000000..c84b1bf7c5d5f3312cdefcf23b26e9e9284f28dd --- /dev/null +++ b/ckpt_w79.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a0dc3c683e4e036f83a82746b15256636374953235e5ab9d33c3a84619f610ed +size 2152006469 diff --git a/ckpt_w8.pt b/ckpt_w8.pt new file mode 100644 index 0000000000000000000000000000000000000000..fdeee2d233fda05bfdf779bbbbe7e9cfe6f0bc85 --- /dev/null +++ b/ckpt_w8.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:78de79df578f942d64df7e5f70a08564fc6d3840449e0d9fd4f39663fb31a9cf +size 2152005695 diff --git a/ckpt_w80.pt b/ckpt_w80.pt new file mode 100644 index 0000000000000000000000000000000000000000..35c6160c7625083474b5c230aa18ba46e7217b47 --- /dev/null +++ b/ckpt_w80.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9d486470f55cbba0ca8660150e827080c4750d40080b23ff1b1fc9b88a0842af +size 2152006469 diff --git a/ckpt_w81.pt b/ckpt_w81.pt new file mode 100644 index 0000000000000000000000000000000000000000..1ef83f206c2a6969daaa154dd148a643d343fa8d --- /dev/null +++ b/ckpt_w81.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:25bba45048505f682466bc95b5b11efe70169b7e3290467bbf4b88d36f1b39c4 +size 2152006469 diff --git a/ckpt_w82.pt b/ckpt_w82.pt new file mode 100644 index 0000000000000000000000000000000000000000..9ebda4cc19f7af17e657519d74d2e17b91e31468 --- /dev/null +++ b/ckpt_w82.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:96460d467afb5a4179183dd27003f86fbdcf7cc6fa4679d7df233d5f542f4a66 +size 2152006469 diff --git a/ckpt_w84.pt b/ckpt_w84.pt new file mode 100644 index 0000000000000000000000000000000000000000..1f81368e72884592c4f992ca3e1281d8d5a83d66 --- /dev/null +++ b/ckpt_w84.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5f3016a262d22aed0909b7700d9b221fbe44cfea1d245e9a498e3d5ba3ea6d38 +size 2152006469 diff --git a/ckpt_w86.pt b/ckpt_w86.pt new file mode 100644 index 0000000000000000000000000000000000000000..0c607ad9f81cd2843698872f5f37090ef8eb3ede --- /dev/null +++ b/ckpt_w86.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:647f063d9e73b67a1b5e6d6936dd2736c629516208c560f97a44fbfcac9a3d36 +size 2152006469 diff --git a/ckpt_w87.pt b/ckpt_w87.pt new file mode 100644 index 0000000000000000000000000000000000000000..d83a9ddb9b1f1e5a39f31612d2cce736bba842b7 --- /dev/null +++ b/ckpt_w87.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:54508a11fc6f536001b063eab49688cabd06b5a7a123361650ab22ac4b2273f2 +size 2152006469 diff --git a/ckpt_w88.pt b/ckpt_w88.pt new file mode 100644 index 0000000000000000000000000000000000000000..74ad54d3e117bf70aee04837e4ddd50c4666f07a --- /dev/null +++ b/ckpt_w88.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:566d856cacd3f0227ec8ecf9d2d5339ce454e8a0b4d20846cbd65a5a8c52caad +size 2152006469 diff --git a/ckpt_w9.pt b/ckpt_w9.pt new file mode 100644 index 0000000000000000000000000000000000000000..706f8761b3fe1825dba54401adc55034987885f2 --- /dev/null +++ b/ckpt_w9.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:765330a588f4d22fce485e5f404544418b5f112acdd1af6112998eac1fc46b8a +size 2152005695 diff --git a/ckpt_w90.pt b/ckpt_w90.pt new file mode 100644 index 0000000000000000000000000000000000000000..a7fc6eaf6806f40e18525795cdf10964ee3f00e1 --- /dev/null +++ b/ckpt_w90.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2363c0c783e981c5f199a8c1d10598a6cdb35d400f3e58ec6c806c18617cd3ad +size 2152006469 diff --git a/ckpt_w92.pt b/ckpt_w92.pt new file mode 100644 index 0000000000000000000000000000000000000000..1051485f2df3c0e20288e142dc4a78088ac2f48e --- /dev/null +++ b/ckpt_w92.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b0d45c85416c8d21276acd21c697b5725fddde7e9c2206351e9cc78d63360e1b +size 2152006469 diff --git a/ckpt_w93.pt b/ckpt_w93.pt new file mode 100644 index 0000000000000000000000000000000000000000..f8981ed3c6713b9084ebe3948022538271eb2304 --- /dev/null +++ b/ckpt_w93.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b3a50c16ded4a45af596d16134bd7ed87a14087c797c5ea0146061ad0d9c9bc5 +size 2152006469 diff --git a/ckpt_w94.pt b/ckpt_w94.pt new file mode 100644 index 0000000000000000000000000000000000000000..904469b2c4606020560e2caf3778b42c6cc414db --- /dev/null +++ b/ckpt_w94.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d57f686f9b051e8de6f5f7bb78ae893a8ed221d51cc0ac12c6f64f62e6f1491c +size 2152006469 diff --git a/ckpt_w95.pt b/ckpt_w95.pt new file mode 100644 index 0000000000000000000000000000000000000000..9afd4dd2f8e19546a68e59263a3f728b85882d7d --- /dev/null +++ b/ckpt_w95.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:efa7e24f72859a5221d9c1945a41cd0e842ed0720cd7f8bb880b2668e1bdd59a +size 2152006469 diff --git a/ckpt_w96.pt b/ckpt_w96.pt new file mode 100644 index 0000000000000000000000000000000000000000..eeccee388eb307d4040f5a2450337221bc1a8ae3 --- /dev/null +++ b/ckpt_w96.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:85e2bc969bc8266509ea983b8b48b44b293440b4aa02212af28d64df6bf760b2 +size 2152006469 diff --git a/ckpt_w97.pt b/ckpt_w97.pt new file mode 100644 index 0000000000000000000000000000000000000000..5153210555281a266e6b88bd52f5235811700f72 --- /dev/null +++ b/ckpt_w97.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9e9101d5a6da74b3a58fdf6096a30748722f23360f8ddc8819ef011746214b82 +size 2152006469 diff --git a/ckpt_w98.pt b/ckpt_w98.pt new file mode 100644 index 0000000000000000000000000000000000000000..e0fd465fc2cfc79e4cfe06faf399b6a7ff40059e --- /dev/null +++ b/ckpt_w98.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:05cbcf329a71ad66b5ce0acdadf017b06c0730f44628ff81abdae7fde142284c +size 2152006469 diff --git a/config.json b/config.json new file mode 100644 index 0000000000000000000000000000000000000000..a58d212cf390614f8248caa6ff51a8b249b75e8f --- /dev/null +++ b/config.json @@ -0,0 +1,144 @@ +{ + "architectures": [ + "MaskedDiffusionLM" + ], + "model_type": "diffurefill", + "_comment": "Not a HuggingFace-loadable config. The reference implementation is big_common.py in this repo; the fields below describe that module exactly.", + "hidden_size": 2048, + "num_hidden_layers": 18, + "num_attention_heads": 16, + "num_key_value_heads": 16, + "head_dim": 128, + "intermediate_size": 5632, + "hidden_act": "silu", + "mlp_type": "swiglu", + "norm_type": "rmsnorm", + "norm_placement": "pre", + "attention_bias": false, + "mlp_bias": false, + "position_encoding": "rope", + "rope_theta": 10000.0, + "rope_applied_to": [ + "q", + "k" + ], + "max_position_embeddings": 2048, + "attention_mask": "bidirectional", + "attention_impl": "scaled_dot_product_attention; flex_attention with a document block mask when BG_DOCMASK=1", + "vocab_size": 73760, + "tokenizer": "openbmb/MiniCPM4-0.5B", + "tokenizer_vocab_size": 73440, + "mask_token_id": 73759, + "pad_token_id": 73758, + "tie_word_embeddings": true, + "torch_dtype": "bfloat16", + "parameters": { + "total": 1076000000, + "embedding_tied_with_head": 151060480, + "per_block": 51384320, + "per_block_attention": 16777216, + "per_block_mlp": 34603008, + "note": "vocab is padded from the tokenizer's 73440 to 73760 to reserve MASK and PAD and keep the head a round size" + }, + "layer_order": [ + "tok_embedding (73760 x 2048, tied to output head)", + "18 x Block:", + " RMSNorm -> Linear(2048 -> 6144, no bias) -> split q,k,v -> RoPE(q,k) -> bidirectional SDPA -> Linear(2048 -> 2048, no bias) -> residual add", + " RMSNorm -> [Linear(2048 -> 5632) SiLU] * [Linear(2048 -> 5632)] -> Linear(5632 -> 2048) -> residual add", + "final RMSNorm", + "output head = tok_embedding transposed (tied)" + ], + "init": { + "embedding": "normal(0, 0.02)", + "block_linears": "normal(0, 0.02 / sqrt(2 * num_hidden_layers))", + "note": "the head is tied to the embedding, so the embedding scale sets the logit scale" + }, + "objective": { + "type": "masked_absorbing_diffusion", + "loss": "cross_entropy on masked positions only", + "mask_rate": "t ~ Uniform(0.15, 1.0) per row, positions masked independently", + "expected_supervised_fraction": 0.575, + "prefix_conditioning_prob": 0.5, + "prefix_note": "on half of rows a random-length clean prefix is never masked, so training sees the prompt-conditioned shape generation uses", + "head_note": "only masked positions are pushed through the 73760-wide head", + "document_mask": "attention is kept inside each packed document (flex_attention block mask)", + "salient_span_masking": { + "rows": 0.25, + "rule": "on a quarter of rows, names and numbers - capitalised words that do not merely open a sentence, and digit runs, merged into multi-word spans - are masked as whole units with probability sqrt(t) >= t; all other positions keep their independent draw", + "why": "masked independently, a two-token name is hidden in full with probability t^2 and the visible half gives it away; hidden as a unit it must be recalled from the rest of the sentence" + }, + "span_masking_prob": 0.0, + "span_note": "contiguous span masking (mean 3) was used until step ~91.9k and then removed: independent masking beat it both by one-pass CE and by decoding accuracy on the 37.9M stand. PMI-unit masking, stratified mask rates and complementary masks were also tested there and lost to plain independent masking." + }, + "training": { + "optimizer": "AdamW", + "betas": [ + 0.9, + 0.95 + ], + "eps": 1e-08, + "weight_decay": 0.1, + "grad_clip": 1.0, + "peak_lr": 0.00015, + "warmup_steps": 500, + "schedule": "cosine to 0.1 of peak over 200000 steps", + "sequence_length": 2048, + "tokens_per_optimizer_step": 131072, + "batch_sequences": 64, + "micro_batch": 16, + "dataset": "see dataset_mix", + "distributed": { + "method": "DiLoCo through this repository", + "workers": "3-5 single-GPU workers (RTX PRO 6000 Blackwell, 96 GB), boxes come and go", + "sharding": "each worker takes every Nth document of the base corpus from its own shuffle; the extra sources are read unsharded", + "local_steps_between_merges": 150, + "outer_optimizer": "plain average (momentum 0, lr 1); outer Nesterov was tried and raised held-out CE", + "global_adoption": "theta <- global (plain replacement)", + "guard": "a merge is rejected if it raises held-out masked CE by more than 25%", + "global_adoption_note": "an earlier version kept local drift across a merge (theta += global - last_pushed). Measured on a fixed probe it made the model worse in 60 of 63 adoptions, mean +0.079 CE, against -0.08 to -0.10 for plain replacement: it preserves exactly the disagreeing component that averaging exists to divide by N, and it accumulates.", + "merge": "the merger folds each worker checkpoint into a running fp32 sum as it arrives, waits at most 150 s past the first arrival, and publishes global_b.pt", + "phase": "workers adopt a new global as soon as it appears (checked every 10 steps) and take its step counter, so the whole fleet starts each 150-step interval together" + }, + "peak_lr_note": "halved from 3e-4. At 3e-4 a worker drifted 0.076 CE away from the merge point over one 150-step interval, which is more than averaging can reconcile; at 1.5e-4 the drift is 0.013 and the merged model fell from 4.336 to 4.288 on a fixed held-out probe.", + "dataset_mix": { + "note": "documents are drawn by source with these probabilities from step ~96.8k; before that the model saw only the two Ultra-FineWeb configs", + "openbmb/Ultra-FineWeb-L3 (Multi-Style-Synthetic + QA-Synthetic)": 0.56, + "wikimedia/wikipedia 20231101.en": 0.15, + "HuggingFaceTB/finemath finemath-4plus": 0.08, + "HuggingFaceTB/smollm-corpus cosmopedia-v2": 0.08, + "OpenCoder-LLM/opc-annealing-corpus algorithmic_corpus": 0.04, + "OpenCoder-LLM/opc-annealing-corpus synthetic_code_snippet": 0.04, + "Asilarkness/DiffuRefill-facts (Wikipedia leads restated as textbook, lecture, museum-guide ... prose)": 0.05 + }, + "ema": { + "beta": 0.9, + "file": "global_ema.pt", + "note": "an exponential average of the merged globals; it beats the latest global on held-out CE every round. It is published for use, never used to seed training." + }, + "compile": "each block is torch.compile'd in place: 17% faster steps and less memory, same parameter names" + }, + "decoding_default": { + "draft_passes_K": 12, + "refill_rounds": 8, + "refill_fraction": 0.2, + "draft_temp_start": 2.2, + "draft_temp_end": 0.9, + "gumbel_tau": 2.5, + "refill_temp": 0.8, + "refill_min_p": 0.1, + "neighbour_ban": 6.0, + "min_commit_gap": 2, + "min_commit_gap_note": "two positions closer than this are never committed in the same pass; co-committed tokens are sampled from their own marginals as if independent, and neighbours are where that is most wrong. Measured on held-out text: adjacent-pair accuracy rises at every pass budget." + }, + "status": { + "stage": "pretraining, about half way", + "lineage_step": 100050, + "target_steps": 200000, + "effective_tokens": 46150000000, + "effective_tokens_note": "summed over every worker interval that went into a merge (one interval = 150 steps x 131072 tokens); the per-worker step counter alone undercounts by the number of workers", + "round": 290, + "best_checkpoint": "global_ema.pt", + "note": "a pretraining checkpoint, not an instruction-tuned model; it writes fluent English but its facts are still unreliable" + }, + "attention_note": "documents are packed back to back, so without a mask a masked position can be filled from an unrelated neighbouring document. The block-diagonal mask is not a speed cost: measured at the training shape it runs 13.2 ms against 17.5 ms unmasked, because blocks lying in no document are skipped entirely." +} \ No newline at end of file diff --git a/global_b.pt b/global_b.pt new file mode 100644 index 0000000000000000000000000000000000000000..a2bc92bb55053b6f10d9166f29b4768546a3d16e --- /dev/null +++ b/global_b.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:37fcd0c599f7f8f8e75075c61debdb240874768173e95795dcd9e4d6a0881fd9 +size 2454120580 diff --git a/global_b_pre_muon.pt b/global_b_pre_muon.pt new file mode 100644 index 0000000000000000000000000000000000000000..23745facbd9dc5f145498c6b05ff32af8fa51b66 --- /dev/null +++ b/global_b_pre_muon.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b9446693bb9cf688a1fea82f5b39917c90795db06e422f8bae48da5471e87ab3 +size 2454120580 diff --git a/global_b_pre_nesterov.pt b/global_b_pre_nesterov.pt new file mode 100644 index 0000000000000000000000000000000000000000..fa92f8659d61e566979cb917a3e5ab1f8f621e43 --- /dev/null +++ b/global_b_pre_nesterov.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:eedb859d0614e73a3012c05b75bbea2dc44cc5d7e93a8856371f90f594a15b33 +size 2454120580 diff --git a/global_b_pre_nospan.pt b/global_b_pre_nospan.pt new file mode 100644 index 0000000000000000000000000000000000000000..93816150fb49905bceee0b9b7f8501e5f509a957 --- /dev/null +++ b/global_b_pre_nospan.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b79f9b17247eeb1d0a129ae0fcf7f2653a6d92d97a5eabe9c5401b1a90d5fce3 +size 2454120580 diff --git a/global_ema.pt b/global_ema.pt new file mode 100644 index 0000000000000000000000000000000000000000..11e978f1399e34a5a5864857dbde467be3329962 --- /dev/null +++ b/global_ema.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a1dfe8ab2fe8312e369d7c011b648efd66f151cd305c3bd36e5fc68a5f699640 +size 2454120914 diff --git a/global_ema_best.pt b/global_ema_best.pt new file mode 100644 index 0000000000000000000000000000000000000000..59cb94ac4e8ccaa3f21ea32b9929c0fe55a24572 --- /dev/null +++ b/global_ema_best.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2127d6a51e48027c8d50cdecc9ba70a08bb734bb2471fa8872f712930e9e29d1 +size 2454111457 diff --git a/global_round_b.txt b/global_round_b.txt new file mode 100644 index 0000000000000000000000000000000000000000..54d5fa6f52fd8cf8765513c6457a9ba6147b718a --- /dev/null +++ b/global_round_b.txt @@ -0,0 +1 @@ +881 \ No newline at end of file diff --git a/global_swa.pt b/global_swa.pt new file mode 100644 index 0000000000000000000000000000000000000000..6a6dd3e0a11c58aa5a447813320358ec07873c54 --- /dev/null +++ b/global_swa.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c46a16ecfc4d417b8883165b3a828277f4a5407b9543b2e3b6b98ce87d729ff1 +size 2454120850 diff --git a/log_w0.txt b/log_w0.txt new file mode 100644 index 0000000000000000000000000000000000000000..91eda3e821c8e40be5e52cd15950764eb8aeae31 --- /dev/null +++ b/log_w0.txt @@ -0,0 +1,69 @@ +launch +13:22:51 name=w0 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:22:56 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:23:06 params 1.08B +13:23:07 resumed step=25200 seen=3.30B round=13 +13:25:00 start: accum=4, 131072 tokens per optimiser step +13:28:05 step 25220 loss=3.7952 lr=2.90e-04 tok=3.306B dt=184 +13:29:49 step 25240 loss=3.9700 lr=2.90e-04 tok=3.308B dt=288 +13:31:32 step 25260 loss=3.8876 lr=2.90e-04 tok=3.311B dt=392 +13:33:16 step 25280 loss=3.8982 lr=2.90e-04 tok=3.314B dt=495 +13:35:01 step 25300 loss=3.7434 lr=2.90e-04 tok=3.316B dt=601 +13:36:46 step 25320 loss=3.8454 lr=2.90e-04 tok=3.319B dt=706 +13:38:31 step 25340 loss=3.8361 lr=2.90e-04 tok=3.321B dt=811 +13:40:16 step 25360 loss=3.8574 lr=2.90e-04 tok=3.324B dt=915 +13:42:01 step 25380 loss=3.7867 lr=2.90e-04 tok=3.327B dt=1020 +13:43:45 step 25400 loss=3.8894 lr=2.90e-04 tok=3.329B dt=1125 +13:45:31 step 25420 loss=3.7817 lr=2.90e-04 tok=3.332B dt=1231 +13:47:16 step 25440 loss=3.7516 lr=2.90e-04 tok=3.334B dt=1336 +13:49:00 step 25460 loss=3.8250 lr=2.90e-04 tok=3.337B dt=1440 +13:50:45 step 25480 loss=3.7767 lr=2.90e-04 tok=3.340B dt=1544 +13:52:29 step 25500 loss=3.8634 lr=2.90e-04 tok=3.342B dt=1648 +13:54:14 step 25520 loss=3.9193 lr=2.90e-04 tok=3.345B dt=1754 +13:56:01 step 25540 loss=3.8720 lr=2.90e-04 tok=3.348B dt=1860 +13:57:46 step 25560 loss=3.8153 lr=2.90e-04 tok=3.350B dt=1966 +13:59:32 step 25580 loss=3.8973 lr=2.90e-04 tok=3.353B dt=2071 +14:01:16 step 25600 loss=3.8464 lr=2.90e-04 tok=3.355B dt=2176 +14:03:01 step 25620 loss=3.7995 lr=2.90e-04 tok=3.358B dt=2280 +14:04:45 step 25640 loss=3.7885 lr=2.90e-04 tok=3.361B dt=2384 +14:06:29 step 25660 loss=3.8488 lr=2.90e-04 tok=3.363B dt=2489 +14:08:14 step 25680 loss=3.7583 lr=2.90e-04 tok=3.366B dt=2594 +14:09:59 step 25700 loss=3.8320 lr=2.90e-04 tok=3.369B dt=2699 +14:11:43 step 25720 loss=3.7939 lr=2.90e-04 tok=3.371B dt=2803 +14:13:27 step 25740 loss=3.7970 lr=2.90e-04 tok=3.374B dt=2907 +14:15:09 step 25760 loss=3.8888 lr=2.90e-04 tok=3.376B dt=3009 +14:16:51 step 25780 loss=3.7357 lr=2.89e-04 tok=3.379B dt=3111 +14:18:34 step 25800 loss=3.8406 lr=2.89e-04 tok=3.382B dt=3214 +14:18:49 pushed ckpt_w0.pt +14:18:49 pushed log_w0.txt +14:20:43 step 25820 loss=3.8413 lr=2.89e-04 tok=3.384B dt=3343 +14:22:28 step 25840 loss=3.7843 lr=2.89e-04 tok=3.387B dt=3448 +14:24:12 step 25860 loss=3.6929 lr=2.89e-04 tok=3.390B dt=3551 +14:25:55 step 25880 loss=3.7323 lr=2.89e-04 tok=3.392B dt=3654 +14:27:37 step 25900 loss=3.7930 lr=2.89e-04 tok=3.395B dt=3757 +14:29:19 step 25920 loss=3.8725 lr=2.89e-04 tok=3.397B dt=3859 +14:31:01 step 25940 loss=3.9744 lr=2.89e-04 tok=3.400B dt=3961 +14:32:44 step 25960 loss=3.8792 lr=2.89e-04 tok=3.403B dt=4063 +14:34:27 step 25980 loss=3.7975 lr=2.89e-04 tok=3.405B dt=4166 +14:36:10 step 26000 loss=3.7912 lr=2.89e-04 tok=3.408B dt=4270 +14:37:53 step 26020 loss=3.8384 lr=2.89e-04 tok=3.410B dt=4373 +14:39:36 step 26040 loss=3.8227 lr=2.89e-04 tok=3.413B dt=4476 +14:41:19 step 26060 loss=3.7872 lr=2.89e-04 tok=3.416B dt=4578 +14:43:01 step 26080 loss=3.9084 lr=2.89e-04 tok=3.418B dt=4681 +14:44:44 step 26100 loss=3.7668 lr=2.89e-04 tok=3.421B dt=4783 +14:46:27 step 26120 loss=3.8284 lr=2.89e-04 tok=3.424B dt=4887 +14:48:10 step 26140 loss=3.7803 lr=2.89e-04 tok=3.426B dt=4990 +14:49:53 step 26160 loss=3.8024 lr=2.89e-04 tok=3.429B dt=5093 +14:51:36 step 26180 loss=3.9283 lr=2.89e-04 tok=3.431B dt=5196 +14:53:19 step 26200 loss=3.8302 lr=2.89e-04 tok=3.434B dt=5298 +14:55:02 step 26220 loss=3.8274 lr=2.89e-04 tok=3.437B dt=5401 +14:56:45 step 26240 loss=3.9285 lr=2.89e-04 tok=3.439B dt=5505 +14:58:28 step 26260 loss=3.8275 lr=2.89e-04 tok=3.442B dt=5608 +15:00:12 step 26280 loss=3.7342 lr=2.89e-04 tok=3.445B dt=5711 +15:01:55 step 26300 loss=3.8104 lr=2.89e-04 tok=3.447B dt=5814 +15:03:37 step 26320 loss=3.7588 lr=2.89e-04 tok=3.450B dt=5917 +15:05:20 step 26340 loss=3.8350 lr=2.89e-04 tok=3.452B dt=6020 +15:07:03 step 26360 loss=3.7668 lr=2.89e-04 tok=3.455B dt=6123 +15:08:46 step 26380 loss=3.7747 lr=2.89e-04 tok=3.458B dt=6226 +15:10:29 step 26400 loss=3.9280 lr=2.89e-04 tok=3.460B dt=6329 +15:10:41 pushed ckpt_w0.pt diff --git a/log_w1.txt b/log_w1.txt new file mode 100644 index 0000000000000000000000000000000000000000..d0738dec6158064d2f678941b240c01827a10aad --- /dev/null +++ b/log_w1.txt @@ -0,0 +1,103 @@ +launch +17:18:25 name=w1 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:18:29 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:18:39 params 1.08B +17:18:43 resumed step=27600 seen=3.62B round=14 +17:18:43 stream salt=27600 +17:21:12 start: accum=4, 131072 tokens per optimiser step +17:24:28 step 27620 loss=3.7281 lr=2.88e-04 tok=3.620B dt=196 +17:25:56 step 27640 loss=3.9008 lr=2.88e-04 tok=3.623B dt=284 +17:27:17 step 27660 loss=3.8425 lr=2.88e-04 tok=3.625B dt=365 +17:28:33 step 27680 loss=3.8907 lr=2.88e-04 tok=3.628B dt=441 +17:29:49 step 27700 loss=3.7396 lr=2.88e-04 tok=3.631B dt=517 +17:31:05 step 27720 loss=3.8170 lr=2.88e-04 tok=3.633B dt=593 +17:32:20 step 27740 loss=3.7942 lr=2.88e-04 tok=3.636B dt=668 +17:33:36 step 27760 loss=3.7912 lr=2.88e-04 tok=3.639B dt=744 +17:34:52 step 27780 loss=3.7103 lr=2.88e-04 tok=3.641B dt=820 +17:36:08 step 27800 loss=3.8160 lr=2.88e-04 tok=3.644B dt=896 +17:37:24 step 27820 loss=3.7508 lr=2.88e-04 tok=3.646B dt=972 +17:38:39 step 27840 loss=3.7149 lr=2.88e-04 tok=3.649B dt=1047 +17:39:55 step 27860 loss=3.7555 lr=2.88e-04 tok=3.652B dt=1123 +17:41:11 step 27880 loss=3.7529 lr=2.88e-04 tok=3.654B dt=1199 +17:42:27 step 27900 loss=3.8036 lr=2.88e-04 tok=3.657B dt=1274 +17:43:42 step 27920 loss=3.8604 lr=2.88e-04 tok=3.660B dt=1350 +17:44:58 step 27940 loss=3.8353 lr=2.88e-04 tok=3.662B dt=1426 +17:46:14 step 27960 loss=3.8136 lr=2.88e-04 tok=3.665B dt=1502 +17:47:30 step 27980 loss=3.8956 lr=2.88e-04 tok=3.667B dt=1578 +17:48:46 step 28000 loss=3.8269 lr=2.88e-04 tok=3.670B dt=1654 +17:50:02 step 28020 loss=3.7757 lr=2.88e-04 tok=3.673B dt=1730 +17:51:17 step 28040 loss=3.7364 lr=2.88e-04 tok=3.675B dt=1805 +17:52:33 step 28060 loss=3.7741 lr=2.88e-04 tok=3.678B dt=1881 +17:53:49 step 28080 loss=3.7179 lr=2.88e-04 tok=3.681B dt=1957 +17:55:04 step 28100 loss=3.8188 lr=2.88e-04 tok=3.683B dt=2032 +17:56:20 step 28120 loss=3.7456 lr=2.87e-04 tok=3.686B dt=2108 +17:57:36 step 28140 loss=3.7412 lr=2.87e-04 tok=3.688B dt=2184 +17:58:52 step 28160 loss=3.8056 lr=2.87e-04 tok=3.691B dt=2260 +18:00:07 step 28180 loss=3.6828 lr=2.87e-04 tok=3.694B dt=2335 +18:01:23 step 28200 loss=3.8146 lr=2.87e-04 tok=3.696B dt=2411 +18:01:36 pushed ckpt_w1.pt +18:01:36 pushed log_w1.txt +18:03:01 step 28220 loss=3.8247 lr=2.87e-04 tok=3.699B dt=2509 +18:04:16 step 28240 loss=3.7955 lr=2.87e-04 tok=3.701B dt=2584 +18:05:32 step 28260 loss=3.7267 lr=2.87e-04 tok=3.704B dt=2660 +18:06:48 step 28280 loss=3.6935 lr=2.87e-04 tok=3.707B dt=2735 +18:08:03 step 28300 loss=3.7249 lr=2.87e-04 tok=3.709B dt=2811 +18:09:19 step 28320 loss=3.8477 lr=2.87e-04 tok=3.712B dt=2887 +18:10:35 step 28340 loss=3.9324 lr=2.87e-04 tok=3.715B dt=2963 +18:11:50 step 28360 loss=3.8366 lr=2.87e-04 tok=3.717B dt=3038 +18:13:06 step 28380 loss=3.7735 lr=2.87e-04 tok=3.720B dt=3114 +18:14:22 step 28400 loss=3.7775 lr=2.87e-04 tok=3.722B dt=3190 +18:15:37 step 28420 loss=3.8220 lr=2.87e-04 tok=3.725B dt=3265 +18:16:53 step 28440 loss=3.8039 lr=2.87e-04 tok=3.728B dt=3341 +18:18:09 step 28460 loss=3.7455 lr=2.87e-04 tok=3.730B dt=3417 +18:19:24 step 28480 loss=3.8257 lr=2.87e-04 tok=3.733B dt=3492 +18:20:40 step 28500 loss=3.6993 lr=2.87e-04 tok=3.736B dt=3568 +18:21:55 step 28520 loss=3.8124 lr=2.87e-04 tok=3.738B dt=3643 +18:23:11 step 28540 loss=3.7771 lr=2.87e-04 tok=3.741B dt=3719 +18:24:27 step 28560 loss=3.7726 lr=2.87e-04 tok=3.743B dt=3795 +18:25:43 step 28580 loss=3.9148 lr=2.87e-04 tok=3.746B dt=3870 +18:26:58 step 28600 loss=3.7933 lr=2.87e-04 tok=3.749B dt=3946 +18:28:14 step 28620 loss=3.8110 lr=2.87e-04 tok=3.751B dt=4022 +18:29:30 step 28640 loss=3.9051 lr=2.87e-04 tok=3.754B dt=4098 +18:30:45 step 28660 loss=3.8393 lr=2.87e-04 tok=3.757B dt=4173 +18:32:01 step 28680 loss=3.7699 lr=2.87e-04 tok=3.759B dt=4249 +18:33:16 step 28700 loss=3.7996 lr=2.87e-04 tok=3.762B dt=4324 +18:34:32 step 28720 loss=3.7227 lr=2.87e-04 tok=3.764B dt=4400 +18:35:48 step 28740 loss=3.8075 lr=2.87e-04 tok=3.767B dt=4476 +18:37:03 step 28760 loss=3.7317 lr=2.87e-04 tok=3.770B dt=4551 +18:38:19 step 28780 loss=3.7391 lr=2.87e-04 tok=3.772B dt=4627 +18:39:35 step 28800 loss=3.8946 lr=2.87e-04 tok=3.775B dt=4703 +18:39:50 pushed ckpt_w1.pt +18:39:51 pushed log_w1.txt +18:40:06 pulled global r=15 4.643 -> 4.520 +18:41:29 step 28820 loss=3.6585 lr=2.87e-04 tok=3.777B dt=4817 +18:42:45 step 28840 loss=3.6516 lr=2.87e-04 tok=3.780B dt=4893 +18:44:00 step 28860 loss=3.6873 lr=2.87e-04 tok=3.783B dt=4968 +18:45:16 step 28880 loss=3.7658 lr=2.87e-04 tok=3.785B dt=5044 +18:46:32 step 28900 loss=3.8062 lr=2.87e-04 tok=3.788B dt=5120 +18:47:48 step 28920 loss=3.7763 lr=2.87e-04 tok=3.791B dt=5195 +18:49:03 step 28940 loss=3.7335 lr=2.87e-04 tok=3.793B dt=5271 +18:50:19 step 28960 loss=3.7836 lr=2.87e-04 tok=3.796B dt=5347 +18:51:34 step 28980 loss=3.8999 lr=2.87e-04 tok=3.798B dt=5422 +18:52:50 step 29000 loss=3.8456 lr=2.87e-04 tok=3.801B dt=5498 +18:54:06 step 29020 loss=3.8311 lr=2.87e-04 tok=3.804B dt=5574 +18:55:21 step 29040 loss=3.8479 lr=2.87e-04 tok=3.806B dt=5649 +18:56:37 step 29060 loss=3.7473 lr=2.87e-04 tok=3.809B dt=5725 +18:57:53 step 29080 loss=3.7842 lr=2.87e-04 tok=3.812B dt=5801 +18:59:08 step 29100 loss=3.7442 lr=2.87e-04 tok=3.814B dt=5876 +19:00:24 step 29120 loss=3.6926 lr=2.87e-04 tok=3.817B dt=5952 +19:01:40 step 29140 loss=3.7266 lr=2.87e-04 tok=3.819B dt=6028 +19:02:55 step 29160 loss=3.7889 lr=2.87e-04 tok=3.822B dt=6103 +19:04:11 step 29180 loss=3.7210 lr=2.87e-04 tok=3.825B dt=6179 +19:05:27 step 29200 loss=3.7715 lr=2.87e-04 tok=3.827B dt=6255 +19:06:42 step 29220 loss=3.7130 lr=2.86e-04 tok=3.830B dt=6330 +19:07:58 step 29240 loss=3.8428 lr=2.86e-04 tok=3.833B dt=6406 +19:09:13 step 29260 loss=3.8331 lr=2.86e-04 tok=3.835B dt=6481 +19:10:29 step 29280 loss=3.7768 lr=2.86e-04 tok=3.838B dt=6557 +19:11:44 step 29300 loss=3.7171 lr=2.86e-04 tok=3.840B dt=6632 +19:13:00 step 29320 loss=3.7483 lr=2.86e-04 tok=3.843B dt=6708 +19:14:15 step 29340 loss=3.6907 lr=2.86e-04 tok=3.846B dt=6783 +19:15:31 step 29360 loss=3.7384 lr=2.86e-04 tok=3.848B dt=6859 +19:16:47 step 29380 loss=3.7700 lr=2.86e-04 tok=3.851B dt=6935 +19:18:02 step 29400 loss=3.7735 lr=2.86e-04 tok=3.854B dt=7010 +19:18:14 pushed ckpt_w1.pt diff --git a/log_w10.txt b/log_w10.txt new file mode 100644 index 0000000000000000000000000000000000000000..9b95f67bd8f2d3c02b46e81198d8c006f2b3548f --- /dev/null +++ b/log_w10.txt @@ -0,0 +1,102 @@ +launch +10:37:54 name=w10 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 4/6 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +10:37:59 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +10:38:08 params 1.08B +10:38:10 resumed step=17400 seen=2.28B round=10 +10:40:14 start: accum=4, 131072 tokens per optimiser step +10:43:30 step 17420 loss=3.8838 lr=2.95e-04 tok=2.283B dt=196 +10:45:25 step 17440 loss=4.0326 lr=2.95e-04 tok=2.286B dt=310 +10:47:18 step 17460 loss=3.8902 lr=2.95e-04 tok=2.289B dt=424 +10:49:12 step 17480 loss=3.9665 lr=2.95e-04 tok=2.291B dt=538 +10:51:04 step 17500 loss=3.8694 lr=2.95e-04 tok=2.294B dt=650 +10:52:56 step 17520 loss=3.8952 lr=2.95e-04 tok=2.296B dt=761 +10:54:47 step 17540 loss=3.9310 lr=2.95e-04 tok=2.299B dt=873 +10:56:38 step 17560 loss=3.9364 lr=2.95e-04 tok=2.302B dt=983 +10:58:29 step 17580 loss=3.8491 lr=2.95e-04 tok=2.304B dt=1095 +11:00:22 step 17600 loss=3.9273 lr=2.95e-04 tok=2.307B dt=1208 +11:02:15 step 17620 loss=3.8759 lr=2.95e-04 tok=2.309B dt=1321 +11:04:06 step 17640 loss=3.8626 lr=2.95e-04 tok=2.312B dt=1432 +11:05:57 step 17660 loss=3.9262 lr=2.95e-04 tok=2.315B dt=1542 +11:07:48 step 17680 loss=3.8593 lr=2.95e-04 tok=2.317B dt=1654 +11:09:40 step 17700 loss=3.9216 lr=2.95e-04 tok=2.320B dt=1766 +11:11:32 step 17720 loss=3.9816 lr=2.95e-04 tok=2.323B dt=1878 +11:13:23 step 17740 loss=3.9599 lr=2.95e-04 tok=2.325B dt=1989 +11:15:15 step 17760 loss=3.9430 lr=2.95e-04 tok=2.328B dt=2101 +11:17:06 step 17780 loss=4.0343 lr=2.95e-04 tok=2.330B dt=2212 +11:18:58 step 17800 loss=3.9362 lr=2.95e-04 tok=2.333B dt=2324 +11:20:51 step 17820 loss=3.8757 lr=2.95e-04 tok=2.336B dt=2437 +11:22:44 step 17840 loss=3.8381 lr=2.95e-04 tok=2.338B dt=2550 +11:24:35 step 17860 loss=3.9232 lr=2.95e-04 tok=2.341B dt=2661 +11:26:26 step 17880 loss=3.8480 lr=2.95e-04 tok=2.344B dt=2772 +11:28:18 step 17900 loss=3.9330 lr=2.95e-04 tok=2.346B dt=2884 +11:30:10 step 17920 loss=3.8686 lr=2.95e-04 tok=2.349B dt=2996 +11:32:02 step 17940 loss=3.8345 lr=2.95e-04 tok=2.351B dt=3108 +11:33:54 step 17960 loss=3.9186 lr=2.95e-04 tok=2.354B dt=3220 +11:35:46 step 17980 loss=3.8002 lr=2.95e-04 tok=2.357B dt=3332 +11:37:37 step 18000 loss=3.9685 lr=2.95e-04 tok=2.359B dt=3443 +11:37:50 pushed ckpt_w10.pt +11:37:50 pushed log_w10.txt +11:39:51 step 18020 loss=3.9489 lr=2.95e-04 tok=2.362B dt=3576 +11:41:42 step 18040 loss=3.9219 lr=2.95e-04 tok=2.365B dt=3688 +11:43:34 step 18060 loss=3.8459 lr=2.95e-04 tok=2.367B dt=3800 +11:45:27 step 18080 loss=3.8357 lr=2.95e-04 tok=2.370B dt=3912 +11:47:18 step 18100 loss=3.8535 lr=2.95e-04 tok=2.372B dt=4024 +11:49:12 step 18120 loss=3.9259 lr=2.95e-04 tok=2.375B dt=4137 +11:51:03 step 18140 loss=4.0206 lr=2.95e-04 tok=2.378B dt=4249 +11:52:55 step 18160 loss=3.9758 lr=2.95e-04 tok=2.380B dt=4360 +11:54:46 step 18180 loss=3.9076 lr=2.95e-04 tok=2.383B dt=4472 +11:56:38 step 18200 loss=3.8919 lr=2.95e-04 tok=2.386B dt=4583 +11:58:30 step 18220 loss=3.9543 lr=2.95e-04 tok=2.388B dt=4696 +12:00:22 step 18240 loss=3.9028 lr=2.95e-04 tok=2.391B dt=4808 +12:02:14 step 18260 loss=3.8479 lr=2.95e-04 tok=2.393B dt=4920 +12:04:06 step 18280 loss=3.9559 lr=2.95e-04 tok=2.396B dt=5032 +12:05:57 step 18300 loss=3.8140 lr=2.95e-04 tok=2.399B dt=5143 +12:07:49 step 18320 loss=3.9288 lr=2.95e-04 tok=2.401B dt=5255 +12:09:42 step 18340 loss=3.8444 lr=2.95e-04 tok=2.404B dt=5367 +12:11:35 step 18360 loss=3.8596 lr=2.95e-04 tok=2.406B dt=5481 +12:13:27 step 18380 loss=4.0026 lr=2.95e-04 tok=2.409B dt=5592 +12:15:17 step 18400 loss=3.9402 lr=2.95e-04 tok=2.412B dt=5703 +12:17:08 step 18420 loss=3.9340 lr=2.95e-04 tok=2.414B dt=5814 +12:18:59 step 18440 loss=4.0256 lr=2.95e-04 tok=2.417B dt=5925 +12:20:51 step 18460 loss=3.9467 lr=2.95e-04 tok=2.420B dt=6037 +12:22:44 step 18480 loss=3.7924 lr=2.95e-04 tok=2.422B dt=6150 +12:24:36 step 18500 loss=3.8554 lr=2.95e-04 tok=2.425B dt=6262 +12:26:27 step 18520 loss=3.8225 lr=2.95e-04 tok=2.427B dt=6373 +12:28:17 step 18540 loss=3.9394 lr=2.95e-04 tok=2.430B dt=6483 +12:30:08 step 18560 loss=3.9006 lr=2.95e-04 tok=2.433B dt=6594 +12:32:00 step 18580 loss=3.8608 lr=2.95e-04 tok=2.435B dt=6706 +12:33:53 step 18600 loss=3.9746 lr=2.95e-04 tok=2.438B dt=6819 +12:34:06 pushed ckpt_w10.pt +12:34:06 pushed log_w10.txt +12:34:17 pulled global r=11 4.741 -> 4.588 +12:36:12 step 18620 loss=3.7107 lr=2.95e-04 tok=2.441B dt=6958 +12:38:05 step 18640 loss=3.7124 lr=2.95e-04 tok=2.443B dt=7070 +12:39:55 step 18660 loss=3.7599 lr=2.95e-04 tok=2.446B dt=7181 +12:41:46 step 18680 loss=3.8498 lr=2.95e-04 tok=2.448B dt=7292 +12:43:38 step 18700 loss=3.9352 lr=2.95e-04 tok=2.451B dt=7404 +12:45:30 step 18720 loss=3.8853 lr=2.95e-04 tok=2.454B dt=7516 +12:47:21 step 18740 loss=3.8042 lr=2.94e-04 tok=2.456B dt=7627 +12:49:13 step 18760 loss=3.9172 lr=2.94e-04 tok=2.459B dt=7739 +12:51:04 step 18780 loss=3.9802 lr=2.94e-04 tok=2.462B dt=7850 +12:52:56 step 18800 loss=3.9509 lr=2.94e-04 tok=2.464B dt=7962 +12:54:48 step 18820 loss=3.9035 lr=2.94e-04 tok=2.467B dt=8074 +12:56:40 step 18840 loss=3.9187 lr=2.94e-04 tok=2.469B dt=8186 +12:58:31 step 18860 loss=3.8453 lr=2.94e-04 tok=2.472B dt=8297 +13:00:22 step 18880 loss=3.8924 lr=2.94e-04 tok=2.475B dt=8408 +13:02:14 step 18900 loss=3.8872 lr=2.94e-04 tok=2.477B dt=8519 +13:04:06 step 18920 loss=3.7970 lr=2.94e-04 tok=2.480B dt=8631 +13:05:57 step 18940 loss=3.8165 lr=2.94e-04 tok=2.483B dt=8743 +13:07:50 step 18960 loss=3.9126 lr=2.94e-04 tok=2.485B dt=8856 +13:09:43 step 18980 loss=3.8026 lr=2.94e-04 tok=2.488B dt=8969 +13:11:33 step 19000 loss=3.8558 lr=2.94e-04 tok=2.490B dt=9079 +13:13:24 step 19020 loss=3.8174 lr=2.94e-04 tok=2.493B dt=9190 +13:15:16 step 19040 loss=3.9621 lr=2.94e-04 tok=2.496B dt=9302 +13:17:07 step 19060 loss=3.9502 lr=2.94e-04 tok=2.498B dt=9413 +13:18:59 step 19080 loss=3.8991 lr=2.94e-04 tok=2.501B dt=9525 +13:20:51 step 19100 loss=3.7775 lr=2.94e-04 tok=2.503B dt=9637 +13:22:42 step 19120 loss=3.8743 lr=2.94e-04 tok=2.506B dt=9748 +13:24:33 step 19140 loss=3.8067 lr=2.94e-04 tok=2.509B dt=9859 +13:26:25 step 19160 loss=3.8757 lr=2.94e-04 tok=2.511B dt=9971 +13:28:16 step 19180 loss=3.9067 lr=2.94e-04 tok=2.514B dt=10082 +13:30:08 step 19200 loss=3.8632 lr=2.94e-04 tok=2.517B dt=10194 +13:30:20 pushed ckpt_w10.pt diff --git a/log_w100.txt b/log_w100.txt new file mode 100644 index 0000000000000000000000000000000000000000..effcb85df8da72e1c455b304847ed2cc24c53e41 --- /dev/null +++ b/log_w100.txt @@ -0,0 +1,197 @@ +launch +09:46:43 name=w100 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 47/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +09:46:50 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +09:46:50 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +09:46:59 params 1.08B +09:47:01 resumed step=117150 seen=15.39B round=390 +09:47:01 stream salt=117150 +09:48:50 blocks compiled +09:48:50 start: accum=4, 131072 tokens per optimiser step +09:52:00 step 117151 loss=3.0346 lr=6.50e-05 tok=15.389B dt=190 wait=144 +09:52:03 step 117152 loss=3.0420 lr=6.50e-05 tok=15.389B dt=193 wait=145 +09:52:31 step 117160 loss=3.4256 lr=6.50e-05 tok=15.390B dt=221 wait=145 +09:53:45 step 117180 loss=3.4099 lr=6.50e-05 tok=15.393B dt=294 wait=146 +09:54:48 step 117200 loss=3.3788 lr=6.50e-05 tok=15.396B dt=358 wait=146 +09:55:52 step 117220 loss=3.3574 lr=6.50e-05 tok=15.398B dt=422 wait=146 +09:56:56 step 117240 loss=3.4285 lr=6.50e-05 tok=15.401B dt=486 wait=147 +09:58:00 step 117260 loss=3.5839 lr=6.49e-05 tok=15.404B dt=550 wait=147 +09:59:04 step 117280 loss=3.3701 lr=6.49e-05 tok=15.406B dt=614 wait=147 +10:00:08 step 117300 loss=3.4265 lr=6.49e-05 tok=15.409B dt=678 wait=148 +10:00:22 pushed ckpt_w100.pt,log_w100.txt +10:01:23 waited 61s for global r=391 +10:01:45 global r=391 load 4.273 -> 4.286 +10:02:52 step 117320 loss=3.3443 lr=6.49e-05 tok=15.411B dt=842 wait=148 +10:03:56 step 117340 loss=3.3786 lr=6.49e-05 tok=15.414B dt=906 wait=148 +10:05:00 step 117360 loss=3.3333 lr=6.48e-05 tok=15.417B dt=970 wait=149 +10:06:04 step 117380 loss=3.3333 lr=6.48e-05 tok=15.419B dt=1033 wait=149 +10:07:07 step 117400 loss=3.3266 lr=6.48e-05 tok=15.422B dt=1097 wait=149 +10:08:11 step 117420 loss=3.4277 lr=6.48e-05 tok=15.425B dt=1161 wait=150 +10:09:15 step 117440 loss=3.3266 lr=6.48e-05 tok=15.427B dt=1224 wait=150 +10:10:06 pushed ckpt_w100.pt,log_w100.txt +10:11:07 waited 61s for global r=392 +10:11:16 global r=392 load 4.273 -> 4.292 +10:11:50 step 117460 loss=3.4107 lr=6.47e-05 tok=15.430B dt=1380 wait=150 +10:12:54 step 117480 loss=3.3313 lr=6.47e-05 tok=15.432B dt=1444 wait=151 +10:13:58 step 117500 loss=3.2987 lr=6.47e-05 tok=15.435B dt=1508 wait=151 +10:15:02 step 117520 loss=3.4118 lr=6.47e-05 tok=15.438B dt=1571 wait=151 +10:16:05 step 117540 loss=3.3125 lr=6.46e-05 tok=15.440B dt=1635 wait=152 +10:17:14 step 117560 loss=3.3976 lr=6.46e-05 tok=15.443B dt=1704 wait=152 +10:18:18 step 117580 loss=3.2860 lr=6.46e-05 tok=15.446B dt=1768 wait=152 +10:19:22 step 117600 loss=3.3752 lr=6.46e-05 tok=15.448B dt=1832 wait=153 +10:19:44 pushed ckpt_w100.pt,log_w100.txt +10:20:30 waited 46s for global r=393 +10:20:39 global r=393 load 4.309 -> 4.289 +10:21:45 step 117620 loss=3.3586 lr=6.46e-05 tok=15.451B dt=1974 wait=153 +10:22:48 step 117640 loss=3.4181 lr=6.45e-05 tok=15.453B dt=2038 wait=154 +10:23:52 step 117660 loss=3.4038 lr=6.45e-05 tok=15.456B dt=2102 wait=154 +10:24:56 step 117680 loss=3.3649 lr=6.45e-05 tok=15.459B dt=2165 wait=154 +10:25:59 step 117700 loss=3.3575 lr=6.45e-05 tok=15.461B dt=2229 wait=154 +10:27:03 step 117720 loss=3.4628 lr=6.45e-05 tok=15.464B dt=2293 wait=155 +10:28:07 step 117740 loss=3.2693 lr=6.44e-05 tok=15.466B dt=2357 wait=155 +10:28:53 pushed ckpt_w100.pt,log_w100.txt +10:30:09 waited 76s for global r=394 +10:30:16 global r=394 load 4.298 -> 4.273 +10:30:49 step 117760 loss=3.4204 lr=6.44e-05 tok=15.469B dt=2519 wait=155 +10:31:53 step 117780 loss=3.3480 lr=6.44e-05 tok=15.472B dt=2583 wait=156 +10:32:56 step 117800 loss=3.3225 lr=6.44e-05 tok=15.474B dt=2646 wait=156 +10:34:00 step 117820 loss=3.3635 lr=6.44e-05 tok=15.477B dt=2710 wait=156 +10:35:04 step 117840 loss=3.2970 lr=6.43e-05 tok=15.480B dt=2774 wait=157 +10:36:08 step 117860 loss=3.3795 lr=6.43e-05 tok=15.482B dt=2838 wait=157 +10:37:11 step 117880 loss=3.3401 lr=6.43e-05 tok=15.485B dt=2901 wait=157 +10:38:15 step 117900 loss=3.3546 lr=6.43e-05 tok=15.487B dt=2965 wait=158 +10:38:40 pushed ckpt_w100.pt,log_w100.txt +10:39:40 waited 61s for global r=395 +10:39:49 global r=395 load 4.292 -> 4.278 +10:40:54 step 117920 loss=3.3332 lr=6.43e-05 tok=15.490B dt=3124 wait=158 +10:41:58 step 117940 loss=3.3670 lr=6.42e-05 tok=15.493B dt=3187 wait=158 +10:43:01 step 117960 loss=3.3600 lr=6.42e-05 tok=15.495B dt=3251 wait=159 +10:44:05 step 117980 loss=3.3787 lr=6.42e-05 tok=15.498B dt=3315 wait=159 +10:45:09 step 118000 loss=3.3669 lr=6.42e-05 tok=15.501B dt=3379 wait=159 +10:46:13 step 118020 loss=3.4170 lr=6.42e-05 tok=15.503B dt=3443 wait=160 +10:47:16 step 118040 loss=3.2652 lr=6.41e-05 tok=15.506B dt=3506 wait=160 +10:48:02 pushed ckpt_w100.pt,log_w100.txt +10:49:11 waited 69s for global r=396 +10:49:17 global r=396 load 4.320 -> 4.294 +10:49:50 step 118060 loss=3.4250 lr=6.41e-05 tok=15.508B dt=3660 wait=160 +10:50:54 step 118080 loss=3.3071 lr=6.41e-05 tok=15.511B dt=3724 wait=161 +10:51:57 step 118100 loss=3.3170 lr=6.41e-05 tok=15.514B dt=3787 wait=161 +10:53:01 step 118120 loss=3.4187 lr=6.41e-05 tok=15.516B dt=3851 wait=161 +10:54:05 step 118140 loss=3.3311 lr=6.40e-05 tok=15.519B dt=3914 wait=162 +10:55:08 step 118160 loss=3.3754 lr=6.40e-05 tok=15.522B dt=3978 wait=162 +10:56:12 step 118180 loss=3.3195 lr=6.40e-05 tok=15.524B dt=4041 wait=162 +10:57:15 step 118200 loss=3.3663 lr=6.40e-05 tok=15.527B dt=4105 wait=163 +10:57:32 pushed ckpt_w100.pt,log_w100.txt +10:58:35 waited 63s for global r=397 +10:58:44 global r=397 load 4.364 -> 4.299 +10:59:49 step 118220 loss=3.3130 lr=6.40e-05 tok=15.529B dt=4259 wait=163 +11:00:53 step 118240 loss=3.3437 lr=6.39e-05 tok=15.532B dt=4323 wait=163 +11:01:57 step 118260 loss=3.3620 lr=6.39e-05 tok=15.535B dt=4387 wait=164 +11:03:00 step 118280 loss=3.3222 lr=6.39e-05 tok=15.537B dt=4450 wait=164 +11:04:04 step 118300 loss=3.3214 lr=6.39e-05 tok=15.540B dt=4514 wait=164 +11:05:08 step 118320 loss=3.4233 lr=6.39e-05 tok=15.543B dt=4578 wait=165 +11:06:11 step 118340 loss=3.3222 lr=6.38e-05 tok=15.545B dt=4641 wait=165 +11:07:04 pushed ckpt_w100.pt,log_w100.txt +11:08:05 waited 61s for global r=398 +11:08:20 global r=398 load 4.313 -> 4.295 +11:08:54 step 118360 loss=3.4353 lr=6.38e-05 tok=15.548B dt=4804 wait=165 +11:09:58 step 118380 loss=3.2644 lr=6.38e-05 tok=15.550B dt=4868 wait=166 +11:11:01 step 118400 loss=3.3153 lr=6.38e-05 tok=15.553B dt=4931 wait=166 +11:12:05 step 118420 loss=3.4200 lr=6.38e-05 tok=15.556B dt=4995 wait=166 +11:13:09 step 118440 loss=3.3624 lr=6.37e-05 tok=15.558B dt=5059 wait=167 +11:14:13 step 118460 loss=3.4197 lr=6.37e-05 tok=15.561B dt=5123 wait=167 +11:15:16 step 118480 loss=3.2930 lr=6.37e-05 tok=15.563B dt=5186 wait=167 +11:16:20 step 118500 loss=3.3163 lr=6.37e-05 tok=15.566B dt=5250 wait=168 +11:16:39 pushed ckpt_w100.pt,log_w100.txt +11:17:39 waited 61s for global r=399 +11:17:50 global r=399 load 4.284 -> 4.292 +11:18:55 step 118520 loss=3.3081 lr=6.36e-05 tok=15.569B dt=5405 wait=168 +11:19:58 step 118540 loss=3.3422 lr=6.36e-05 tok=15.571B dt=5468 wait=168 +11:21:02 step 118560 loss=3.3675 lr=6.36e-05 tok=15.574B dt=5532 wait=168 +11:22:06 step 118580 loss=3.2850 lr=6.36e-05 tok=15.577B dt=5595 wait=169 +11:23:10 step 118600 loss=3.3137 lr=6.36e-05 tok=15.579B dt=5660 wait=169 +11:24:14 step 118620 loss=3.4353 lr=6.35e-05 tok=15.582B dt=5723 wait=169 +11:25:17 step 118640 loss=3.3014 lr=6.35e-05 tok=15.584B dt=5787 wait=170 +11:26:06 pushed ckpt_w100.pt,log_w100.txt +11:28:37 waited 152s for global r=400 +11:28:46 global r=400 load 4.304 -> 4.285 +11:29:19 step 118660 loss=3.4279 lr=6.35e-05 tok=15.587B dt=6029 wait=170 +11:30:23 step 118680 loss=3.3346 lr=6.35e-05 tok=15.590B dt=6093 wait=171 +11:31:27 step 118700 loss=3.3399 lr=6.35e-05 tok=15.592B dt=6157 wait=171 +11:32:31 step 118720 loss=3.4192 lr=6.34e-05 tok=15.595B dt=6221 wait=171 +11:33:40 step 118740 loss=3.3021 lr=6.34e-05 tok=15.598B dt=6290 wait=172 +11:34:43 step 118760 loss=3.4213 lr=6.34e-05 tok=15.600B dt=6353 wait=172 +11:35:47 step 118780 loss=3.3433 lr=6.34e-05 tok=15.603B dt=6417 wait=172 +11:36:51 step 118800 loss=3.3648 lr=6.34e-05 tok=15.605B dt=6481 wait=173 +11:37:08 pushed ckpt_w100.pt,log_w100.txt +11:38:54 waited 106s for global r=401 +11:39:01 global r=401 load 4.325 -> 4.288 +11:40:06 step 118820 loss=3.3723 lr=6.33e-05 tok=15.608B dt=6676 wait=173 +11:41:10 step 118840 loss=3.3967 lr=6.33e-05 tok=15.611B dt=6739 wait=173 +11:42:14 step 118860 loss=3.2918 lr=6.33e-05 tok=15.613B dt=6804 wait=174 +11:43:17 step 118880 loss=3.2916 lr=6.33e-05 tok=15.616B dt=6867 wait=174 +11:44:21 step 118900 loss=3.3856 lr=6.33e-05 tok=15.619B dt=6931 wait=174 +11:45:25 step 118920 loss=3.4231 lr=6.32e-05 tok=15.621B dt=6995 wait=175 +11:46:29 step 118940 loss=3.2861 lr=6.32e-05 tok=15.624B dt=7059 wait=175 +11:47:15 pushed ckpt_w100.pt,log_w100.txt +11:48:18 waited 62s for global r=402 +11:48:26 global r=402 load 4.291 -> 4.284 +11:48:59 step 118960 loss=3.4417 lr=6.32e-05 tok=15.626B dt=7209 wait=175 +11:50:02 step 118980 loss=3.3012 lr=6.32e-05 tok=15.629B dt=7272 wait=176 +11:51:06 step 119000 loss=3.3254 lr=6.32e-05 tok=15.632B dt=7336 wait=176 +11:52:10 step 119020 loss=3.4338 lr=6.31e-05 tok=15.634B dt=7400 wait=176 +11:53:13 step 119040 loss=3.3010 lr=6.31e-05 tok=15.637B dt=7463 wait=177 +11:54:17 step 119060 loss=3.3773 lr=6.31e-05 tok=15.640B dt=7527 wait=177 +11:55:21 step 119080 loss=3.3470 lr=6.31e-05 tok=15.642B dt=7591 wait=177 +11:56:26 step 119100 loss=3.4141 lr=6.31e-05 tok=15.645B dt=7656 wait=178 +11:56:42 pushed ckpt_w100.pt,log_w100.txt +11:57:43 waited 61s for global r=403 +11:57:51 global r=403 load 4.297 -> 4.283 +11:58:56 step 119120 loss=3.3516 lr=6.30e-05 tok=15.647B dt=7806 wait=178 +12:00:00 step 119140 loss=3.3902 lr=6.30e-05 tok=15.650B dt=7870 wait=178 +12:01:04 step 119160 loss=3.3569 lr=6.30e-05 tok=15.653B dt=7934 wait=179 +12:02:08 step 119180 loss=3.2397 lr=6.30e-05 tok=15.655B dt=7997 wait=179 +12:03:11 step 119200 loss=3.3366 lr=6.30e-05 tok=15.658B dt=8061 wait=179 +12:04:16 step 119220 loss=3.2836 lr=6.29e-05 tok=15.660B dt=8126 wait=180 +12:05:19 step 119240 loss=3.3045 lr=6.29e-05 tok=15.663B dt=8189 wait=180 +12:06:09 pushed ckpt_w100.pt,log_w100.txt +12:07:10 waited 61s for global r=404 +12:07:18 global r=404 load 4.290 -> 4.297 +12:07:52 step 119260 loss=3.4215 lr=6.29e-05 tok=15.666B dt=8342 wait=180 +12:08:56 step 119280 loss=3.2921 lr=6.29e-05 tok=15.668B dt=8406 wait=181 +12:10:00 step 119300 loss=3.3373 lr=6.29e-05 tok=15.671B dt=8470 wait=181 +12:11:04 step 119320 loss=3.2733 lr=6.28e-05 tok=15.674B dt=8534 wait=182 +12:12:08 step 119340 loss=3.2496 lr=6.28e-05 tok=15.676B dt=8598 wait=182 +12:13:12 step 119360 loss=3.4388 lr=6.28e-05 tok=15.679B dt=8662 wait=182 +12:14:16 step 119380 loss=3.3066 lr=6.28e-05 tok=15.681B dt=8726 wait=183 +12:15:20 step 119400 loss=3.3703 lr=6.28e-05 tok=15.684B dt=8789 wait=183 +12:15:45 pushed ckpt_w100.pt,log_w100.txt +12:16:31 waited 46s for global r=405 +12:16:39 global r=405 load 4.288 -> 4.283 +12:17:45 step 119420 loss=3.4039 lr=6.27e-05 tok=15.687B dt=8935 wait=183 +12:18:49 step 119440 loss=3.3226 lr=6.27e-05 tok=15.689B dt=8999 wait=184 +12:19:52 step 119460 loss=3.3891 lr=6.27e-05 tok=15.692B dt=9062 wait=184 +12:20:56 step 119480 loss=3.3212 lr=6.27e-05 tok=15.695B dt=9126 wait=184 +12:21:59 step 119500 loss=3.4128 lr=6.27e-05 tok=15.697B dt=9189 wait=184 +12:23:03 step 119520 loss=3.3691 lr=6.26e-05 tok=15.700B dt=9253 wait=185 +12:24:07 step 119540 loss=3.2832 lr=6.26e-05 tok=15.702B dt=9317 wait=185 +12:24:52 pushed ckpt_w100.pt,log_w100.txt +12:25:58 waited 66s for global r=406 +12:26:08 global r=406 load 4.285 -> 4.282 +12:26:42 step 119560 loss=3.2623 lr=6.26e-05 tok=15.705B dt=9471 wait=185 +12:27:45 step 119580 loss=3.2836 lr=6.26e-05 tok=15.708B dt=9535 wait=186 +12:28:48 step 119600 loss=3.3217 lr=6.26e-05 tok=15.710B dt=9598 wait=186 +12:29:52 step 119620 loss=3.3743 lr=6.25e-05 tok=15.713B dt=9662 wait=186 +12:30:55 step 119640 loss=3.2983 lr=6.25e-05 tok=15.716B dt=9725 wait=187 +12:31:59 step 119660 loss=3.3800 lr=6.25e-05 tok=15.718B dt=9789 wait=187 +12:33:04 step 119680 loss=3.2634 lr=6.25e-05 tok=15.721B dt=9854 wait=187 +12:34:08 step 119700 loss=3.3602 lr=6.25e-05 tok=15.723B dt=9918 wait=188 +12:34:20 pushed ckpt_w100.pt,log_w100.txt +12:35:22 waited 62s for global r=407 +12:35:35 global r=407 load 4.322 -> 4.275 +12:36:40 step 119720 loss=3.3669 lr=6.24e-05 tok=15.726B dt=10070 wait=188 +12:37:44 step 119740 loss=3.3451 lr=6.24e-05 tok=15.729B dt=10134 wait=188 +12:38:48 step 119760 loss=3.3575 lr=6.24e-05 tok=15.731B dt=10198 wait=189 +12:39:52 step 119780 loss=3.3386 lr=6.24e-05 tok=15.734B dt=10262 wait=189 +12:40:56 step 119800 loss=3.3595 lr=6.23e-05 tok=15.737B dt=10326 wait=189 +12:42:00 step 119820 loss=3.4059 lr=6.23e-05 tok=15.739B dt=10390 wait=190 +12:43:03 step 119840 loss=3.3053 lr=6.23e-05 tok=15.742B dt=10453 wait=190 diff --git a/log_w101.txt b/log_w101.txt new file mode 100644 index 0000000000000000000000000000000000000000..28073963566b0cfda74ab10280528a823798a9db --- /dev/null +++ b/log_w101.txt @@ -0,0 +1,197 @@ +launch +09:46:53 name=w101 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 48/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +09:46:58 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +09:46:59 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +09:47:08 params 1.08B +09:47:10 resumed step=117150 seen=15.39B round=390 +09:47:10 stream salt=117150 +09:48:57 blocks compiled +09:48:57 start: accum=4, 131072 tokens per optimiser step +09:52:11 step 117151 loss=3.3095 lr=6.50e-05 tok=15.389B dt=194 wait=146 +09:52:14 step 117152 loss=3.0934 lr=6.50e-05 tok=15.389B dt=197 wait=146 +09:52:42 step 117160 loss=3.4089 lr=6.50e-05 tok=15.390B dt=225 wait=147 +09:53:54 step 117180 loss=3.4035 lr=6.50e-05 tok=15.393B dt=297 wait=147 +09:54:58 step 117200 loss=3.4338 lr=6.50e-05 tok=15.396B dt=361 wait=148 +09:56:01 step 117220 loss=3.3946 lr=6.50e-05 tok=15.398B dt=424 wait=148 +09:57:06 step 117240 loss=3.4259 lr=6.50e-05 tok=15.401B dt=489 wait=148 +09:58:10 step 117260 loss=3.5868 lr=6.49e-05 tok=15.404B dt=553 wait=149 +09:59:14 step 117280 loss=3.3845 lr=6.49e-05 tok=15.406B dt=617 wait=149 +10:00:18 step 117300 loss=3.4475 lr=6.49e-05 tok=15.409B dt=681 wait=149 +10:00:32 pushed ckpt_w101.pt,log_w101.txt +10:01:33 waited 60s for global r=391 +10:01:46 global r=391 load 4.288 -> 4.286 +10:02:58 step 117320 loss=3.4117 lr=6.49e-05 tok=15.411B dt=841 wait=150 +10:04:17 step 117340 loss=3.4179 lr=6.49e-05 tok=15.414B dt=920 wait=150 +10:05:21 step 117360 loss=3.3515 lr=6.48e-05 tok=15.417B dt=984 wait=150 +10:06:25 step 117380 loss=3.3526 lr=6.48e-05 tok=15.419B dt=1048 wait=151 +10:07:29 step 117400 loss=3.3358 lr=6.48e-05 tok=15.422B dt=1112 wait=151 +10:08:33 step 117420 loss=3.4233 lr=6.48e-05 tok=15.425B dt=1176 wait=151 +10:09:37 step 117440 loss=3.3292 lr=6.48e-05 tok=15.427B dt=1240 wait=152 +10:10:21 pushed ckpt_w101.pt,log_w101.txt +10:11:07 waited 45s for global r=392 +10:11:18 global r=392 load 4.315 -> 4.292 +10:11:52 step 117460 loss=3.4002 lr=6.47e-05 tok=15.430B dt=1375 wait=152 +10:13:04 step 117480 loss=3.3265 lr=6.47e-05 tok=15.432B dt=1447 wait=152 +10:14:20 step 117500 loss=3.3769 lr=6.47e-05 tok=15.435B dt=1523 wait=153 +10:15:23 step 117520 loss=3.4253 lr=6.47e-05 tok=15.438B dt=1586 wait=153 +10:16:27 step 117540 loss=3.2970 lr=6.46e-05 tok=15.440B dt=1650 wait=153 +10:17:31 step 117560 loss=3.4153 lr=6.46e-05 tok=15.443B dt=1714 wait=154 +10:18:35 step 117580 loss=3.3247 lr=6.46e-05 tok=15.446B dt=1778 wait=154 +10:19:39 step 117600 loss=3.3276 lr=6.46e-05 tok=15.448B dt=1842 wait=154 +10:19:53 pushed ckpt_w101.pt,log_w101.txt +10:20:38 waited 45s for global r=393 +10:20:51 global r=393 load 4.289 -> 4.289 +10:21:57 step 117620 loss=3.3249 lr=6.46e-05 tok=15.451B dt=1980 wait=155 +10:23:19 step 117640 loss=3.4088 lr=6.45e-05 tok=15.453B dt=2062 wait=155 +10:24:24 step 117660 loss=3.4266 lr=6.45e-05 tok=15.456B dt=2127 wait=155 +10:25:28 step 117680 loss=3.3296 lr=6.45e-05 tok=15.459B dt=2191 wait=156 +10:26:32 step 117700 loss=3.3521 lr=6.45e-05 tok=15.461B dt=2255 wait=156 +10:27:35 step 117720 loss=3.4530 lr=6.45e-05 tok=15.464B dt=2318 wait=156 +10:28:39 step 117740 loss=3.3174 lr=6.44e-05 tok=15.466B dt=2382 wait=157 +10:29:23 pushed ckpt_w101.pt,log_w101.txt +10:30:09 waited 45s for global r=394 +10:30:16 global r=394 load 4.255 -> 4.273 +10:30:50 step 117760 loss=3.4021 lr=6.44e-05 tok=15.469B dt=2513 wait=157 +10:32:00 step 117780 loss=3.3592 lr=6.44e-05 tok=15.472B dt=2583 wait=158 +10:33:17 step 117800 loss=3.3500 lr=6.44e-05 tok=15.474B dt=2660 wait=158 +10:34:21 step 117820 loss=3.4316 lr=6.44e-05 tok=15.477B dt=2724 wait=158 +10:35:25 step 117840 loss=3.3071 lr=6.43e-05 tok=15.480B dt=2788 wait=159 +10:36:29 step 117860 loss=3.4521 lr=6.43e-05 tok=15.482B dt=2852 wait=159 +10:37:32 step 117880 loss=3.3882 lr=6.43e-05 tok=15.485B dt=2915 wait=159 +10:38:36 step 117900 loss=3.4108 lr=6.43e-05 tok=15.487B dt=2979 wait=159 +10:38:50 pushed ckpt_w101.pt,log_w101.txt +10:39:36 waited 45s for global r=395 +10:39:44 global r=395 load 4.278 -> 4.278 +10:40:50 step 117920 loss=3.3490 lr=6.43e-05 tok=15.490B dt=3113 wait=160 +10:42:01 step 117940 loss=3.3708 lr=6.42e-05 tok=15.493B dt=3184 wait=160 +10:43:17 step 117960 loss=3.4020 lr=6.42e-05 tok=15.495B dt=3260 wait=161 +10:44:21 step 117980 loss=3.2540 lr=6.42e-05 tok=15.498B dt=3324 wait=161 +10:45:26 step 118000 loss=3.4136 lr=6.42e-05 tok=15.501B dt=3389 wait=161 +10:46:29 step 118020 loss=3.4122 lr=6.42e-05 tok=15.503B dt=3452 wait=162 +10:47:33 step 118040 loss=3.3720 lr=6.41e-05 tok=15.506B dt=3516 wait=162 +10:48:21 pushed ckpt_w101.pt,log_w101.txt +10:49:07 waited 45s for global r=396 +10:49:14 global r=396 load 4.303 -> 4.294 +10:49:47 step 118060 loss=3.3752 lr=6.41e-05 tok=15.508B dt=3650 wait=162 +10:50:51 step 118080 loss=3.3748 lr=6.41e-05 tok=15.511B dt=3714 wait=163 +10:52:14 step 118100 loss=3.3235 lr=6.41e-05 tok=15.514B dt=3797 wait=163 +10:53:19 step 118120 loss=3.4912 lr=6.41e-05 tok=15.516B dt=3862 wait=163 +10:54:22 step 118140 loss=3.3379 lr=6.40e-05 tok=15.519B dt=3925 wait=164 +10:55:26 step 118160 loss=3.4394 lr=6.40e-05 tok=15.522B dt=3989 wait=164 +10:56:30 step 118180 loss=3.3208 lr=6.40e-05 tok=15.524B dt=4053 wait=164 +10:57:35 step 118200 loss=3.3279 lr=6.40e-05 tok=15.527B dt=4118 wait=165 +10:57:47 pushed ckpt_w101.pt,log_w101.txt +10:58:32 waited 45s for global r=397 +10:58:38 global r=397 load 4.290 -> 4.299 +10:59:43 step 118220 loss=3.2977 lr=6.40e-05 tok=15.529B dt=4246 wait=165 +11:00:54 step 118240 loss=3.4364 lr=6.39e-05 tok=15.532B dt=4317 wait=165 +11:02:11 step 118260 loss=3.3806 lr=6.39e-05 tok=15.535B dt=4394 wait=166 +11:03:14 step 118280 loss=3.3707 lr=6.39e-05 tok=15.537B dt=4457 wait=166 +11:04:18 step 118300 loss=3.3542 lr=6.39e-05 tok=15.540B dt=4521 wait=166 +11:05:21 step 118320 loss=3.4152 lr=6.39e-05 tok=15.543B dt=4584 wait=167 +11:06:25 step 118340 loss=3.3087 lr=6.38e-05 tok=15.545B dt=4648 wait=167 +11:07:09 pushed ckpt_w101.pt,log_w101.txt +11:08:09 waited 60s for global r=398 +11:08:15 global r=398 load 4.308 -> 4.295 +11:08:55 step 118360 loss=3.4412 lr=6.38e-05 tok=15.548B dt=4798 wait=167 +11:10:11 step 118380 loss=3.2901 lr=6.38e-05 tok=15.550B dt=4874 wait=168 +11:11:15 step 118400 loss=3.2712 lr=6.38e-05 tok=15.553B dt=4938 wait=168 +11:12:19 step 118420 loss=3.3839 lr=6.38e-05 tok=15.556B dt=5002 wait=168 +11:13:22 step 118440 loss=3.2470 lr=6.37e-05 tok=15.558B dt=5065 wait=169 +11:14:26 step 118460 loss=3.4140 lr=6.37e-05 tok=15.561B dt=5129 wait=169 +11:15:30 step 118480 loss=3.2982 lr=6.37e-05 tok=15.563B dt=5193 wait=169 +11:16:34 step 118500 loss=3.3759 lr=6.37e-05 tok=15.566B dt=5257 wait=170 +11:16:46 pushed ckpt_w101.pt,log_w101.txt +11:17:31 waited 45s for global r=399 +11:17:37 global r=399 load 4.302 -> 4.292 +11:18:58 step 118520 loss=3.3423 lr=6.36e-05 tok=15.569B dt=5401 wait=170 +11:20:06 step 118540 loss=3.4098 lr=6.36e-05 tok=15.571B dt=5469 wait=170 +11:21:09 step 118560 loss=3.3721 lr=6.36e-05 tok=15.574B dt=5532 wait=171 +11:22:13 step 118580 loss=3.3249 lr=6.36e-05 tok=15.577B dt=5596 wait=171 +11:23:17 step 118600 loss=3.3019 lr=6.36e-05 tok=15.579B dt=5660 wait=171 +11:24:21 step 118620 loss=3.4100 lr=6.35e-05 tok=15.582B dt=5724 wait=172 +11:25:25 step 118640 loss=3.3053 lr=6.35e-05 tok=15.584B dt=5788 wait=172 +11:26:09 pushed ckpt_w101.pt,log_w101.txt +11:28:40 waited 151s for global r=400 +11:28:50 global r=400 load 4.281 -> 4.285 +11:29:23 step 118660 loss=3.4705 lr=6.35e-05 tok=15.587B dt=6026 wait=172 +11:30:46 step 118680 loss=3.3180 lr=6.35e-05 tok=15.590B dt=6109 wait=173 +11:31:50 step 118700 loss=3.3568 lr=6.35e-05 tok=15.592B dt=6173 wait=173 +11:32:53 step 118720 loss=3.4260 lr=6.34e-05 tok=15.595B dt=6236 wait=173 +11:33:57 step 118740 loss=3.3443 lr=6.34e-05 tok=15.598B dt=6300 wait=174 +11:35:01 step 118760 loss=3.4415 lr=6.34e-05 tok=15.600B dt=6364 wait=174 +11:36:05 step 118780 loss=3.3596 lr=6.34e-05 tok=15.603B dt=6428 wait=174 +11:37:09 step 118800 loss=3.3625 lr=6.34e-05 tok=15.605B dt=6492 wait=175 +11:37:21 pushed ckpt_w101.pt,log_w101.txt +11:38:52 waited 91s for global r=401 +11:38:58 global r=401 load 4.274 -> 4.288 +11:40:24 step 118820 loss=3.3430 lr=6.33e-05 tok=15.608B dt=6687 wait=175 +11:41:27 step 118840 loss=3.3891 lr=6.33e-05 tok=15.611B dt=6750 wait=176 +11:42:31 step 118860 loss=3.4411 lr=6.33e-05 tok=15.613B dt=6814 wait=176 +11:43:35 step 118880 loss=3.3385 lr=6.33e-05 tok=15.616B dt=6878 wait=176 +11:44:39 step 118900 loss=3.3198 lr=6.33e-05 tok=15.619B dt=6942 wait=177 +11:45:43 step 118920 loss=3.4216 lr=6.32e-05 tok=15.621B dt=7006 wait=177 +11:46:46 step 118940 loss=3.2866 lr=6.32e-05 tok=15.624B dt=7069 wait=177 +11:47:31 pushed ckpt_w101.pt,log_w101.txt +11:48:16 waited 45s for global r=402 +11:48:24 global r=402 load 4.294 -> 4.284 +11:49:05 step 118960 loss=3.4375 lr=6.32e-05 tok=15.626B dt=7208 wait=178 +11:50:20 step 118980 loss=3.3128 lr=6.32e-05 tok=15.629B dt=7283 wait=178 +11:51:24 step 119000 loss=3.3299 lr=6.32e-05 tok=15.632B dt=7347 wait=178 +11:52:28 step 119020 loss=3.4101 lr=6.31e-05 tok=15.634B dt=7411 wait=179 +11:53:31 step 119040 loss=3.3777 lr=6.31e-05 tok=15.637B dt=7474 wait=179 +11:54:35 step 119060 loss=3.4028 lr=6.31e-05 tok=15.640B dt=7538 wait=179 +11:55:39 step 119080 loss=3.3559 lr=6.31e-05 tok=15.642B dt=7602 wait=180 +11:56:43 step 119100 loss=3.3819 lr=6.31e-05 tok=15.645B dt=7666 wait=180 +11:56:55 pushed ckpt_w101.pt,log_w101.txt +11:57:40 waited 45s for global r=403 +11:57:48 global r=403 load 4.298 -> 4.283 +11:59:00 step 119120 loss=3.3206 lr=6.30e-05 tok=15.647B dt=7803 wait=180 +12:00:16 step 119140 loss=3.1387 lr=6.30e-05 tok=15.650B dt=7879 wait=181 +12:01:20 step 119160 loss=3.3543 lr=6.30e-05 tok=15.653B dt=7943 wait=181 +12:02:24 step 119180 loss=3.2909 lr=6.30e-05 tok=15.655B dt=8007 wait=181 +12:03:28 step 119200 loss=3.3795 lr=6.30e-05 tok=15.658B dt=8071 wait=182 +12:04:32 step 119220 loss=3.4319 lr=6.29e-05 tok=15.660B dt=8135 wait=182 +12:05:36 step 119240 loss=3.2856 lr=6.29e-05 tok=15.663B dt=8199 wait=182 +12:06:24 pushed ckpt_w101.pt,log_w101.txt +12:07:10 waited 45s for global r=404 +12:07:16 global r=404 load 4.345 -> 4.297 +12:07:49 step 119260 loss=3.3775 lr=6.29e-05 tok=15.666B dt=8332 wait=183 +12:09:13 step 119280 loss=3.3368 lr=6.29e-05 tok=15.668B dt=8416 wait=183 +12:10:16 step 119300 loss=3.3483 lr=6.29e-05 tok=15.671B dt=8479 wait=183 +12:11:20 step 119320 loss=3.4823 lr=6.28e-05 tok=15.674B dt=8543 wait=184 +12:12:24 step 119340 loss=3.3250 lr=6.28e-05 tok=15.676B dt=8607 wait=184 +12:13:28 step 119360 loss=3.4078 lr=6.28e-05 tok=15.679B dt=8671 wait=184 +12:14:32 step 119380 loss=3.3566 lr=6.28e-05 tok=15.681B dt=8735 wait=185 +12:15:36 step 119400 loss=3.4391 lr=6.28e-05 tok=15.684B dt=8799 wait=185 +12:15:48 pushed ckpt_w101.pt,log_w101.txt +12:16:33 waited 45s for global r=405 +12:16:43 global r=405 load 4.285 -> 4.283 +12:17:55 step 119420 loss=3.3275 lr=6.27e-05 tok=15.687B dt=8938 wait=185 +12:19:11 step 119440 loss=3.3914 lr=6.27e-05 tok=15.689B dt=9014 wait=186 +12:20:15 step 119460 loss=3.3857 lr=6.27e-05 tok=15.692B dt=9078 wait=186 +12:21:18 step 119480 loss=3.3320 lr=6.27e-05 tok=15.695B dt=9141 wait=186 +12:22:22 step 119500 loss=3.3672 lr=6.27e-05 tok=15.697B dt=9205 wait=187 +12:23:26 step 119520 loss=3.4529 lr=6.26e-05 tok=15.700B dt=9269 wait=187 +12:24:30 step 119540 loss=3.2912 lr=6.26e-05 tok=15.702B dt=9333 wait=187 +12:25:16 pushed ckpt_w101.pt,log_w101.txt +12:26:02 waited 45s for global r=406 +12:26:07 global r=406 load 4.278 -> 4.282 +12:26:40 step 119560 loss=3.4600 lr=6.26e-05 tok=15.705B dt=9463 wait=188 +12:27:52 step 119580 loss=3.3449 lr=6.26e-05 tok=15.708B dt=9535 wait=188 +12:29:08 step 119600 loss=3.4419 lr=6.26e-05 tok=15.710B dt=9611 wait=188 +12:30:12 step 119620 loss=3.3536 lr=6.25e-05 tok=15.713B dt=9675 wait=189 +12:31:16 step 119640 loss=3.3187 lr=6.25e-05 tok=15.716B dt=9739 wait=189 +12:32:19 step 119660 loss=3.4240 lr=6.25e-05 tok=15.718B dt=9802 wait=189 +12:33:23 step 119680 loss=3.3799 lr=6.25e-05 tok=15.721B dt=9866 wait=190 +12:34:26 step 119700 loss=3.3561 lr=6.25e-05 tok=15.723B dt=9929 wait=190 +12:34:38 pushed ckpt_w101.pt,log_w101.txt +12:35:23 waited 45s for global r=407 +12:35:30 global r=407 load 4.280 -> 4.275 +12:36:35 step 119720 loss=3.3060 lr=6.24e-05 tok=15.726B dt=10058 wait=190 +12:37:55 step 119740 loss=3.3876 lr=6.24e-05 tok=15.729B dt=10138 wait=191 +12:39:03 step 119760 loss=3.3433 lr=6.24e-05 tok=15.731B dt=10206 wait=191 +12:40:07 step 119780 loss=3.3593 lr=6.24e-05 tok=15.734B dt=10270 wait=191 +12:41:10 step 119800 loss=3.3051 lr=6.23e-05 tok=15.737B dt=10333 wait=192 +12:42:14 step 119820 loss=3.4342 lr=6.23e-05 tok=15.739B dt=10397 wait=192 +12:43:18 step 119840 loss=3.2993 lr=6.23e-05 tok=15.742B dt=10461 wait=192 diff --git a/log_w102.txt b/log_w102.txt new file mode 100644 index 0000000000000000000000000000000000000000..2d0efa52e01eff41effb67835e8f60127b719f8b --- /dev/null +++ b/log_w102.txt @@ -0,0 +1,187 @@ +launch +09:46:48 name=w102 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 49/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +09:46:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +09:46:55 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +09:47:04 params 1.08B +09:47:05 resumed step=117150 seen=15.39B round=390 +09:47:05 stream salt=117150 +09:48:50 blocks compiled +09:48:50 start: accum=4, 131072 tokens per optimiser step +09:51:21 step 117151 loss=3.1895 lr=6.50e-05 tok=15.389B dt=151 wait=109 +09:51:25 step 117152 loss=3.0752 lr=6.50e-05 tok=15.389B dt=154 wait=109 +09:51:52 step 117160 loss=3.3907 lr=6.50e-05 tok=15.390B dt=182 wait=109 +09:53:01 step 117180 loss=3.2835 lr=6.50e-05 tok=15.393B dt=251 wait=110 +09:54:05 step 117200 loss=3.2837 lr=6.50e-05 tok=15.396B dt=314 wait=110 +09:55:08 step 117220 loss=3.3342 lr=6.50e-05 tok=15.398B dt=377 wait=111 +09:56:11 step 117240 loss=3.3658 lr=6.50e-05 tok=15.401B dt=440 wait=111 +09:57:14 step 117260 loss=3.5328 lr=6.49e-05 tok=15.404B dt=504 wait=111 +09:58:18 step 117280 loss=3.3467 lr=6.49e-05 tok=15.406B dt=567 wait=112 +09:59:21 step 117300 loss=3.3270 lr=6.49e-05 tok=15.409B dt=630 wait=112 +09:59:34 pushed ckpt_w102.pt,log_w102.txt +10:01:20 waited 106s for global r=391 +10:01:38 global r=391 load 4.309 -> 4.286 +10:02:44 step 117320 loss=3.2357 lr=6.49e-05 tok=15.411B dt=834 wait=112 +10:03:47 step 117340 loss=3.2917 lr=6.49e-05 tok=15.414B dt=897 wait=113 +10:04:50 step 117360 loss=3.2900 lr=6.48e-05 tok=15.417B dt=960 wait=113 +10:05:53 step 117380 loss=3.2670 lr=6.48e-05 tok=15.419B dt=1023 wait=113 +10:06:56 step 117400 loss=3.3320 lr=6.48e-05 tok=15.422B dt=1086 wait=113 +10:07:59 step 117420 loss=3.3744 lr=6.48e-05 tok=15.425B dt=1148 wait=114 +10:09:02 step 117440 loss=3.2871 lr=6.48e-05 tok=15.427B dt=1211 wait=114 +10:09:46 pushed ckpt_w102.pt,log_w102.txt +10:11:17 waited 91s for global r=392 +10:11:23 global r=392 load 4.311 -> 4.292 +10:11:57 step 117460 loss=3.3683 lr=6.47e-05 tok=15.430B dt=1387 wait=114 +10:13:00 step 117480 loss=3.2830 lr=6.47e-05 tok=15.432B dt=1450 wait=115 +10:14:03 step 117500 loss=3.2678 lr=6.47e-05 tok=15.435B dt=1513 wait=115 +10:15:07 step 117520 loss=3.3387 lr=6.47e-05 tok=15.438B dt=1576 wait=115 +10:16:10 step 117540 loss=3.2339 lr=6.46e-05 tok=15.440B dt=1639 wait=116 +10:17:13 step 117560 loss=3.3206 lr=6.46e-05 tok=15.443B dt=1703 wait=116 +10:18:16 step 117580 loss=3.2837 lr=6.46e-05 tok=15.446B dt=1766 wait=116 +10:19:20 step 117600 loss=3.3804 lr=6.46e-05 tok=15.448B dt=1829 wait=117 +10:19:32 pushed ckpt_w102.pt,log_w102.txt +10:20:32 waited 61s for global r=393 +10:20:47 global r=393 load 4.307 -> 4.289 +10:21:51 step 117620 loss=3.2594 lr=6.46e-05 tok=15.451B dt=1981 wait=117 +10:22:55 step 117640 loss=3.3475 lr=6.45e-05 tok=15.453B dt=2044 wait=117 +10:23:58 step 117660 loss=3.3529 lr=6.45e-05 tok=15.456B dt=2107 wait=118 +10:25:01 step 117680 loss=3.3187 lr=6.45e-05 tok=15.459B dt=2171 wait=118 +10:26:04 step 117700 loss=3.3231 lr=6.45e-05 tok=15.461B dt=2234 wait=118 +10:27:07 step 117720 loss=3.3900 lr=6.45e-05 tok=15.464B dt=2297 wait=119 +10:28:11 step 117740 loss=3.3158 lr=6.44e-05 tok=15.466B dt=2360 wait=119 +10:28:56 pushed ckpt_w102.pt,log_w102.txt +10:30:11 waited 76s for global r=394 +10:30:17 global r=394 load 4.295 -> 4.273 +10:30:50 step 117760 loss=3.3428 lr=6.44e-05 tok=15.469B dt=2519 wait=119 +10:31:53 step 117780 loss=3.3156 lr=6.44e-05 tok=15.472B dt=2582 wait=119 +10:32:56 step 117800 loss=3.3210 lr=6.44e-05 tok=15.474B dt=2646 wait=120 +10:34:00 step 117820 loss=3.2295 lr=6.44e-05 tok=15.477B dt=2709 wait=120 +10:35:03 step 117840 loss=3.2964 lr=6.43e-05 tok=15.480B dt=2773 wait=120 +10:36:07 step 117860 loss=3.3405 lr=6.43e-05 tok=15.482B dt=2836 wait=121 +10:37:10 step 117880 loss=3.2369 lr=6.43e-05 tok=15.485B dt=2899 wait=121 +10:38:13 step 117900 loss=3.3588 lr=6.43e-05 tok=15.487B dt=2962 wait=121 +10:38:25 pushed ckpt_w102.pt,log_w102.txt +10:39:40 waited 76s for global r=395 +10:39:47 global r=395 load 4.275 -> 4.278 +10:40:51 step 117920 loss=3.2592 lr=6.43e-05 tok=15.490B dt=3121 wait=122 +10:41:55 step 117940 loss=3.3279 lr=6.42e-05 tok=15.493B dt=3184 wait=122 +10:42:59 step 117960 loss=3.3455 lr=6.42e-05 tok=15.495B dt=3248 wait=122 +10:44:02 step 117980 loss=3.2995 lr=6.42e-05 tok=15.498B dt=3312 wait=123 +10:45:05 step 118000 loss=3.3159 lr=6.42e-05 tok=15.501B dt=3375 wait=123 +10:46:09 step 118020 loss=3.4357 lr=6.42e-05 tok=15.503B dt=3438 wait=123 +10:47:13 step 118040 loss=3.2478 lr=6.41e-05 tok=15.506B dt=3503 wait=123 +10:47:58 pushed ckpt_w102.pt,log_w102.txt +10:49:13 waited 76s for global r=396 +10:49:19 global r=396 load 4.295 -> 4.294 +10:49:52 step 118060 loss=3.4367 lr=6.41e-05 tok=15.508B dt=3661 wait=124 +10:50:55 step 118080 loss=3.3135 lr=6.41e-05 tok=15.511B dt=3725 wait=124 +10:51:59 step 118100 loss=3.3122 lr=6.41e-05 tok=15.514B dt=3788 wait=124 +10:53:02 step 118120 loss=3.4223 lr=6.41e-05 tok=15.516B dt=3851 wait=125 +10:54:05 step 118140 loss=3.2574 lr=6.40e-05 tok=15.519B dt=3914 wait=125 +10:55:08 step 118160 loss=3.3557 lr=6.40e-05 tok=15.522B dt=3977 wait=125 +10:56:11 step 118180 loss=3.2842 lr=6.40e-05 tok=15.524B dt=4040 wait=126 +10:57:14 step 118200 loss=3.2681 lr=6.40e-05 tok=15.527B dt=4104 wait=126 +10:57:26 pushed ckpt_w102.pt,log_w102.txt +10:58:42 waited 76s for global r=397 +10:58:49 global r=397 load 4.292 -> 4.299 +10:59:53 step 118220 loss=3.3086 lr=6.40e-05 tok=15.529B dt=4262 wait=126 +11:00:55 step 118240 loss=3.3114 lr=6.39e-05 tok=15.532B dt=4325 wait=127 +11:01:59 step 118260 loss=3.2838 lr=6.39e-05 tok=15.535B dt=4388 wait=127 +11:03:02 step 118280 loss=3.2411 lr=6.39e-05 tok=15.537B dt=4451 wait=127 +11:04:05 step 118300 loss=3.2995 lr=6.39e-05 tok=15.540B dt=4515 wait=127 +11:05:09 step 118320 loss=3.3899 lr=6.39e-05 tok=15.543B dt=4578 wait=128 +11:06:12 step 118340 loss=3.2324 lr=6.38e-05 tok=15.545B dt=4641 wait=128 +11:06:58 pushed ckpt_w102.pt,log_w102.txt +11:07:59 waited 61s for global r=398 +11:08:05 global r=398 load 4.304 -> 4.295 +11:08:38 step 118360 loss=3.3456 lr=6.38e-05 tok=15.548B dt=4787 wait=128 +11:09:41 step 118380 loss=3.2540 lr=6.38e-05 tok=15.550B dt=4850 wait=129 +11:10:44 step 118400 loss=3.2390 lr=6.38e-05 tok=15.553B dt=4914 wait=129 +11:11:47 step 118420 loss=3.3986 lr=6.38e-05 tok=15.556B dt=4977 wait=129 +11:12:50 step 118440 loss=3.2593 lr=6.37e-05 tok=15.558B dt=5040 wait=130 +11:13:54 step 118460 loss=3.3677 lr=6.37e-05 tok=15.561B dt=5103 wait=130 +11:14:57 step 118480 loss=3.3350 lr=6.37e-05 tok=15.563B dt=5167 wait=130 +11:16:00 step 118500 loss=3.3127 lr=6.37e-05 tok=15.566B dt=5230 wait=131 +11:16:13 pushed ckpt_w102.pt,log_w102.txt +11:17:28 waited 76s for global r=399 +11:17:34 global r=399 load 4.288 -> 4.292 +11:18:39 step 118520 loss=3.2859 lr=6.36e-05 tok=15.569B dt=5388 wait=131 +11:19:42 step 118540 loss=3.3492 lr=6.36e-05 tok=15.571B dt=5452 wait=131 +11:20:45 step 118560 loss=3.2987 lr=6.36e-05 tok=15.574B dt=5515 wait=132 +11:21:48 step 118580 loss=3.2771 lr=6.36e-05 tok=15.577B dt=5578 wait=132 +11:22:52 step 118600 loss=3.3065 lr=6.36e-05 tok=15.579B dt=5641 wait=132 +11:23:55 step 118620 loss=3.3389 lr=6.35e-05 tok=15.582B dt=5704 wait=132 +11:24:58 step 118640 loss=3.2828 lr=6.35e-05 tok=15.584B dt=5767 wait=133 +11:25:43 pushed ckpt_w102.pt,log_w102.txt +11:28:29 waited 166s for global r=400 +11:28:36 global r=400 load 4.308 -> 4.285 +11:29:09 step 118660 loss=3.3910 lr=6.35e-05 tok=15.587B dt=6019 wait=133 +11:30:13 step 118680 loss=3.2801 lr=6.35e-05 tok=15.590B dt=6082 wait=133 +11:31:16 step 118700 loss=3.2424 lr=6.35e-05 tok=15.592B dt=6145 wait=134 +11:32:19 step 118720 loss=3.3784 lr=6.34e-05 tok=15.595B dt=6209 wait=134 +11:33:23 step 118740 loss=3.1836 lr=6.34e-05 tok=15.598B dt=6273 wait=134 +11:34:27 step 118760 loss=3.2816 lr=6.34e-05 tok=15.600B dt=6336 wait=135 +11:35:30 step 118780 loss=3.2659 lr=6.34e-05 tok=15.603B dt=6399 wait=135 +11:36:33 step 118800 loss=3.3272 lr=6.34e-05 tok=15.605B dt=6462 wait=135 +11:36:45 pushed ckpt_w102.pt,log_w102.txt +11:38:46 waited 121s for global r=401 +11:38:53 global r=401 load 4.323 -> 4.288 +11:39:57 step 118820 loss=3.2673 lr=6.33e-05 tok=15.608B dt=6667 wait=136 +11:41:01 step 118840 loss=3.3189 lr=6.33e-05 tok=15.611B dt=6730 wait=136 +11:42:04 step 118860 loss=3.3539 lr=6.33e-05 tok=15.613B dt=6794 wait=136 +11:43:07 step 118880 loss=3.3255 lr=6.33e-05 tok=15.616B dt=6857 wait=137 +11:44:10 step 118900 loss=3.3254 lr=6.33e-05 tok=15.619B dt=6920 wait=137 +11:45:13 step 118920 loss=3.3884 lr=6.32e-05 tok=15.621B dt=6983 wait=137 +11:46:16 step 118940 loss=3.2627 lr=6.32e-05 tok=15.624B dt=7046 wait=138 +11:47:00 pushed ckpt_w102.pt,log_w102.txt +11:48:15 waited 76s for global r=402 +11:48:23 global r=402 load 4.303 -> 4.284 +11:48:56 step 118960 loss=3.3169 lr=6.32e-05 tok=15.626B dt=7205 wait=138 +11:49:59 step 118980 loss=3.2219 lr=6.32e-05 tok=15.629B dt=7269 wait=138 +11:51:02 step 119000 loss=3.2975 lr=6.32e-05 tok=15.632B dt=7332 wait=139 +11:52:06 step 119020 loss=3.2878 lr=6.31e-05 tok=15.634B dt=7396 wait=139 +11:53:10 step 119040 loss=3.2508 lr=6.31e-05 tok=15.637B dt=7459 wait=139 +11:54:13 step 119060 loss=3.3613 lr=6.31e-05 tok=15.640B dt=7522 wait=140 +11:55:16 step 119080 loss=3.2327 lr=6.31e-05 tok=15.642B dt=7586 wait=140 +11:56:20 step 119100 loss=3.3387 lr=6.31e-05 tok=15.645B dt=7650 wait=140 +11:56:34 pushed ckpt_w102.pt,log_w102.txt +11:57:34 waited 61s for global r=403 +11:57:42 global r=403 load 4.281 -> 4.283 +11:58:47 step 119120 loss=3.2420 lr=6.30e-05 tok=15.647B dt=7796 wait=141 +11:59:50 step 119140 loss=3.3005 lr=6.30e-05 tok=15.650B dt=7860 wait=141 +12:00:53 step 119160 loss=3.3380 lr=6.30e-05 tok=15.653B dt=7923 wait=141 +12:01:56 step 119180 loss=3.3025 lr=6.30e-05 tok=15.655B dt=7986 wait=142 +12:02:59 step 119200 loss=3.2807 lr=6.30e-05 tok=15.658B dt=8049 wait=142 +12:04:03 step 119220 loss=3.3427 lr=6.29e-05 tok=15.660B dt=8112 wait=142 +12:05:06 step 119240 loss=3.2627 lr=6.29e-05 tok=15.663B dt=8175 wait=142 +12:05:49 pushed ckpt_w102.pt,log_w102.txt +12:07:05 waited 76s for global r=404 +12:07:10 global r=404 load 4.306 -> 4.297 +12:07:43 step 119260 loss=3.4003 lr=6.29e-05 tok=15.666B dt=8333 wait=143 +12:08:47 step 119280 loss=3.2320 lr=6.29e-05 tok=15.668B dt=8396 wait=143 +12:09:50 step 119300 loss=3.2784 lr=6.29e-05 tok=15.671B dt=8460 wait=143 +12:10:53 step 119320 loss=3.3329 lr=6.28e-05 tok=15.674B dt=8522 wait=144 +12:11:56 step 119340 loss=3.1604 lr=6.28e-05 tok=15.676B dt=8586 wait=144 +12:13:00 step 119360 loss=3.2709 lr=6.28e-05 tok=15.679B dt=8650 wait=144 +12:14:04 step 119380 loss=3.2801 lr=6.28e-05 tok=15.681B dt=8713 wait=145 +12:15:07 step 119400 loss=3.2750 lr=6.28e-05 tok=15.684B dt=8776 wait=145 +12:15:19 pushed ckpt_w102.pt,log_w102.txt +12:16:35 waited 76s for global r=405 +12:16:42 global r=405 load 4.309 -> 4.283 +12:17:47 step 119420 loss=3.2869 lr=6.27e-05 tok=15.687B dt=8936 wait=145 +12:18:50 step 119440 loss=3.3098 lr=6.27e-05 tok=15.689B dt=8999 wait=146 +12:19:53 step 119460 loss=3.3048 lr=6.27e-05 tok=15.692B dt=9062 wait=146 +12:20:56 step 119480 loss=3.2813 lr=6.27e-05 tok=15.695B dt=9125 wait=146 +12:21:59 step 119500 loss=3.2986 lr=6.27e-05 tok=15.697B dt=9188 wait=147 +12:23:02 step 119520 loss=3.4027 lr=6.26e-05 tok=15.700B dt=9252 wait=147 +12:24:05 step 119540 loss=3.2487 lr=6.26e-05 tok=15.702B dt=9315 wait=147 +12:24:49 pushed ckpt_w102.pt,log_w102.txt +12:26:04 waited 76s for global r=406 +12:26:12 global r=406 load 4.301 -> 4.282 +12:26:45 step 119560 loss=3.3507 lr=6.26e-05 tok=15.705B dt=9474 wait=148 +12:27:48 step 119580 loss=3.2276 lr=6.26e-05 tok=15.708B dt=9538 wait=148 +12:28:51 step 119600 loss=3.3160 lr=6.26e-05 tok=15.710B dt=9601 wait=148 +12:29:54 step 119620 loss=3.3115 lr=6.25e-05 tok=15.713B dt=9664 wait=148 +12:30:58 step 119640 loss=3.2616 lr=6.25e-05 tok=15.716B dt=9727 wait=149 +12:32:02 step 119660 loss=3.3372 lr=6.25e-05 tok=15.718B dt=9791 wait=149 +12:33:05 step 119680 loss=3.2475 lr=6.25e-05 tok=15.721B dt=9855 wait=150 +12:34:09 step 119700 loss=3.3753 lr=6.25e-05 tok=15.723B dt=9918 wait=150 diff --git a/log_w103.txt b/log_w103.txt new file mode 100644 index 0000000000000000000000000000000000000000..b7feb8c6c1ea79bac047c53bc46089cb7c976b54 --- /dev/null +++ b/log_w103.txt @@ -0,0 +1,187 @@ +launch +09:46:49 name=w103 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 50/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +09:46:55 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +09:46:55 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +09:47:05 params 1.08B +09:47:06 resumed step=117150 seen=15.39B round=390 +09:47:06 stream salt=117150 +09:48:52 blocks compiled +09:48:52 start: accum=4, 131072 tokens per optimiser step +09:52:18 step 117151 loss=3.2204 lr=6.50e-05 tok=15.389B dt=206 wait=160 +09:52:22 step 117152 loss=3.2276 lr=6.50e-05 tok=15.389B dt=210 wait=160 +09:52:51 step 117160 loss=3.5138 lr=6.50e-05 tok=15.390B dt=239 wait=161 +09:54:02 step 117180 loss=3.4057 lr=6.50e-05 tok=15.393B dt=311 wait=161 +09:55:07 step 117200 loss=3.4172 lr=6.50e-05 tok=15.396B dt=375 wait=162 +09:56:11 step 117220 loss=3.3566 lr=6.50e-05 tok=15.398B dt=439 wait=162 +09:57:15 step 117240 loss=3.4843 lr=6.50e-05 tok=15.401B dt=504 wait=163 +09:58:20 step 117260 loss=3.6000 lr=6.49e-05 tok=15.404B dt=568 wait=163 +09:59:25 step 117280 loss=3.4496 lr=6.49e-05 tok=15.406B dt=633 wait=163 +10:00:28 step 117300 loss=3.4504 lr=6.49e-05 tok=15.409B dt=697 wait=163 +10:00:43 pushed ckpt_w103.pt,log_w103.txt +10:01:29 waited 45s for global r=391 +10:01:41 global r=391 load 4.317 -> 4.286 +10:02:48 step 117320 loss=3.3626 lr=6.49e-05 tok=15.411B dt=837 wait=164 +10:03:52 step 117340 loss=3.4500 lr=6.49e-05 tok=15.414B dt=901 wait=164 +10:04:57 step 117360 loss=3.4021 lr=6.48e-05 tok=15.417B dt=965 wait=165 +10:06:00 step 117380 loss=3.3672 lr=6.48e-05 tok=15.419B dt=1029 wait=165 +10:07:05 step 117400 loss=3.3870 lr=6.48e-05 tok=15.422B dt=1093 wait=165 +10:08:09 step 117420 loss=3.5208 lr=6.48e-05 tok=15.425B dt=1157 wait=166 +10:09:13 step 117440 loss=3.3264 lr=6.48e-05 tok=15.427B dt=1221 wait=166 +10:09:58 pushed ckpt_w103.pt,log_w103.txt +10:11:14 waited 76s for global r=392 +10:11:21 global r=392 load 4.313 -> 4.292 +10:11:55 step 117460 loss=3.4859 lr=6.47e-05 tok=15.430B dt=1383 wait=166 +10:12:59 step 117480 loss=3.3667 lr=6.47e-05 tok=15.432B dt=1447 wait=167 +10:14:03 step 117500 loss=3.3734 lr=6.47e-05 tok=15.435B dt=1511 wait=167 +10:15:07 step 117520 loss=3.4316 lr=6.47e-05 tok=15.438B dt=1575 wait=167 +10:16:11 step 117540 loss=3.2972 lr=6.46e-05 tok=15.440B dt=1640 wait=168 +10:17:15 step 117560 loss=3.4276 lr=6.46e-05 tok=15.443B dt=1704 wait=168 +10:18:20 step 117580 loss=3.4502 lr=6.46e-05 tok=15.446B dt=1768 wait=168 +10:19:23 step 117600 loss=3.4164 lr=6.46e-05 tok=15.448B dt=1832 wait=169 +10:19:35 pushed ckpt_w103.pt,log_w103.txt +10:20:36 waited 61s for global r=393 +10:20:46 global r=393 load 4.292 -> 4.289 +10:21:52 step 117620 loss=3.3491 lr=6.46e-05 tok=15.451B dt=1980 wait=169 +10:22:56 step 117640 loss=3.3962 lr=6.45e-05 tok=15.453B dt=2044 wait=169 +10:24:00 step 117660 loss=3.4244 lr=6.45e-05 tok=15.456B dt=2109 wait=170 +10:25:04 step 117680 loss=3.3623 lr=6.45e-05 tok=15.459B dt=2173 wait=170 +10:26:08 step 117700 loss=3.3423 lr=6.45e-05 tok=15.461B dt=2237 wait=170 +10:27:13 step 117720 loss=3.4766 lr=6.45e-05 tok=15.464B dt=2301 wait=171 +10:28:17 step 117740 loss=3.3216 lr=6.44e-05 tok=15.466B dt=2365 wait=171 +10:29:01 pushed ckpt_w103.pt,log_w103.txt +10:30:01 waited 61s for global r=394 +10:30:08 global r=394 load 4.299 -> 4.273 +10:30:41 step 117760 loss=3.4604 lr=6.44e-05 tok=15.469B dt=2509 wait=171 +10:31:45 step 117780 loss=3.3204 lr=6.44e-05 tok=15.472B dt=2574 wait=172 +10:32:49 step 117800 loss=3.3053 lr=6.44e-05 tok=15.474B dt=2637 wait=172 +10:33:53 step 117820 loss=3.3884 lr=6.44e-05 tok=15.477B dt=2702 wait=172 +10:34:58 step 117840 loss=3.3439 lr=6.43e-05 tok=15.480B dt=2766 wait=173 +10:36:02 step 117860 loss=3.3927 lr=6.43e-05 tok=15.482B dt=2831 wait=173 +10:37:06 step 117880 loss=3.4369 lr=6.43e-05 tok=15.485B dt=2895 wait=173 +10:38:11 step 117900 loss=3.5045 lr=6.43e-05 tok=15.487B dt=2959 wait=174 +10:38:25 pushed ckpt_w103.pt,log_w103.txt +10:39:40 waited 76s for global r=395 +10:39:49 global r=395 load 4.305 -> 4.278 +10:40:54 step 117920 loss=3.3449 lr=6.43e-05 tok=15.490B dt=3123 wait=174 +10:41:59 step 117940 loss=3.4631 lr=6.42e-05 tok=15.493B dt=3187 wait=174 +10:43:03 step 117960 loss=3.4310 lr=6.42e-05 tok=15.495B dt=3251 wait=174 +10:44:07 step 117980 loss=3.4289 lr=6.42e-05 tok=15.498B dt=3315 wait=175 +10:45:12 step 118000 loss=3.3561 lr=6.42e-05 tok=15.501B dt=3380 wait=175 +10:46:15 step 118020 loss=3.4589 lr=6.42e-05 tok=15.503B dt=3444 wait=175 +10:47:19 step 118040 loss=3.3376 lr=6.41e-05 tok=15.506B dt=3508 wait=176 +10:48:06 pushed ckpt_w103.pt,log_w103.txt +10:49:07 waited 61s for global r=396 +10:49:14 global r=396 load 4.310 -> 4.294 +10:49:47 step 118060 loss=3.4979 lr=6.41e-05 tok=15.508B dt=3656 wait=176 +10:50:52 step 118080 loss=3.4180 lr=6.41e-05 tok=15.511B dt=3720 wait=176 +10:51:56 step 118100 loss=3.3736 lr=6.41e-05 tok=15.514B dt=3784 wait=177 +10:53:00 step 118120 loss=3.4923 lr=6.41e-05 tok=15.516B dt=3849 wait=177 +10:54:04 step 118140 loss=3.3657 lr=6.40e-05 tok=15.519B dt=3913 wait=177 +10:55:08 step 118160 loss=3.4472 lr=6.40e-05 tok=15.522B dt=3977 wait=178 +10:56:13 step 118180 loss=3.5400 lr=6.40e-05 tok=15.524B dt=4041 wait=178 +10:57:17 step 118200 loss=3.4367 lr=6.40e-05 tok=15.527B dt=4106 wait=178 +10:57:32 pushed ckpt_w103.pt,log_w103.txt +10:58:32 waited 60s for global r=397 +10:58:38 global r=397 load 4.306 -> 4.299 +10:59:43 step 118220 loss=3.3852 lr=6.40e-05 tok=15.529B dt=4252 wait=179 +11:00:47 step 118240 loss=3.4108 lr=6.39e-05 tok=15.532B dt=4316 wait=179 +11:01:51 step 118260 loss=3.3915 lr=6.39e-05 tok=15.535B dt=4380 wait=179 +11:02:55 step 118280 loss=3.2983 lr=6.39e-05 tok=15.537B dt=4444 wait=180 +11:03:59 step 118300 loss=3.3156 lr=6.39e-05 tok=15.540B dt=4507 wait=180 +11:05:02 step 118320 loss=3.4280 lr=6.39e-05 tok=15.543B dt=4571 wait=181 +11:06:06 step 118340 loss=3.2766 lr=6.38e-05 tok=15.545B dt=4634 wait=181 +11:06:52 pushed ckpt_w103.pt,log_w103.txt +11:08:07 waited 76s for global r=398 +11:08:15 global r=398 load 4.298 -> 4.295 +11:08:48 step 118360 loss=3.4434 lr=6.38e-05 tok=15.548B dt=4796 wait=181 +11:09:52 step 118380 loss=3.2939 lr=6.38e-05 tok=15.550B dt=4861 wait=182 +11:10:56 step 118400 loss=3.3296 lr=6.38e-05 tok=15.553B dt=4924 wait=182 +11:11:59 step 118420 loss=3.3924 lr=6.38e-05 tok=15.556B dt=4988 wait=182 +11:13:03 step 118440 loss=3.3095 lr=6.37e-05 tok=15.558B dt=5051 wait=183 +11:14:06 step 118460 loss=3.3876 lr=6.37e-05 tok=15.561B dt=5114 wait=183 +11:15:09 step 118480 loss=3.3065 lr=6.37e-05 tok=15.563B dt=5178 wait=183 +11:16:13 step 118500 loss=3.3328 lr=6.37e-05 tok=15.566B dt=5241 wait=184 +11:16:25 pushed ckpt_w103.pt,log_w103.txt +11:17:26 waited 61s for global r=399 +11:17:33 global r=399 load 4.327 -> 4.292 +11:18:38 step 118520 loss=3.3416 lr=6.36e-05 tok=15.569B dt=5387 wait=184 +11:19:42 step 118540 loss=3.3075 lr=6.36e-05 tok=15.571B dt=5450 wait=184 +11:20:45 step 118560 loss=3.3328 lr=6.36e-05 tok=15.574B dt=5513 wait=185 +11:21:49 step 118580 loss=3.3302 lr=6.36e-05 tok=15.577B dt=5577 wait=185 +11:22:52 step 118600 loss=3.2923 lr=6.36e-05 tok=15.579B dt=5640 wait=185 +11:24:11 step 118620 loss=3.4236 lr=6.35e-05 tok=15.582B dt=5720 wait=185 +11:26:32 step 118640 loss=3.2566 lr=6.35e-05 tok=15.584B dt=5860 wait=186 +11:27:55 pushed ckpt_w103.pt,log_w103.txt +11:28:40 waited 45s for global r=400 +11:28:47 global r=400 load 4.286 -> 4.285 +11:29:56 step 118660 loss=3.3834 lr=6.35e-05 tok=15.587B dt=6064 wait=186 +11:31:34 step 118680 loss=3.3149 lr=6.35e-05 tok=15.590B dt=6163 wait=187 +11:32:38 step 118700 loss=3.3212 lr=6.35e-05 tok=15.592B dt=6226 wait=187 +11:33:41 step 118720 loss=3.4289 lr=6.34e-05 tok=15.595B dt=6290 wait=187 +11:34:46 step 118740 loss=3.3134 lr=6.34e-05 tok=15.598B dt=6354 wait=187 +11:35:49 step 118760 loss=3.4082 lr=6.34e-05 tok=15.600B dt=6418 wait=188 +11:36:53 step 118780 loss=3.3686 lr=6.34e-05 tok=15.603B dt=6481 wait=188 +11:37:56 step 118800 loss=3.3900 lr=6.34e-05 tok=15.605B dt=6544 wait=188 +11:38:08 pushed ckpt_w103.pt,log_w103.txt +11:38:53 waited 45s for global r=401 +11:39:00 global r=401 load 4.281 -> 4.288 +11:40:06 step 118820 loss=3.3365 lr=6.33e-05 tok=15.608B dt=6674 wait=189 +11:41:09 step 118840 loss=3.3432 lr=6.33e-05 tok=15.611B dt=6738 wait=189 +11:42:13 step 118860 loss=3.3289 lr=6.33e-05 tok=15.613B dt=6801 wait=189 +11:43:16 step 118880 loss=3.3044 lr=6.33e-05 tok=15.616B dt=6865 wait=190 +11:44:20 step 118900 loss=3.3618 lr=6.33e-05 tok=15.619B dt=6928 wait=190 +11:45:26 step 118920 loss=2.9176 lr=6.32e-05 tok=15.621B dt=6994 wait=190 +11:46:29 step 118940 loss=3.3381 lr=6.32e-05 tok=15.624B dt=7058 wait=191 +11:47:13 pushed ckpt_w103.pt,log_w103.txt +11:48:14 waited 61s for global r=402 +11:48:20 global r=402 load 4.278 -> 4.284 +11:48:53 step 118960 loss=3.4487 lr=6.32e-05 tok=15.626B dt=7201 wait=191 +11:49:57 step 118980 loss=3.2676 lr=6.32e-05 tok=15.629B dt=7265 wait=191 +11:51:01 step 119000 loss=3.2487 lr=6.32e-05 tok=15.632B dt=7329 wait=192 +11:52:05 step 119020 loss=3.4016 lr=6.31e-05 tok=15.634B dt=7393 wait=192 +11:53:08 step 119040 loss=3.2879 lr=6.31e-05 tok=15.637B dt=7457 wait=192 +11:54:12 step 119060 loss=3.4324 lr=6.31e-05 tok=15.640B dt=7520 wait=193 +11:55:15 step 119080 loss=3.3829 lr=6.31e-05 tok=15.642B dt=7584 wait=193 +11:56:19 step 119100 loss=3.3803 lr=6.31e-05 tok=15.645B dt=7647 wait=193 +11:56:31 pushed ckpt_w103.pt,log_w103.txt +11:57:32 waited 61s for global r=403 +11:57:40 global r=403 load 4.301 -> 4.283 +11:58:45 step 119120 loss=3.3204 lr=6.30e-05 tok=15.647B dt=7793 wait=194 +11:59:48 step 119140 loss=3.3259 lr=6.30e-05 tok=15.650B dt=7857 wait=194 +12:00:52 step 119160 loss=3.3640 lr=6.30e-05 tok=15.653B dt=7920 wait=194 +12:01:55 step 119180 loss=3.3469 lr=6.30e-05 tok=15.655B dt=7984 wait=195 +12:02:58 step 119200 loss=3.2959 lr=6.30e-05 tok=15.658B dt=8047 wait=195 +12:04:02 step 119220 loss=3.4635 lr=6.29e-05 tok=15.660B dt=8111 wait=195 +12:05:05 step 119240 loss=3.2947 lr=6.29e-05 tok=15.663B dt=8174 wait=196 +12:05:52 pushed ckpt_w103.pt,log_w103.txt +12:07:07 waited 76s for global r=404 +12:07:13 global r=404 load 4.298 -> 4.297 +12:07:45 step 119260 loss=3.4028 lr=6.29e-05 tok=15.666B dt=8334 wait=196 +12:08:49 step 119280 loss=3.2983 lr=6.29e-05 tok=15.668B dt=8397 wait=196 +12:09:53 step 119300 loss=3.3129 lr=6.29e-05 tok=15.671B dt=8461 wait=197 +12:10:57 step 119320 loss=3.4347 lr=6.28e-05 tok=15.674B dt=8525 wait=197 +12:12:00 step 119340 loss=3.2894 lr=6.28e-05 tok=15.676B dt=8588 wait=197 +12:13:04 step 119360 loss=3.3984 lr=6.28e-05 tok=15.679B dt=8652 wait=197 +12:14:08 step 119380 loss=3.3040 lr=6.28e-05 tok=15.681B dt=8717 wait=198 +12:15:11 step 119400 loss=3.3408 lr=6.28e-05 tok=15.684B dt=8780 wait=198 +12:15:23 pushed ckpt_w103.pt,log_w103.txt +12:16:39 waited 76s for global r=405 +12:16:47 global r=405 load 4.286 -> 4.283 +12:17:51 step 119420 loss=3.3081 lr=6.27e-05 tok=15.687B dt=8940 wait=198 +12:18:55 step 119440 loss=3.3471 lr=6.27e-05 tok=15.689B dt=9004 wait=199 +12:19:59 step 119460 loss=3.3696 lr=6.27e-05 tok=15.692B dt=9067 wait=199 +12:21:02 step 119480 loss=3.3091 lr=6.27e-05 tok=15.695B dt=9131 wait=199 +12:22:06 step 119500 loss=3.3328 lr=6.27e-05 tok=15.697B dt=9194 wait=200 +12:23:09 step 119520 loss=3.3785 lr=6.26e-05 tok=15.700B dt=9257 wait=200 +12:24:13 step 119540 loss=3.1272 lr=6.26e-05 tok=15.702B dt=9321 wait=200 +12:24:57 pushed ckpt_w103.pt,log_w103.txt +12:25:58 waited 61s for global r=406 +12:26:03 global r=406 load 4.297 -> 4.282 +12:26:36 step 119560 loss=3.4406 lr=6.26e-05 tok=15.705B dt=9465 wait=201 +12:27:40 step 119580 loss=3.2640 lr=6.26e-05 tok=15.708B dt=9529 wait=201 +12:28:44 step 119600 loss=3.3527 lr=6.26e-05 tok=15.710B dt=9592 wait=201 +12:29:47 step 119620 loss=3.3687 lr=6.25e-05 tok=15.713B dt=9655 wait=202 +12:30:50 step 119640 loss=3.3169 lr=6.25e-05 tok=15.716B dt=9719 wait=202 +12:31:54 step 119660 loss=3.3733 lr=6.25e-05 tok=15.718B dt=9782 wait=202 +12:32:57 step 119680 loss=3.3543 lr=6.25e-05 tok=15.721B dt=9845 wait=203 +12:34:00 step 119700 loss=3.3726 lr=6.25e-05 tok=15.723B dt=9909 wait=203 diff --git a/log_w106.txt b/log_w106.txt new file mode 100644 index 0000000000000000000000000000000000000000..48c2169e6cabdff96dbbf9ac8d4fd8adfb8e976b --- /dev/null +++ b/log_w106.txt @@ -0,0 +1,18 @@ +launch +12:39:05 name=w106 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 53/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:39:10 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:39:10 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:39:19 params 1.08B +12:39:21 resumed step=119700 seen=15.72B round=407 +12:39:21 stream salt=119700 +12:41:06 blocks compiled +12:41:06 start: accum=4, 131072 tokens per optimiser step +12:43:42 step 119701 loss=3.1665 lr=6.24e-05 tok=15.724B dt=156 wait=116 +12:43:46 step 119702 loss=3.2567 lr=6.24e-05 tok=15.724B dt=160 wait=116 +12:44:50 step 119720 loss=3.4389 lr=6.24e-05 tok=15.726B dt=224 wait=117 +12:45:54 step 119740 loss=3.3538 lr=6.24e-05 tok=15.729B dt=288 wait=117 +12:46:59 step 119760 loss=3.3825 lr=6.24e-05 tok=15.731B dt=352 wait=117 +12:48:03 step 119780 loss=3.4355 lr=6.24e-05 tok=15.734B dt=416 wait=118 +12:49:07 step 119800 loss=3.5528 lr=6.23e-05 tok=15.737B dt=481 wait=118 +12:50:12 step 119820 loss=3.3916 lr=6.23e-05 tok=15.739B dt=546 wait=118 +12:51:16 step 119840 loss=3.3676 lr=6.23e-05 tok=15.742B dt=610 wait=119 diff --git a/log_w107.txt b/log_w107.txt new file mode 100644 index 0000000000000000000000000000000000000000..77113572156ffc25baeb2b97a93ea208d34c02df --- /dev/null +++ b/log_w107.txt @@ -0,0 +1,176 @@ +launch +13:51:56 name=w107 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 54/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:52:02 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:52:02 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:52:12 params 1.08B +13:52:13 resumed step=119850 seen=15.74B round=408 +13:52:13 stream salt=119850 +13:53:58 blocks compiled +13:53:58 start: accum=4, 131072 tokens per optimiser step +13:56:22 step 119851 loss=3.1048 lr=6.23e-05 tok=15.743B dt=145 wait=102 +13:56:26 step 119852 loss=3.0426 lr=6.23e-05 tok=15.743B dt=148 wait=102 +13:56:54 step 119860 loss=3.3993 lr=6.23e-05 tok=15.744B dt=176 wait=102 +13:58:04 step 119880 loss=3.3165 lr=6.23e-05 tok=15.747B dt=246 wait=102 +13:59:07 step 119900 loss=3.3497 lr=6.22e-05 tok=15.750B dt=309 wait=103 +14:00:10 step 119920 loss=3.3127 lr=6.22e-05 tok=15.752B dt=372 wait=103 +14:01:13 step 119940 loss=3.3999 lr=6.22e-05 tok=15.755B dt=436 wait=104 +14:02:17 step 119960 loss=3.4968 lr=6.22e-05 tok=15.757B dt=499 wait=104 +14:03:20 step 119980 loss=3.1487 lr=6.22e-05 tok=15.760B dt=562 wait=104 +14:04:24 step 120000 loss=3.4370 lr=6.21e-05 tok=15.763B dt=626 wait=104 +14:04:40 pushed ckpt_w107.pt,log_w107.txt +14:07:11 waited 151s for global r=409 +14:07:26 global r=409 load 4.306 -> 4.279 +14:08:33 step 120020 loss=3.3063 lr=6.21e-05 tok=15.765B dt=875 wait=105 +14:09:36 step 120040 loss=3.3457 lr=6.21e-05 tok=15.768B dt=938 wait=105 +14:10:38 step 120060 loss=3.3195 lr=6.21e-05 tok=15.771B dt=1001 wait=105 +14:11:41 step 120080 loss=3.2744 lr=6.21e-05 tok=15.773B dt=1064 wait=106 +14:12:45 step 120100 loss=3.2439 lr=6.20e-05 tok=15.776B dt=1127 wait=106 +14:13:48 step 120120 loss=3.3738 lr=6.20e-05 tok=15.778B dt=1190 wait=106 +14:14:51 step 120140 loss=3.2382 lr=6.20e-05 tok=15.781B dt=1253 wait=107 +14:15:35 pushed ckpt_w107.pt,log_w107.txt +14:16:36 waited 61s for global r=410 +14:16:44 global r=410 load 4.291 -> 4.291 +14:17:17 step 120160 loss=3.3668 lr=6.20e-05 tok=15.784B dt=1399 wait=107 +14:18:20 step 120180 loss=3.2628 lr=6.20e-05 tok=15.786B dt=1462 wait=107 +14:19:23 step 120200 loss=3.2894 lr=6.19e-05 tok=15.789B dt=1525 wait=108 +14:20:26 step 120220 loss=3.3917 lr=6.19e-05 tok=15.792B dt=1588 wait=108 +14:21:29 step 120240 loss=3.2631 lr=6.19e-05 tok=15.794B dt=1652 wait=108 +14:22:33 step 120260 loss=3.3534 lr=6.19e-05 tok=15.797B dt=1715 wait=109 +14:23:36 step 120280 loss=3.2859 lr=6.19e-05 tok=15.799B dt=1778 wait=109 +14:24:40 step 120300 loss=3.2992 lr=6.18e-05 tok=15.802B dt=1842 wait=109 +14:24:52 pushed ckpt_w107.pt,log_w107.txt +14:26:08 waited 76s for global r=411 +14:26:16 global r=411 load 4.280 -> 4.277 +14:27:21 step 120320 loss=3.3247 lr=6.18e-05 tok=15.805B dt=2003 wait=110 +14:28:25 step 120340 loss=3.3470 lr=6.18e-05 tok=15.807B dt=2067 wait=110 +14:29:28 step 120360 loss=3.3635 lr=6.18e-05 tok=15.810B dt=2130 wait=110 +14:30:31 step 120380 loss=3.3138 lr=6.18e-05 tok=15.813B dt=2193 wait=111 +14:31:34 step 120400 loss=3.3051 lr=6.17e-05 tok=15.815B dt=2256 wait=111 +14:32:37 step 120420 loss=3.3728 lr=6.17e-05 tok=15.818B dt=2319 wait=111 +14:33:40 step 120440 loss=3.2475 lr=6.17e-05 tok=15.820B dt=2383 wait=112 +14:34:24 pushed ckpt_w107.pt,log_w107.txt +14:35:40 waited 76s for global r=412 +14:35:49 global r=412 load 4.294 -> 4.284 +14:36:21 step 120460 loss=3.4033 lr=6.17e-05 tok=15.823B dt=2544 wait=112 +14:37:25 step 120480 loss=3.1860 lr=6.17e-05 tok=15.826B dt=2607 wait=112 +14:38:28 step 120500 loss=3.2757 lr=6.16e-05 tok=15.828B dt=2670 wait=113 +14:39:31 step 120520 loss=3.3260 lr=6.16e-05 tok=15.831B dt=2733 wait=113 +14:40:36 step 120540 loss=3.2358 lr=6.16e-05 tok=15.833B dt=2798 wait=113 +14:41:39 step 120560 loss=3.3869 lr=6.16e-05 tok=15.836B dt=2861 wait=114 +14:42:42 step 120580 loss=3.3619 lr=6.16e-05 tok=15.839B dt=2924 wait=114 +14:43:45 step 120600 loss=3.3390 lr=6.15e-05 tok=15.841B dt=2987 wait=114 +14:43:58 pushed ckpt_w107.pt,log_w107.txt +14:45:14 waited 76s for global r=413 +14:45:24 global r=413 load 4.301 -> 4.292 +14:46:29 step 120620 loss=3.2315 lr=6.15e-05 tok=15.844B dt=3151 wait=115 +14:47:33 step 120640 loss=3.3096 lr=6.15e-05 tok=15.847B dt=3215 wait=115 +14:48:36 step 120660 loss=3.3580 lr=6.15e-05 tok=15.849B dt=3278 wait=115 +14:49:40 step 120680 loss=3.3281 lr=6.15e-05 tok=15.852B dt=3342 wait=115 +14:50:43 step 120700 loss=3.2821 lr=6.14e-05 tok=15.854B dt=3405 wait=116 +14:51:46 step 120720 loss=3.3733 lr=6.14e-05 tok=15.857B dt=3469 wait=116 +14:52:49 step 120740 loss=3.2524 lr=6.14e-05 tok=15.860B dt=3531 wait=116 +14:53:36 pushed ckpt_w107.pt,log_w107.txt +14:54:54 waited 78s for global r=414 +14:55:04 global r=414 load 4.297 -> 4.279 +14:55:37 step 120760 loss=3.3418 lr=6.14e-05 tok=15.862B dt=3699 wait=117 +14:56:39 step 120780 loss=3.2791 lr=6.14e-05 tok=15.865B dt=3762 wait=117 +14:57:42 step 120800 loss=3.2929 lr=6.13e-05 tok=15.868B dt=3824 wait=117 +14:58:45 step 120820 loss=3.4097 lr=6.13e-05 tok=15.870B dt=3888 wait=118 +14:59:49 step 120840 loss=3.2609 lr=6.13e-05 tok=15.873B dt=3951 wait=118 +15:00:52 step 120860 loss=3.3975 lr=6.13e-05 tok=15.875B dt=4014 wait=118 +15:01:55 step 120880 loss=3.3166 lr=6.13e-05 tok=15.878B dt=4077 wait=119 +15:02:58 step 120900 loss=3.3098 lr=6.12e-05 tok=15.881B dt=4140 wait=119 +15:03:14 pushed ckpt_w107.pt,log_w107.txt +15:04:30 waited 76s for global r=415 +15:04:36 global r=415 load 4.297 -> 4.273 +15:05:40 step 120920 loss=3.3230 lr=6.12e-05 tok=15.883B dt=4302 wait=119 +15:06:43 step 120940 loss=3.3796 lr=6.12e-05 tok=15.886B dt=4365 wait=120 +15:07:47 step 120960 loss=3.3455 lr=6.12e-05 tok=15.889B dt=4429 wait=120 +15:08:50 step 120980 loss=3.2663 lr=6.12e-05 tok=15.891B dt=4492 wait=120 +15:09:53 step 121000 loss=3.2908 lr=6.11e-05 tok=15.894B dt=4555 wait=121 +15:10:56 step 121020 loss=3.4436 lr=6.11e-05 tok=15.896B dt=4618 wait=121 +15:11:59 step 121040 loss=3.2751 lr=6.11e-05 tok=15.899B dt=4681 wait=121 +15:12:49 pushed ckpt_w107.pt,log_w107.txt +15:13:51 waited 62s for global r=416 +15:13:58 global r=416 load 4.301 -> 4.281 +15:14:31 step 121060 loss=3.3114 lr=6.11e-05 tok=15.902B dt=4833 wait=122 +15:15:34 step 121080 loss=3.2452 lr=6.11e-05 tok=15.904B dt=4896 wait=122 +15:16:37 step 121100 loss=3.2460 lr=6.10e-05 tok=15.907B dt=4959 wait=122 +15:17:40 step 121120 loss=3.3482 lr=6.10e-05 tok=15.910B dt=5022 wait=123 +15:18:43 step 121140 loss=3.2513 lr=6.10e-05 tok=15.912B dt=5086 wait=123 +15:19:46 step 121160 loss=3.3653 lr=6.10e-05 tok=15.915B dt=5148 wait=123 +15:20:49 step 121180 loss=3.3153 lr=6.10e-05 tok=15.917B dt=5212 wait=124 +15:21:53 step 121200 loss=3.2911 lr=6.09e-05 tok=15.920B dt=5275 wait=124 +15:22:06 pushed ckpt_w107.pt,log_w107.txt +15:23:22 waited 76s for global r=417 +15:23:31 global r=417 load 4.309 -> 4.286 +15:24:35 step 121220 loss=3.2916 lr=6.09e-05 tok=15.923B dt=5437 wait=124 +15:25:38 step 121240 loss=3.3421 lr=6.09e-05 tok=15.925B dt=5501 wait=125 +15:26:42 step 121260 loss=3.3076 lr=6.09e-05 tok=15.928B dt=5564 wait=125 +15:27:45 step 121280 loss=3.2590 lr=6.09e-05 tok=15.930B dt=5627 wait=125 +15:28:48 step 121300 loss=3.3237 lr=6.08e-05 tok=15.933B dt=5690 wait=125 +15:29:52 step 121320 loss=3.3164 lr=6.08e-05 tok=15.936B dt=5754 wait=126 +15:30:55 step 121340 loss=3.2513 lr=6.08e-05 tok=15.938B dt=5817 wait=126 +15:31:46 pushed ckpt_w107.pt,log_w107.txt +15:33:02 waited 76s for global r=418 +15:33:10 global r=418 load 4.314 -> 4.291 +15:33:42 step 121360 loss=3.3465 lr=6.08e-05 tok=15.941B dt=5985 wait=126 +15:34:46 step 121380 loss=3.2937 lr=6.08e-05 tok=15.944B dt=6048 wait=127 +15:35:49 step 121400 loss=3.2034 lr=6.07e-05 tok=15.946B dt=6111 wait=127 +15:36:53 step 121420 loss=3.3756 lr=6.07e-05 tok=15.949B dt=6175 wait=127 +15:37:56 step 121440 loss=3.2436 lr=6.07e-05 tok=15.951B dt=6238 wait=128 +15:38:59 step 121460 loss=3.3930 lr=6.07e-05 tok=15.954B dt=6301 wait=128 +15:40:02 step 121480 loss=3.2276 lr=6.07e-05 tok=15.957B dt=6364 wait=128 +15:41:06 step 121500 loss=3.3596 lr=6.06e-05 tok=15.959B dt=6428 wait=129 +15:41:25 pushed ckpt_w107.pt,log_w107.txt +15:42:25 waited 61s for global r=419 +15:42:36 global r=419 load 4.318 -> 4.294 +15:43:40 step 121520 loss=3.2778 lr=6.06e-05 tok=15.962B dt=6582 wait=129 +15:44:43 step 121540 loss=3.3438 lr=6.06e-05 tok=15.965B dt=6645 wait=129 +15:45:46 step 121560 loss=3.3478 lr=6.06e-05 tok=15.967B dt=6708 wait=130 +15:46:49 step 121580 loss=3.2906 lr=6.06e-05 tok=15.970B dt=6771 wait=130 +15:47:52 step 121600 loss=3.3104 lr=6.05e-05 tok=15.972B dt=6834 wait=130 +15:48:56 step 121620 loss=3.4527 lr=6.05e-05 tok=15.975B dt=6898 wait=130 +15:49:59 step 121640 loss=3.2985 lr=6.05e-05 tok=15.978B dt=6961 wait=131 +15:50:46 pushed ckpt_w107.pt,log_w107.txt +15:52:01 waited 76s for global r=420 +15:52:09 global r=420 load 4.303 -> 4.302 +15:52:41 step 121660 loss=3.4183 lr=6.05e-05 tok=15.980B dt=7124 wait=131 +15:53:44 step 121680 loss=3.3312 lr=6.05e-05 tok=15.983B dt=7186 wait=131 +15:54:47 step 121700 loss=3.2950 lr=6.04e-05 tok=15.986B dt=7250 wait=132 +15:55:50 step 121720 loss=3.3966 lr=6.04e-05 tok=15.988B dt=7313 wait=132 +15:56:54 step 121740 loss=3.3030 lr=6.04e-05 tok=15.991B dt=7376 wait=132 +15:57:57 step 121760 loss=3.3810 lr=6.04e-05 tok=15.993B dt=7439 wait=133 +15:59:00 step 121780 loss=3.3230 lr=6.04e-05 tok=15.996B dt=7502 wait=133 +16:00:03 step 121800 loss=3.2067 lr=6.03e-05 tok=15.999B dt=7565 wait=133 +16:00:15 pushed ckpt_w107.pt,log_w107.txt +16:01:33 waited 78s for global r=421 +16:01:39 global r=421 load 4.315 -> 4.298 +16:02:44 step 121820 loss=3.3077 lr=6.03e-05 tok=16.001B dt=7726 wait=134 +16:03:48 step 121840 loss=3.0818 lr=6.03e-05 tok=16.004B dt=7790 wait=134 +16:04:51 step 121860 loss=3.3478 lr=6.03e-05 tok=16.007B dt=7853 wait=134 +16:05:54 step 121880 loss=3.2981 lr=6.03e-05 tok=16.009B dt=7916 wait=135 +16:06:57 step 121900 loss=3.3170 lr=6.02e-05 tok=16.012B dt=7979 wait=135 +16:08:00 step 121920 loss=3.3963 lr=6.02e-05 tok=16.014B dt=8043 wait=135 +16:09:04 step 121940 loss=3.3001 lr=6.02e-05 tok=16.017B dt=8106 wait=136 +16:09:47 pushed ckpt_w107.pt,log_w107.txt +16:11:19 waited 92s for global r=422 +16:11:26 global r=422 load 4.306 -> 4.295 +16:11:59 step 121960 loss=3.4057 lr=6.02e-05 tok=16.020B dt=8281 wait=136 +16:13:02 step 121980 loss=3.2901 lr=6.02e-05 tok=16.022B dt=8344 wait=136 +16:14:05 step 122000 loss=3.2983 lr=6.01e-05 tok=16.025B dt=8407 wait=137 +16:15:08 step 122020 loss=3.4200 lr=6.01e-05 tok=16.027B dt=8470 wait=137 +16:16:12 step 122040 loss=3.1852 lr=6.01e-05 tok=16.030B dt=8534 wait=137 +16:17:15 step 122060 loss=3.3840 lr=6.01e-05 tok=16.033B dt=8598 wait=138 +16:18:19 step 122080 loss=3.2558 lr=6.01e-05 tok=16.035B dt=8661 wait=138 +16:19:22 step 122100 loss=3.3549 lr=6.00e-05 tok=16.038B dt=8725 wait=138 +16:19:35 pushed ckpt_w107.pt,log_w107.txt +16:20:53 waited 78s for global r=423 +16:21:01 global r=423 load 4.286 -> 4.291 +16:22:05 step 122120 loss=3.2881 lr=6.00e-05 tok=16.041B dt=8887 wait=138 +16:23:08 step 122140 loss=3.3901 lr=6.00e-05 tok=16.043B dt=8950 wait=139 +16:24:11 step 122160 loss=3.3635 lr=6.00e-05 tok=16.046B dt=9013 wait=139 +16:25:14 step 122180 loss=3.2800 lr=6.00e-05 tok=16.048B dt=9076 wait=139 +16:26:17 step 122200 loss=3.2587 lr=5.99e-05 tok=16.051B dt=9139 wait=140 +16:27:20 step 122220 loss=3.3795 lr=5.99e-05 tok=16.054B dt=9202 wait=140 +16:28:23 step 122240 loss=3.2548 lr=5.99e-05 tok=16.056B dt=9266 wait=140 diff --git a/log_w109.txt b/log_w109.txt new file mode 100644 index 0000000000000000000000000000000000000000..f5d918f4bc6fb8b6ed6c8dacad94abbfb6d5386c --- /dev/null +++ b/log_w109.txt @@ -0,0 +1,176 @@ +launch +13:51:12 name=w109 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 56/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:51:19 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:51:19 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:51:28 params 1.08B +13:51:30 resumed step=119850 seen=15.74B round=408 +13:51:30 stream salt=119850 +13:53:14 blocks compiled +13:53:14 start: accum=4, 131072 tokens per optimiser step +13:55:50 step 119851 loss=3.2457 lr=6.23e-05 tok=15.743B dt=156 wait=113 +13:55:54 step 119852 loss=3.2718 lr=6.23e-05 tok=15.743B dt=160 wait=113 +13:56:22 step 119860 loss=3.5617 lr=6.23e-05 tok=15.744B dt=188 wait=113 +13:57:31 step 119880 loss=3.4063 lr=6.23e-05 tok=15.747B dt=257 wait=114 +13:58:35 step 119900 loss=3.4062 lr=6.22e-05 tok=15.750B dt=321 wait=114 +13:59:40 step 119920 loss=3.4372 lr=6.22e-05 tok=15.752B dt=385 wait=114 +14:00:44 step 119940 loss=3.4309 lr=6.22e-05 tok=15.755B dt=449 wait=114 +14:01:48 step 119960 loss=3.5463 lr=6.22e-05 tok=15.757B dt=513 wait=115 +14:02:52 step 119980 loss=3.3705 lr=6.22e-05 tok=15.760B dt=578 wait=115 +14:03:56 step 120000 loss=3.4369 lr=6.21e-05 tok=15.763B dt=641 wait=115 +14:04:13 pushed ckpt_w109.pt,log_w109.txt +14:06:59 waited 166s for global r=409 +14:07:12 global r=409 load 4.326 -> 4.279 +14:08:39 step 120020 loss=3.3848 lr=6.21e-05 tok=15.765B dt=924 wait=116 +14:09:42 step 120040 loss=3.3962 lr=6.21e-05 tok=15.768B dt=988 wait=116 +14:10:46 step 120060 loss=3.4102 lr=6.21e-05 tok=15.771B dt=1052 wait=117 +14:11:50 step 120080 loss=3.4065 lr=6.21e-05 tok=15.773B dt=1116 wait=117 +14:12:53 step 120100 loss=3.3154 lr=6.20e-05 tok=15.776B dt=1179 wait=117 +14:13:57 step 120120 loss=3.4706 lr=6.20e-05 tok=15.778B dt=1243 wait=117 +14:15:01 step 120140 loss=3.3817 lr=6.20e-05 tok=15.781B dt=1307 wait=118 +14:15:45 pushed ckpt_w109.pt,log_w109.txt +14:16:31 waited 45s for global r=410 +14:16:40 global r=410 load 4.296 -> 4.291 +14:17:22 step 120160 loss=3.4366 lr=6.20e-05 tok=15.784B dt=1448 wait=118 +14:18:38 step 120180 loss=3.3474 lr=6.20e-05 tok=15.786B dt=1524 wait=118 +14:19:42 step 120200 loss=3.3378 lr=6.19e-05 tok=15.789B dt=1588 wait=119 +14:20:46 step 120220 loss=3.4663 lr=6.19e-05 tok=15.792B dt=1651 wait=119 +14:21:50 step 120240 loss=3.3397 lr=6.19e-05 tok=15.794B dt=1715 wait=119 +14:22:53 step 120260 loss=3.4308 lr=6.19e-05 tok=15.797B dt=1779 wait=120 +14:23:57 step 120280 loss=3.3605 lr=6.19e-05 tok=15.799B dt=1842 wait=120 +14:25:00 step 120300 loss=3.4186 lr=6.18e-05 tok=15.802B dt=1906 wait=120 +14:25:15 pushed ckpt_w109.pt,log_w109.txt +14:26:15 waited 61s for global r=411 +14:26:21 global r=411 load 4.286 -> 4.277 +14:27:46 step 120320 loss=3.4026 lr=6.18e-05 tok=15.805B dt=2071 wait=121 +14:28:49 step 120340 loss=3.3793 lr=6.18e-05 tok=15.807B dt=2135 wait=121 +14:29:53 step 120360 loss=3.4084 lr=6.18e-05 tok=15.810B dt=2199 wait=121 +14:30:57 step 120380 loss=3.4350 lr=6.18e-05 tok=15.813B dt=2263 wait=122 +14:32:01 step 120400 loss=3.3327 lr=6.17e-05 tok=15.815B dt=2326 wait=122 +14:33:05 step 120420 loss=3.4988 lr=6.17e-05 tok=15.818B dt=2391 wait=122 +14:34:09 step 120440 loss=3.3475 lr=6.17e-05 tok=15.820B dt=2455 wait=123 +14:34:53 pushed ckpt_w109.pt,log_w109.txt +14:35:38 waited 46s for global r=412 +14:35:44 global r=412 load 4.296 -> 4.284 +14:36:25 step 120460 loss=3.4341 lr=6.17e-05 tok=15.823B dt=2591 wait=123 +14:37:41 step 120480 loss=3.3928 lr=6.17e-05 tok=15.826B dt=2667 wait=123 +14:38:45 step 120500 loss=3.4093 lr=6.16e-05 tok=15.828B dt=2730 wait=124 +14:39:49 step 120520 loss=3.4857 lr=6.16e-05 tok=15.831B dt=2794 wait=124 +14:40:53 step 120540 loss=3.3013 lr=6.16e-05 tok=15.833B dt=2858 wait=124 +14:41:57 step 120560 loss=3.4625 lr=6.16e-05 tok=15.836B dt=2922 wait=125 +14:43:01 step 120580 loss=3.4085 lr=6.16e-05 tok=15.839B dt=2986 wait=125 +14:44:05 step 120600 loss=3.4373 lr=6.15e-05 tok=15.841B dt=3050 wait=125 +14:44:18 pushed ckpt_w109.pt,log_w109.txt +14:45:25 waited 66s for global r=413 +14:45:31 global r=413 load 4.300 -> 4.292 +14:46:57 step 120620 loss=3.3436 lr=6.15e-05 tok=15.844B dt=3222 wait=126 +14:48:01 step 120640 loss=3.4327 lr=6.15e-05 tok=15.847B dt=3286 wait=126 +14:49:04 step 120660 loss=3.4146 lr=6.15e-05 tok=15.849B dt=3350 wait=126 +14:50:09 step 120680 loss=3.4160 lr=6.15e-05 tok=15.852B dt=3414 wait=127 +14:51:12 step 120700 loss=3.3946 lr=6.14e-05 tok=15.854B dt=3478 wait=127 +14:52:16 step 120720 loss=3.3871 lr=6.14e-05 tok=15.857B dt=3542 wait=127 +14:53:20 step 120740 loss=3.3390 lr=6.14e-05 tok=15.860B dt=3606 wait=127 +14:54:05 pushed ckpt_w109.pt,log_w109.txt +14:54:50 waited 45s for global r=414 +14:54:57 global r=414 load 4.290 -> 4.279 +14:55:40 step 120760 loss=3.4454 lr=6.14e-05 tok=15.862B dt=3745 wait=128 +14:56:55 step 120780 loss=3.3698 lr=6.14e-05 tok=15.865B dt=3821 wait=128 +14:57:59 step 120800 loss=3.3626 lr=6.13e-05 tok=15.868B dt=3885 wait=128 +14:59:03 step 120820 loss=3.4085 lr=6.13e-05 tok=15.870B dt=3949 wait=129 +15:00:08 step 120840 loss=3.2343 lr=6.13e-05 tok=15.873B dt=4014 wait=129 +15:01:12 step 120860 loss=3.4681 lr=6.13e-05 tok=15.875B dt=4078 wait=129 +15:02:16 step 120880 loss=3.3647 lr=6.13e-05 tok=15.878B dt=4142 wait=130 +15:03:20 step 120900 loss=3.4025 lr=6.12e-05 tok=15.881B dt=4206 wait=130 +15:03:34 pushed ckpt_w109.pt,log_w109.txt +15:04:19 waited 46s for global r=415 +15:04:26 global r=415 load 4.293 -> 4.273 +15:05:45 step 120920 loss=3.3616 lr=6.12e-05 tok=15.883B dt=4351 wait=130 +15:06:54 step 120940 loss=3.4129 lr=6.12e-05 tok=15.886B dt=4420 wait=131 +15:07:57 step 120960 loss=3.4056 lr=6.12e-05 tok=15.889B dt=4483 wait=131 +15:09:01 step 120980 loss=3.3489 lr=6.12e-05 tok=15.891B dt=4547 wait=131 +15:10:05 step 121000 loss=3.3355 lr=6.11e-05 tok=15.894B dt=4611 wait=132 +15:11:09 step 121020 loss=3.4514 lr=6.11e-05 tok=15.896B dt=4674 wait=132 +15:12:12 step 121040 loss=3.3244 lr=6.11e-05 tok=15.899B dt=4737 wait=132 +15:12:57 pushed ckpt_w109.pt,log_w109.txt +15:13:57 waited 61s for global r=416 +15:14:05 global r=416 load 4.283 -> 4.281 +15:14:47 step 121060 loss=3.4261 lr=6.11e-05 tok=15.902B dt=4892 wait=133 +15:16:02 step 121080 loss=3.3004 lr=6.11e-05 tok=15.904B dt=4967 wait=133 +15:17:05 step 121100 loss=3.2979 lr=6.10e-05 tok=15.907B dt=5030 wait=133 +15:18:08 step 121120 loss=3.4005 lr=6.10e-05 tok=15.910B dt=5094 wait=133 +15:19:11 step 121140 loss=3.2543 lr=6.10e-05 tok=15.912B dt=5157 wait=134 +15:20:14 step 121160 loss=3.3464 lr=6.10e-05 tok=15.915B dt=5220 wait=134 +15:21:18 step 121180 loss=3.2993 lr=6.10e-05 tok=15.917B dt=5283 wait=134 +15:22:21 step 121200 loss=3.2745 lr=6.09e-05 tok=15.920B dt=5347 wait=135 +15:22:37 pushed ckpt_w109.pt,log_w109.txt +15:23:22 waited 45s for global r=417 +15:23:29 global r=417 load 4.280 -> 4.286 +15:24:42 step 121220 loss=3.3063 lr=6.09e-05 tok=15.923B dt=5487 wait=135 +15:25:57 step 121240 loss=3.2948 lr=6.09e-05 tok=15.925B dt=5563 wait=135 +15:27:00 step 121260 loss=3.2830 lr=6.09e-05 tok=15.928B dt=5626 wait=136 +15:28:04 step 121280 loss=3.3082 lr=6.09e-05 tok=15.930B dt=5689 wait=136 +15:29:07 step 121300 loss=3.2973 lr=6.08e-05 tok=15.933B dt=5753 wait=136 +15:30:10 step 121320 loss=3.3965 lr=6.08e-05 tok=15.936B dt=5816 wait=137 +15:31:14 step 121340 loss=3.2671 lr=6.08e-05 tok=15.938B dt=5880 wait=137 +15:31:58 pushed ckpt_w109.pt,log_w109.txt +15:32:58 waited 60s for global r=418 +15:33:06 global r=418 load 4.299 -> 4.291 +15:33:44 step 121360 loss=3.4334 lr=6.08e-05 tok=15.941B dt=6030 wait=137 +15:35:01 step 121380 loss=3.3458 lr=6.08e-05 tok=15.944B dt=6107 wait=138 +15:36:05 step 121400 loss=3.3444 lr=6.07e-05 tok=15.946B dt=6171 wait=138 +15:37:08 step 121420 loss=3.4183 lr=6.07e-05 tok=15.949B dt=6234 wait=138 +15:38:12 step 121440 loss=3.3051 lr=6.07e-05 tok=15.951B dt=6297 wait=138 +15:39:15 step 121460 loss=3.4082 lr=6.07e-05 tok=15.954B dt=6361 wait=139 +15:40:19 step 121480 loss=3.3226 lr=6.07e-05 tok=15.957B dt=6425 wait=139 +15:41:22 step 121500 loss=3.3784 lr=6.06e-05 tok=15.959B dt=6488 wait=139 +15:41:35 pushed ckpt_w109.pt,log_w109.txt +15:42:20 waited 45s for global r=419 +15:42:29 global r=419 load 4.311 -> 4.294 +15:43:41 step 121520 loss=3.2991 lr=6.06e-05 tok=15.962B dt=6627 wait=140 +15:44:56 step 121540 loss=3.4056 lr=6.06e-05 tok=15.965B dt=6702 wait=140 +15:45:59 step 121560 loss=3.3491 lr=6.06e-05 tok=15.967B dt=6765 wait=140 +15:47:03 step 121580 loss=3.2819 lr=6.06e-05 tok=15.970B dt=6828 wait=141 +15:48:06 step 121600 loss=3.3226 lr=6.05e-05 tok=15.972B dt=6892 wait=141 +15:49:10 step 121620 loss=3.3609 lr=6.05e-05 tok=15.975B dt=6955 wait=141 +15:50:13 step 121640 loss=3.2847 lr=6.05e-05 tok=15.978B dt=7019 wait=142 +15:50:57 pushed ckpt_w109.pt,log_w109.txt +15:51:57 waited 61s for global r=420 +15:52:03 global r=420 load 4.289 -> 4.302 +15:52:37 step 121660 loss=3.4181 lr=6.05e-05 tok=15.980B dt=7162 wait=142 +15:53:59 step 121680 loss=3.3008 lr=6.05e-05 tok=15.983B dt=7245 wait=142 +15:55:03 step 121700 loss=3.3202 lr=6.04e-05 tok=15.986B dt=7308 wait=143 +15:56:07 step 121720 loss=3.4076 lr=6.04e-05 tok=15.988B dt=7372 wait=143 +15:57:10 step 121740 loss=3.2986 lr=6.04e-05 tok=15.991B dt=7436 wait=143 +15:58:14 step 121760 loss=3.3764 lr=6.04e-05 tok=15.993B dt=7499 wait=144 +15:59:17 step 121780 loss=3.3142 lr=6.04e-05 tok=15.996B dt=7563 wait=144 +16:00:20 step 121800 loss=3.3403 lr=6.03e-05 tok=15.999B dt=7626 wait=144 +16:00:34 pushed ckpt_w109.pt,log_w109.txt +16:01:35 waited 61s for global r=421 +16:01:42 global r=421 load 4.344 -> 4.298 +16:02:55 step 121820 loss=3.3303 lr=6.03e-05 tok=16.001B dt=7781 wait=145 +16:04:10 step 121840 loss=3.3744 lr=6.03e-05 tok=16.004B dt=7856 wait=145 +16:05:13 step 121860 loss=3.3686 lr=6.03e-05 tok=16.007B dt=7919 wait=145 +16:06:17 step 121880 loss=3.3072 lr=6.03e-05 tok=16.009B dt=7982 wait=146 +16:07:20 step 121900 loss=3.0807 lr=6.02e-05 tok=16.012B dt=8046 wait=146 +16:08:23 step 121920 loss=3.4030 lr=6.02e-05 tok=16.014B dt=8109 wait=146 +16:09:27 step 121940 loss=3.2515 lr=6.02e-05 tok=16.017B dt=8172 wait=147 +16:10:13 pushed ckpt_w109.pt,log_w109.txt +16:11:13 waited 61s for global r=422 +16:11:20 global r=422 load 4.312 -> 4.295 +16:11:59 step 121960 loss=3.4102 lr=6.02e-05 tok=16.020B dt=8324 wait=147 +16:13:16 step 121980 loss=3.2821 lr=6.02e-05 tok=16.022B dt=8401 wait=147 +16:14:19 step 122000 loss=3.2520 lr=6.01e-05 tok=16.025B dt=8465 wait=148 +16:15:23 step 122020 loss=3.4529 lr=6.01e-05 tok=16.027B dt=8528 wait=148 +16:16:26 step 122040 loss=3.3172 lr=6.01e-05 tok=16.030B dt=8592 wait=148 +16:17:30 step 122060 loss=3.4268 lr=6.01e-05 tok=16.033B dt=8656 wait=149 +16:18:33 step 122080 loss=3.3550 lr=6.01e-05 tok=16.035B dt=8719 wait=149 +16:19:37 step 122100 loss=3.3450 lr=6.00e-05 tok=16.038B dt=8782 wait=149 +16:19:50 pushed ckpt_w109.pt,log_w109.txt +16:20:51 waited 61s for global r=423 +16:21:00 global r=423 load 4.300 -> 4.291 +16:22:20 step 122120 loss=3.3326 lr=6.00e-05 tok=16.041B dt=8945 wait=150 +16:23:27 step 122140 loss=3.3879 lr=6.00e-05 tok=16.043B dt=9013 wait=150 +16:24:31 step 122160 loss=3.3017 lr=6.00e-05 tok=16.046B dt=9077 wait=150 +16:25:34 step 122180 loss=3.2979 lr=6.00e-05 tok=16.048B dt=9140 wait=151 +16:26:37 step 122200 loss=3.3536 lr=5.99e-05 tok=16.051B dt=9203 wait=151 +16:27:41 step 122220 loss=3.4064 lr=5.99e-05 tok=16.054B dt=9266 wait=151 +16:28:44 step 122240 loss=3.3040 lr=5.99e-05 tok=16.056B dt=9329 wait=152 diff --git a/log_w11.txt b/log_w11.txt new file mode 100644 index 0000000000000000000000000000000000000000..0b1a544c3078422fe51c3a391c7998ebecad3e46 --- /dev/null +++ b/log_w11.txt @@ -0,0 +1,38 @@ +launch +00:42:00 name=w11 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +00:42:06 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +00:42:15 params 1.08B +00:42:17 resumed step=34200 seen=4.48B round=16 +00:42:17 stream salt=34200 +00:44:08 start: accum=4, 131072 tokens per optimiser step +00:46:42 step 34220 loss=3.6868 lr=2.82e-04 tok=4.485B dt=153 +00:48:07 step 34240 loss=3.8478 lr=2.81e-04 tok=4.488B dt=239 +00:49:23 step 34260 loss=3.7780 lr=2.81e-04 tok=4.491B dt=314 +00:50:39 step 34280 loss=3.8302 lr=2.81e-04 tok=4.493B dt=390 +00:51:54 step 34300 loss=3.7231 lr=2.81e-04 tok=4.496B dt=466 +00:53:10 step 34320 loss=3.7785 lr=2.81e-04 tok=4.498B dt=541 +00:54:26 step 34340 loss=3.7900 lr=2.81e-04 tok=4.501B dt=617 +00:55:42 step 34360 loss=3.7989 lr=2.81e-04 tok=4.504B dt=693 +00:56:57 step 34380 loss=3.6834 lr=2.81e-04 tok=4.506B dt=769 +00:58:13 step 34400 loss=3.7871 lr=2.81e-04 tok=4.509B dt=844 +00:59:28 step 34420 loss=3.7258 lr=2.81e-04 tok=4.511B dt=920 +01:00:44 step 34440 loss=3.7002 lr=2.81e-04 tok=4.514B dt=996 +01:02:00 step 34460 loss=3.7448 lr=2.81e-04 tok=4.517B dt=1071 +01:03:15 step 34480 loss=3.6907 lr=2.81e-04 tok=4.519B dt=1147 +01:04:31 step 34500 loss=3.7706 lr=2.81e-04 tok=4.522B dt=1223 +01:05:47 step 34520 loss=3.8426 lr=2.81e-04 tok=4.525B dt=1298 +01:07:02 step 34540 loss=3.7975 lr=2.81e-04 tok=4.527B dt=1374 +01:08:18 step 34560 loss=3.7845 lr=2.81e-04 tok=4.530B dt=1450 +01:09:34 step 34580 loss=3.8485 lr=2.81e-04 tok=4.532B dt=1526 +01:10:50 step 34600 loss=3.7856 lr=2.81e-04 tok=4.535B dt=1601 +01:12:05 step 34620 loss=3.7488 lr=2.81e-04 tok=4.538B dt=1677 +01:13:21 step 34640 loss=3.6618 lr=2.81e-04 tok=4.540B dt=1753 +01:14:37 step 34660 loss=3.7533 lr=2.81e-04 tok=4.543B dt=1828 +01:15:52 step 34680 loss=3.6644 lr=2.81e-04 tok=4.546B dt=1904 +01:17:08 step 34700 loss=3.7690 lr=2.81e-04 tok=4.548B dt=1979 +01:18:23 step 34720 loss=3.7228 lr=2.81e-04 tok=4.551B dt=2055 +01:19:39 step 34740 loss=3.6967 lr=2.81e-04 tok=4.553B dt=2130 +01:20:55 step 34760 loss=3.7637 lr=2.81e-04 tok=4.556B dt=2206 +01:22:10 step 34780 loss=3.6501 lr=2.81e-04 tok=4.559B dt=2282 +01:23:26 step 34800 loss=3.7580 lr=2.81e-04 tok=4.561B dt=2357 +01:23:40 pushed ckpt_w11.pt diff --git a/log_w110.txt b/log_w110.txt new file mode 100644 index 0000000000000000000000000000000000000000..c55729aa4e6d8947fc24d26ab213846bd53b1006 --- /dev/null +++ b/log_w110.txt @@ -0,0 +1,587 @@ +launch +13:51:21 name=w110 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 57/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:51:27 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:51:27 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:51:37 params 1.08B +13:51:39 resumed step=119850 seen=15.74B round=408 +13:51:39 stream salt=119850 +13:53:22 blocks compiled +13:53:22 start: accum=4, 131072 tokens per optimiser step +13:55:57 step 119851 loss=3.2733 lr=6.23e-05 tok=15.743B dt=156 wait=113 +13:56:01 step 119852 loss=3.3165 lr=6.23e-05 tok=15.743B dt=160 wait=113 +13:56:29 step 119860 loss=3.5064 lr=6.23e-05 tok=15.744B dt=187 wait=114 +13:57:37 step 119880 loss=3.3857 lr=6.23e-05 tok=15.747B dt=255 wait=114 +13:58:41 step 119900 loss=3.4123 lr=6.22e-05 tok=15.750B dt=320 wait=114 +13:59:45 step 119920 loss=3.3707 lr=6.22e-05 tok=15.752B dt=384 wait=115 +14:00:50 step 119940 loss=3.4965 lr=6.22e-05 tok=15.755B dt=448 wait=115 +14:01:54 step 119960 loss=3.5440 lr=6.22e-05 tok=15.757B dt=512 wait=115 +14:02:59 step 119980 loss=3.4454 lr=6.22e-05 tok=15.760B dt=577 wait=116 +14:04:03 step 120000 loss=3.4777 lr=6.21e-05 tok=15.763B dt=641 wait=116 +14:04:18 pushed ckpt_w110.pt,log_w110.txt +14:07:04 waited 166s for global r=409 +14:07:14 global r=409 load 4.307 -> 4.279 +14:08:21 step 120020 loss=3.3349 lr=6.21e-05 tok=15.765B dt=900 wait=116 +14:09:26 step 120040 loss=3.3647 lr=6.21e-05 tok=15.768B dt=964 wait=117 +14:10:29 step 120060 loss=3.4617 lr=6.21e-05 tok=15.771B dt=1028 wait=117 +14:11:34 step 120080 loss=3.3259 lr=6.21e-05 tok=15.773B dt=1092 wait=117 +14:12:38 step 120100 loss=3.3988 lr=6.20e-05 tok=15.776B dt=1156 wait=118 +14:13:42 step 120120 loss=3.4524 lr=6.20e-05 tok=15.778B dt=1220 wait=118 +14:14:46 step 120140 loss=3.3192 lr=6.20e-05 tok=15.781B dt=1284 wait=118 +14:15:32 pushed ckpt_w110.pt,log_w110.txt +14:16:32 waited 60s for global r=410 +14:16:39 global r=410 load 4.321 -> 4.291 +14:17:12 step 120160 loss=3.4523 lr=6.20e-05 tok=15.784B dt=1431 wait=119 +14:18:17 step 120180 loss=3.3603 lr=6.20e-05 tok=15.786B dt=1495 wait=119 +14:19:21 step 120200 loss=3.3025 lr=6.19e-05 tok=15.789B dt=1559 wait=119 +14:20:25 step 120220 loss=3.4277 lr=6.19e-05 tok=15.792B dt=1623 wait=120 +14:21:29 step 120240 loss=3.3361 lr=6.19e-05 tok=15.794B dt=1687 wait=120 +14:22:33 step 120260 loss=3.3968 lr=6.19e-05 tok=15.797B dt=1751 wait=120 +14:23:37 step 120280 loss=3.3763 lr=6.19e-05 tok=15.799B dt=1815 wait=121 +14:24:41 step 120300 loss=3.3773 lr=6.18e-05 tok=15.802B dt=1879 wait=121 +14:24:53 pushed ckpt_w110.pt,log_w110.txt +14:26:09 waited 76s for global r=411 +14:26:15 global r=411 load 4.290 -> 4.277 +14:27:21 step 120320 loss=3.3627 lr=6.18e-05 tok=15.805B dt=2039 wait=121 +14:28:24 step 120340 loss=3.3550 lr=6.18e-05 tok=15.807B dt=2103 wait=121 +14:29:28 step 120360 loss=3.3543 lr=6.18e-05 tok=15.810B dt=2166 wait=122 +14:30:32 step 120380 loss=3.3703 lr=6.18e-05 tok=15.813B dt=2230 wait=122 +14:31:36 step 120400 loss=3.3491 lr=6.17e-05 tok=15.815B dt=2294 wait=122 +14:32:40 step 120420 loss=3.4287 lr=6.17e-05 tok=15.818B dt=2358 wait=123 +14:33:44 step 120440 loss=3.3338 lr=6.17e-05 tok=15.820B dt=2422 wait=123 +14:34:30 pushed ckpt_w110.pt,log_w110.txt +14:35:45 waited 76s for global r=412 +14:35:53 global r=412 load 4.276 -> 4.284 +14:36:26 step 120460 loss=3.4094 lr=6.17e-05 tok=15.823B dt=2585 wait=123 +14:37:30 step 120480 loss=3.3189 lr=6.17e-05 tok=15.826B dt=2649 wait=124 +14:38:34 step 120500 loss=3.3571 lr=6.16e-05 tok=15.828B dt=2713 wait=124 +14:39:38 step 120520 loss=3.3915 lr=6.16e-05 tok=15.831B dt=2777 wait=124 +14:40:43 step 120540 loss=3.3905 lr=6.16e-05 tok=15.833B dt=2841 wait=125 +14:41:47 step 120560 loss=3.4160 lr=6.16e-05 tok=15.836B dt=2905 wait=125 +14:42:51 step 120580 loss=3.3573 lr=6.16e-05 tok=15.839B dt=2969 wait=125 +14:43:56 step 120600 loss=3.4350 lr=6.15e-05 tok=15.841B dt=3034 wait=126 +14:44:08 pushed ckpt_w110.pt,log_w110.txt +14:45:23 waited 76s for global r=413 +14:45:29 global r=413 load 4.307 -> 4.292 +14:46:34 step 120620 loss=3.3304 lr=6.15e-05 tok=15.844B dt=3193 wait=126 +14:47:40 step 120640 loss=3.3955 lr=6.15e-05 tok=15.847B dt=3258 wait=126 +14:48:44 step 120660 loss=3.4252 lr=6.15e-05 tok=15.849B dt=3322 wait=127 +14:49:49 step 120680 loss=3.3189 lr=6.15e-05 tok=15.852B dt=3387 wait=127 +14:50:53 step 120700 loss=3.3910 lr=6.14e-05 tok=15.854B dt=3451 wait=127 +14:51:57 step 120720 loss=3.3988 lr=6.14e-05 tok=15.857B dt=3516 wait=128 +14:53:02 step 120740 loss=3.2999 lr=6.14e-05 tok=15.860B dt=3580 wait=128 +14:53:48 pushed ckpt_w110.pt,log_w110.txt +14:54:48 waited 60s for global r=414 +14:54:55 global r=414 load 4.286 -> 4.279 +14:55:29 step 120760 loss=3.5070 lr=6.14e-05 tok=15.862B dt=3727 wait=128 +14:56:33 step 120780 loss=3.3384 lr=6.14e-05 tok=15.865B dt=3791 wait=128 +14:57:37 step 120800 loss=3.3801 lr=6.13e-05 tok=15.868B dt=3855 wait=129 +14:58:41 step 120820 loss=3.5102 lr=6.13e-05 tok=15.870B dt=3919 wait=129 +14:59:45 step 120840 loss=3.3607 lr=6.13e-05 tok=15.873B dt=3983 wait=129 +15:00:50 step 120860 loss=3.5070 lr=6.13e-05 tok=15.875B dt=4048 wait=130 +15:01:54 step 120880 loss=3.3810 lr=6.13e-05 tok=15.878B dt=4113 wait=130 +15:02:58 step 120900 loss=3.3725 lr=6.12e-05 tok=15.881B dt=4177 wait=130 +15:03:11 pushed ckpt_w110.pt,log_w110.txt +15:04:27 waited 76s for global r=415 +15:04:32 global r=415 load 4.278 -> 4.273 +15:05:37 step 120920 loss=3.3658 lr=6.12e-05 tok=15.883B dt=4335 wait=131 +15:06:41 step 120940 loss=3.3989 lr=6.12e-05 tok=15.886B dt=4399 wait=131 +15:07:45 step 120960 loss=3.3790 lr=6.12e-05 tok=15.889B dt=4463 wait=131 +15:08:49 step 120980 loss=3.3685 lr=6.12e-05 tok=15.891B dt=4527 wait=132 +15:09:53 step 121000 loss=3.3964 lr=6.11e-05 tok=15.894B dt=4591 wait=132 +15:10:56 step 121020 loss=3.4340 lr=6.11e-05 tok=15.896B dt=4655 wait=132 +15:12:00 step 121040 loss=3.2783 lr=6.11e-05 tok=15.899B dt=4718 wait=133 +15:12:46 pushed ckpt_w110.pt,log_w110.txt +15:13:46 waited 60s for global r=416 +15:13:54 global r=416 load 4.315 -> 4.281 +15:14:27 step 121060 loss=3.3761 lr=6.11e-05 tok=15.902B dt=4865 wait=133 +15:15:31 step 121080 loss=3.3215 lr=6.11e-05 tok=15.904B dt=4929 wait=133 +15:16:34 step 121100 loss=3.2700 lr=6.10e-05 tok=15.907B dt=4992 wait=134 +15:17:38 step 121120 loss=3.3715 lr=6.10e-05 tok=15.910B dt=5056 wait=134 +15:18:41 step 121140 loss=3.2902 lr=6.10e-05 tok=15.912B dt=5119 wait=134 +15:19:45 step 121160 loss=3.3673 lr=6.10e-05 tok=15.915B dt=5183 wait=135 +15:20:48 step 121180 loss=3.3361 lr=6.10e-05 tok=15.917B dt=5246 wait=135 +15:21:52 step 121200 loss=3.3888 lr=6.09e-05 tok=15.920B dt=5310 wait=135 +15:22:05 pushed ckpt_w110.pt,log_w110.txt +15:23:21 waited 76s for global r=417 +15:23:28 global r=417 load 4.296 -> 4.286 +15:24:33 step 121220 loss=3.3077 lr=6.09e-05 tok=15.923B dt=5471 wait=136 +15:25:37 step 121240 loss=3.3261 lr=6.09e-05 tok=15.925B dt=5535 wait=136 +15:26:40 step 121260 loss=3.2929 lr=6.09e-05 tok=15.928B dt=5598 wait=136 +15:27:44 step 121280 loss=3.2930 lr=6.09e-05 tok=15.930B dt=5662 wait=136 +15:28:47 step 121300 loss=3.2916 lr=6.08e-05 tok=15.933B dt=5725 wait=137 +15:29:50 step 121320 loss=3.4089 lr=6.08e-05 tok=15.936B dt=5789 wait=137 +15:30:54 step 121340 loss=3.2525 lr=6.08e-05 tok=15.938B dt=5852 wait=137 +15:31:38 pushed ckpt_w110.pt,log_w110.txt +15:32:53 waited 76s for global r=418 +15:32:58 global r=418 load 4.314 -> 4.291 +15:33:31 step 121360 loss=3.3620 lr=6.08e-05 tok=15.941B dt=6009 wait=138 +15:34:35 step 121380 loss=3.2887 lr=6.08e-05 tok=15.944B dt=6073 wait=138 +15:35:38 step 121400 loss=3.3016 lr=6.07e-05 tok=15.946B dt=6136 wait=138 +15:36:41 step 121420 loss=3.3834 lr=6.07e-05 tok=15.949B dt=6199 wait=139 +15:37:45 step 121440 loss=3.1954 lr=6.07e-05 tok=15.951B dt=6263 wait=139 +15:38:49 step 121460 loss=3.2977 lr=6.07e-05 tok=15.954B dt=6327 wait=139 +15:39:53 step 121480 loss=3.3284 lr=6.07e-05 tok=15.957B dt=6391 wait=139 +15:40:56 step 121500 loss=3.3556 lr=6.06e-05 tok=15.959B dt=6455 wait=140 +15:41:09 pushed ckpt_w110.pt,log_w110.txt +15:42:25 waited 76s for global r=419 +15:42:30 global r=419 load 4.298 -> 4.294 +15:43:35 step 121520 loss=3.3603 lr=6.06e-05 tok=15.962B dt=6613 wait=140 +15:44:38 step 121540 loss=3.3915 lr=6.06e-05 tok=15.965B dt=6676 wait=140 +15:45:42 step 121560 loss=3.2609 lr=6.06e-05 tok=15.967B dt=6740 wait=141 +15:46:45 step 121580 loss=3.2235 lr=6.06e-05 tok=15.970B dt=6803 wait=141 +15:47:48 step 121600 loss=3.3026 lr=6.05e-05 tok=15.972B dt=6867 wait=141 +15:48:52 step 121620 loss=3.4132 lr=6.05e-05 tok=15.975B dt=6931 wait=142 +15:49:56 step 121640 loss=3.2859 lr=6.05e-05 tok=15.978B dt=6994 wait=142 +15:50:42 pushed ckpt_w110.pt,log_w110.txt +15:51:58 waited 76s for global r=420 +15:52:03 global r=420 load 4.322 -> 4.302 +15:52:36 step 121660 loss=3.4485 lr=6.05e-05 tok=15.980B dt=7155 wait=142 +15:53:40 step 121680 loss=3.2745 lr=6.05e-05 tok=15.983B dt=7218 wait=142 +15:54:44 step 121700 loss=3.3629 lr=6.04e-05 tok=15.986B dt=7282 wait=143 +15:55:48 step 121720 loss=3.4222 lr=6.04e-05 tok=15.988B dt=7346 wait=143 +15:56:52 step 121740 loss=3.2146 lr=6.04e-05 tok=15.991B dt=7410 wait=143 +15:57:55 step 121760 loss=3.4347 lr=6.04e-05 tok=15.993B dt=7473 wait=144 +15:58:59 step 121780 loss=3.3454 lr=6.04e-05 tok=15.996B dt=7537 wait=144 +16:00:02 step 121800 loss=3.3785 lr=6.03e-05 tok=15.999B dt=7600 wait=144 +16:00:14 pushed ckpt_w110.pt,log_w110.txt +16:01:30 waited 76s for global r=421 +16:01:36 global r=421 load 4.290 -> 4.298 +16:02:41 step 121820 loss=3.2545 lr=6.03e-05 tok=16.001B dt=7760 wait=145 +16:03:45 step 121840 loss=3.3172 lr=6.03e-05 tok=16.004B dt=7823 wait=145 +16:04:48 step 121860 loss=3.3535 lr=6.03e-05 tok=16.007B dt=7886 wait=145 +16:05:52 step 121880 loss=3.2842 lr=6.03e-05 tok=16.009B dt=7950 wait=146 +16:06:55 step 121900 loss=3.3510 lr=6.02e-05 tok=16.012B dt=8013 wait=146 +16:07:58 step 121920 loss=3.3875 lr=6.02e-05 tok=16.014B dt=8077 wait=146 +16:09:02 step 121940 loss=3.2433 lr=6.02e-05 tok=16.017B dt=8140 wait=147 +16:09:46 pushed ckpt_w110.pt,log_w110.txt +16:11:16 waited 91s for global r=422 +16:11:22 global r=422 load 4.312 -> 4.295 +16:11:55 step 121960 loss=3.4070 lr=6.02e-05 tok=16.020B dt=8314 wait=147 +16:12:59 step 121980 loss=3.2936 lr=6.02e-05 tok=16.022B dt=8377 wait=147 +16:14:03 step 122000 loss=3.3313 lr=6.01e-05 tok=16.025B dt=8441 wait=148 +16:15:07 step 122020 loss=3.4360 lr=6.01e-05 tok=16.027B dt=8506 wait=148 +16:16:11 step 122040 loss=3.2587 lr=6.01e-05 tok=16.030B dt=8569 wait=148 +16:17:15 step 122060 loss=3.3940 lr=6.01e-05 tok=16.033B dt=8633 wait=149 +16:18:18 step 122080 loss=3.3547 lr=6.01e-05 tok=16.035B dt=8697 wait=149 +16:19:22 step 122100 loss=3.3379 lr=6.00e-05 tok=16.038B dt=8760 wait=149 +16:19:38 pushed ckpt_w110.pt,log_w110.txt +16:20:53 waited 76s for global r=423 +16:21:00 global r=423 load 4.327 -> 4.291 +16:22:05 step 122120 loss=3.2727 lr=6.00e-05 tok=16.041B dt=8923 wait=150 +16:23:08 step 122140 loss=3.3491 lr=6.00e-05 tok=16.043B dt=8987 wait=150 +16:24:12 step 122160 loss=3.2807 lr=6.00e-05 tok=16.046B dt=9050 wait=150 +16:25:15 step 122180 loss=3.2746 lr=6.00e-05 tok=16.048B dt=9114 wait=151 +16:26:19 step 122200 loss=3.3032 lr=5.99e-05 tok=16.051B dt=9177 wait=151 +16:27:23 step 122220 loss=3.3810 lr=5.99e-05 tok=16.054B dt=9241 wait=151 +16:28:26 step 122240 loss=3.1842 lr=5.99e-05 tok=16.056B dt=9305 wait=152 +16:29:10 pushed ckpt_w110.pt,log_w110.txt +16:30:25 waited 76s for global r=424 +16:30:32 global r=424 load 4.288 -> 4.289 +16:31:05 step 122260 loss=3.4244 lr=5.99e-05 tok=16.059B dt=9463 wait=152 +16:32:09 step 122280 loss=3.2628 lr=5.99e-05 tok=16.062B dt=9527 wait=152 +16:33:12 step 122300 loss=3.2999 lr=5.98e-05 tok=16.064B dt=9590 wait=153 +16:34:16 step 122320 loss=3.3973 lr=5.98e-05 tok=16.067B dt=9654 wait=153 +16:35:19 step 122340 loss=3.3450 lr=5.98e-05 tok=16.069B dt=9718 wait=153 +16:36:24 step 122360 loss=3.3603 lr=5.98e-05 tok=16.072B dt=9782 wait=154 +16:37:28 step 122380 loss=3.3055 lr=5.98e-05 tok=16.075B dt=9846 wait=154 +16:38:31 step 122400 loss=3.4160 lr=5.97e-05 tok=16.077B dt=9910 wait=154 +16:38:46 pushed ckpt_w110.pt,log_w110.txt +16:42:53 step 122420 loss=3.3890 lr=5.97e-05 tok=16.080B dt=10171 wait=155 +16:43:56 step 122440 loss=3.3390 lr=5.97e-05 tok=16.083B dt=10234 wait=155 +16:45:00 step 122460 loss=3.2676 lr=5.97e-05 tok=16.085B dt=10298 wait=155 +16:46:03 step 122480 loss=3.2486 lr=5.97e-05 tok=16.088B dt=10361 wait=155 +16:47:07 step 122500 loss=3.2296 lr=5.96e-05 tok=16.090B dt=10425 wait=156 +16:48:11 step 122520 loss=3.3801 lr=5.96e-05 tok=16.093B dt=10489 wait=156 +16:49:15 step 122540 loss=3.3062 lr=5.96e-05 tok=16.096B dt=10553 wait=156 +16:50:09 pushed ckpt_w110.pt,log_w110.txt +16:50:24 waited 15s for global r=425 +16:50:33 global r=425 load 4.286 -> 4.278 +16:51:37 step 122420 loss=3.2999 lr=5.97e-05 tok=16.100B dt=10696 wait=157 +16:52:41 step 122440 loss=3.3939 lr=5.97e-05 tok=16.102B dt=10759 wait=157 +16:53:45 step 122460 loss=3.3004 lr=5.97e-05 tok=16.105B dt=10823 wait=158 +16:54:49 step 122480 loss=3.3266 lr=5.97e-05 tok=16.107B dt=10887 wait=158 +16:55:53 step 122500 loss=3.2769 lr=5.96e-05 tok=16.110B dt=10951 wait=158 +16:56:56 step 122520 loss=3.4177 lr=5.96e-05 tok=16.113B dt=11014 wait=159 +16:58:01 step 122540 loss=3.3514 lr=5.96e-05 tok=16.115B dt=11079 wait=159 +16:58:46 pushed ckpt_w110.pt,log_w110.txt +17:02:21 step 122560 loss=3.4534 lr=5.96e-05 tok=16.118B dt=11339 wait=159 +17:03:25 step 122580 loss=3.3564 lr=5.96e-05 tok=16.121B dt=11403 wait=160 +17:04:28 step 122600 loss=3.3048 lr=5.95e-05 tok=16.123B dt=11466 wait=160 +17:05:31 step 122620 loss=3.1939 lr=5.95e-05 tok=16.126B dt=11530 wait=160 +17:06:35 step 122640 loss=3.2569 lr=5.95e-05 tok=16.128B dt=11593 wait=161 +17:07:39 step 122660 loss=3.3098 lr=5.95e-05 tok=16.131B dt=11657 wait=161 +17:08:42 step 122680 loss=3.2737 lr=5.95e-05 tok=16.134B dt=11720 wait=161 +17:09:25 global r=426 load 4.309 -> 4.287 +17:09:57 step 122560 loss=3.4124 lr=5.96e-05 tok=16.136B dt=11795 wait=162 +17:11:01 step 122580 loss=3.2995 lr=5.96e-05 tok=16.139B dt=11859 wait=162 +17:12:05 step 122600 loss=3.2497 lr=5.95e-05 tok=16.142B dt=11923 wait=162 +17:13:09 step 122620 loss=3.3402 lr=5.95e-05 tok=16.144B dt=11987 wait=163 +17:14:12 step 122640 loss=3.2817 lr=5.95e-05 tok=16.147B dt=12051 wait=163 +17:15:16 step 122660 loss=3.3750 lr=5.95e-05 tok=16.149B dt=12114 wait=163 +17:16:19 step 122680 loss=3.3437 lr=5.95e-05 tok=16.152B dt=12177 wait=164 +17:17:23 step 122700 loss=3.4386 lr=5.94e-05 tok=16.155B dt=12241 wait=164 +17:17:37 pushed ckpt_w110.pt,log_w110.txt +17:19:24 waited 107s for global r=427 +17:19:30 global r=427 load 4.291 -> 4.284 +17:20:35 step 122720 loss=3.2625 lr=5.94e-05 tok=16.157B dt=12434 wait=164 +17:21:39 step 122740 loss=3.3683 lr=5.94e-05 tok=16.160B dt=12497 wait=165 +17:22:42 step 122760 loss=3.3465 lr=5.94e-05 tok=16.162B dt=12560 wait=165 +17:23:46 step 122780 loss=3.2911 lr=5.94e-05 tok=16.165B dt=12624 wait=165 +17:24:49 step 122800 loss=3.3179 lr=5.93e-05 tok=16.168B dt=12687 wait=166 +17:25:53 step 122820 loss=3.4251 lr=5.93e-05 tok=16.170B dt=12751 wait=166 +17:26:56 step 122840 loss=3.2629 lr=5.93e-05 tok=16.173B dt=12814 wait=166 +17:27:44 pushed ckpt_w110.pt,log_w110.txt +17:28:59 waited 76s for global r=428 +17:29:07 global r=428 load 4.295 -> 4.284 +17:29:40 step 122860 loss=3.3947 lr=5.93e-05 tok=16.176B dt=12979 wait=167 +17:30:44 step 122880 loss=3.3010 lr=5.93e-05 tok=16.178B dt=13042 wait=167 +17:31:47 step 122900 loss=3.3156 lr=5.92e-05 tok=16.181B dt=13105 wait=167 +17:32:50 step 122920 loss=3.3927 lr=5.92e-05 tok=16.183B dt=13169 wait=168 +17:33:55 step 122940 loss=3.1900 lr=5.92e-05 tok=16.186B dt=13234 wait=168 +17:34:59 step 122960 loss=3.3678 lr=5.92e-05 tok=16.189B dt=13297 wait=168 +17:36:02 step 122980 loss=3.3003 lr=5.92e-05 tok=16.191B dt=13361 wait=168 +17:37:06 step 123000 loss=3.3842 lr=5.91e-05 tok=16.194B dt=13424 wait=169 +17:37:20 pushed ckpt_w110.pt,log_w110.txt +17:38:36 waited 76s for global r=429 +17:38:45 global r=429 load 4.283 -> 4.277 +17:39:50 step 123020 loss=3.3043 lr=5.91e-05 tok=16.197B dt=13588 wait=169 +17:40:53 step 123040 loss=3.3367 lr=5.91e-05 tok=16.199B dt=13651 wait=169 +17:41:57 step 123060 loss=3.3512 lr=5.91e-05 tok=16.202B dt=13715 wait=170 +17:43:00 step 123080 loss=3.3216 lr=5.91e-05 tok=16.204B dt=13779 wait=170 +17:44:04 step 123100 loss=3.3470 lr=5.90e-05 tok=16.207B dt=13842 wait=170 +17:45:08 step 123120 loss=3.3648 lr=5.90e-05 tok=16.210B dt=13906 wait=171 +17:46:12 step 123140 loss=3.2203 lr=5.90e-05 tok=16.212B dt=13970 wait=171 +17:47:00 pushed ckpt_w110.pt,log_w110.txt +17:48:01 waited 61s for global r=430 +17:48:08 global r=430 load 4.306 -> 4.288 +17:48:41 step 123160 loss=3.3717 lr=5.90e-05 tok=16.215B dt=14120 wait=171 +17:49:45 step 123180 loss=3.2569 lr=5.90e-05 tok=16.218B dt=14183 wait=172 +17:50:48 step 123200 loss=3.2818 lr=5.89e-05 tok=16.220B dt=14246 wait=172 +17:51:52 step 123220 loss=3.3518 lr=5.89e-05 tok=16.223B dt=14310 wait=172 +17:52:55 step 123240 loss=3.2779 lr=5.89e-05 tok=16.225B dt=14373 wait=173 +17:53:58 step 123260 loss=3.3499 lr=5.89e-05 tok=16.228B dt=14437 wait=173 +17:55:02 step 123280 loss=3.3229 lr=5.89e-05 tok=16.231B dt=14500 wait=173 +17:56:06 step 123300 loss=3.3478 lr=5.88e-05 tok=16.233B dt=14564 wait=173 +17:56:20 pushed ckpt_w110.pt,log_w110.txt +17:57:36 waited 76s for global r=431 +17:57:45 global r=431 load 4.298 -> 4.295 +17:58:51 step 123320 loss=3.2763 lr=5.88e-05 tok=16.236B dt=14729 wait=174 +17:59:54 step 123340 loss=3.3119 lr=5.88e-05 tok=16.239B dt=14793 wait=174 +18:00:58 step 123360 loss=3.3247 lr=5.88e-05 tok=16.241B dt=14857 wait=174 +18:02:02 step 123380 loss=3.2847 lr=5.88e-05 tok=16.244B dt=14920 wait=175 +18:03:05 step 123400 loss=3.3125 lr=5.87e-05 tok=16.246B dt=14983 wait=175 +18:04:09 step 123420 loss=3.4284 lr=5.87e-05 tok=16.249B dt=15047 wait=175 +18:05:12 step 123440 loss=3.2906 lr=5.87e-05 tok=16.252B dt=15110 wait=176 +18:05:56 pushed ckpt_w110.pt,log_w110.txt +18:07:17 waited 81s for global r=432 +18:07:26 global r=432 load 4.293 -> 4.294 +18:08:00 step 123460 loss=3.3947 lr=5.87e-05 tok=16.254B dt=15278 wait=176 +18:09:04 step 123480 loss=3.3167 lr=5.87e-05 tok=16.257B dt=15342 wait=176 +18:10:07 step 123500 loss=3.3515 lr=5.86e-05 tok=16.259B dt=15405 wait=177 +18:11:11 step 123520 loss=3.4255 lr=5.86e-05 tok=16.262B dt=15469 wait=177 +18:12:14 step 123540 loss=3.2570 lr=5.86e-05 tok=16.265B dt=15532 wait=177 +18:13:17 step 123560 loss=3.3405 lr=5.86e-05 tok=16.267B dt=15596 wait=178 +18:14:21 step 123580 loss=3.2691 lr=5.86e-05 tok=16.270B dt=15659 wait=178 +18:15:25 step 123600 loss=3.3417 lr=5.85e-05 tok=16.273B dt=15723 wait=178 +18:15:36 pushed ckpt_w110.pt,log_w110.txt +18:16:39 waited 62s for global r=433 +18:16:50 global r=433 load 4.288 -> 4.281 +18:17:55 step 123620 loss=3.2217 lr=5.85e-05 tok=16.275B dt=15873 wait=179 +18:18:58 step 123640 loss=3.3113 lr=5.85e-05 tok=16.278B dt=15936 wait=179 +18:20:02 step 123660 loss=3.3280 lr=5.85e-05 tok=16.280B dt=16000 wait=179 +18:21:05 step 123680 loss=3.2800 lr=5.85e-05 tok=16.283B dt=16064 wait=180 +18:22:09 step 123700 loss=3.2993 lr=5.84e-05 tok=16.286B dt=16127 wait=180 +18:23:12 step 123720 loss=3.3667 lr=5.84e-05 tok=16.288B dt=16190 wait=180 +18:24:16 step 123740 loss=3.3278 lr=5.84e-05 tok=16.291B dt=16254 wait=180 +18:25:08 pushed ckpt_w110.pt,log_w110.txt +18:26:09 waited 61s for global r=434 +18:26:16 global r=434 load 4.317 -> 4.290 +18:26:49 step 123760 loss=3.3912 lr=5.84e-05 tok=16.294B dt=16407 wait=181 +18:27:53 step 123780 loss=3.2888 lr=5.84e-05 tok=16.296B dt=16471 wait=181 +18:28:56 step 123800 loss=3.2819 lr=5.83e-05 tok=16.299B dt=16535 wait=181 +18:30:00 step 123820 loss=3.4052 lr=5.83e-05 tok=16.301B dt=16598 wait=182 +18:31:04 step 123840 loss=3.3011 lr=5.83e-05 tok=16.304B dt=16662 wait=182 +18:32:07 step 123860 loss=3.3310 lr=5.83e-05 tok=16.307B dt=16726 wait=182 +18:33:11 step 123880 loss=3.2666 lr=5.83e-05 tok=16.309B dt=16789 wait=183 +18:34:15 step 123900 loss=3.3101 lr=5.82e-05 tok=16.312B dt=16853 wait=183 +18:34:28 pushed ckpt_w110.pt,log_w110.txt +18:38:35 step 123920 loss=3.3911 lr=5.82e-05 tok=16.315B dt=17113 wait=183 +18:39:39 step 123940 loss=3.2913 lr=5.82e-05 tok=16.317B dt=17177 wait=184 +18:40:42 step 123960 loss=3.2670 lr=5.82e-05 tok=16.320B dt=17240 wait=184 +18:41:45 step 123980 loss=3.2234 lr=5.82e-05 tok=16.322B dt=17303 wait=184 +18:42:48 step 124000 loss=3.2430 lr=5.81e-05 tok=16.325B dt=17367 wait=185 +18:43:52 step 124020 loss=3.4048 lr=5.81e-05 tok=16.328B dt=17430 wait=185 +18:44:55 step 124040 loss=3.3005 lr=5.81e-05 tok=16.330B dt=17493 wait=185 +18:45:41 pushed ckpt_w110.pt,log_w110.txt +18:46:42 waited 61s for global r=435 +18:46:55 global r=435 load 4.280 -> 4.286 +18:47:30 step 124060 loss=3.4115 lr=5.81e-05 tok=16.333B dt=17648 wait=186 +18:48:33 step 124080 loss=3.2956 lr=5.81e-05 tok=16.336B dt=17711 wait=186 +18:49:37 step 124100 loss=3.3214 lr=5.80e-05 tok=16.338B dt=17775 wait=186 +18:50:46 step 124120 loss=3.3335 lr=5.80e-05 tok=16.341B dt=17844 wait=186 +18:51:49 step 124140 loss=3.2805 lr=5.80e-05 tok=16.343B dt=17907 wait=187 +18:52:52 step 124160 loss=3.3915 lr=5.80e-05 tok=16.346B dt=17971 wait=187 +18:53:56 step 124180 loss=3.2648 lr=5.80e-05 tok=16.349B dt=18034 wait=187 +18:54:59 step 124200 loss=3.3489 lr=5.79e-05 tok=16.351B dt=18097 wait=188 +18:55:17 pushed ckpt_w110.pt,log_w110.txt +18:56:18 waited 61s for global r=436 +18:56:33 global r=436 load 4.297 -> 4.277 +18:57:38 step 124220 loss=3.3338 lr=5.79e-05 tok=16.354B dt=18256 wait=188 +18:58:41 step 124240 loss=3.4219 lr=5.79e-05 tok=16.356B dt=18319 wait=188 +18:59:45 step 124260 loss=3.3354 lr=5.79e-05 tok=16.359B dt=18383 wait=189 +19:00:48 step 124280 loss=3.2887 lr=5.79e-05 tok=16.362B dt=18446 wait=189 +19:01:52 step 124300 loss=3.3288 lr=5.78e-05 tok=16.364B dt=18510 wait=189 +19:02:55 step 124320 loss=3.3766 lr=5.78e-05 tok=16.367B dt=18574 wait=190 +19:03:59 step 124340 loss=3.3074 lr=5.78e-05 tok=16.370B dt=18637 wait=190 +19:04:51 pushed ckpt_w110.pt,log_w110.txt +19:05:51 waited 61s for global r=437 +19:05:58 global r=437 load 4.290 -> 4.282 +19:06:32 step 124360 loss=3.3806 lr=5.78e-05 tok=16.372B dt=18790 wait=190 +19:07:35 step 124380 loss=3.2816 lr=5.78e-05 tok=16.375B dt=18854 wait=191 +19:08:39 step 124400 loss=3.2977 lr=5.78e-05 tok=16.377B dt=18917 wait=191 +19:09:42 step 124420 loss=3.3934 lr=5.77e-05 tok=16.380B dt=18980 wait=191 +19:10:46 step 124440 loss=3.3064 lr=5.77e-05 tok=16.383B dt=19044 wait=192 +19:11:49 step 124460 loss=3.3640 lr=5.77e-05 tok=16.385B dt=19107 wait=192 +19:12:52 step 124480 loss=3.3658 lr=5.77e-05 tok=16.388B dt=19171 wait=192 +19:13:56 step 124500 loss=3.3779 lr=5.77e-05 tok=16.391B dt=19234 wait=192 +19:14:08 pushed ckpt_w110.pt,log_w110.txt +19:15:15 waited 67s for global r=438 +19:15:25 global r=438 load 4.294 -> 4.283 +19:16:30 step 124520 loss=3.3427 lr=5.76e-05 tok=16.393B dt=19389 wait=193 +19:17:34 step 124540 loss=3.4048 lr=5.76e-05 tok=16.396B dt=19452 wait=193 +19:18:37 step 124560 loss=3.3578 lr=5.76e-05 tok=16.398B dt=19516 wait=194 +19:19:41 step 124580 loss=3.2769 lr=5.76e-05 tok=16.401B dt=19579 wait=194 +19:20:44 step 124600 loss=3.2844 lr=5.76e-05 tok=16.404B dt=19642 wait=194 +19:21:47 step 124620 loss=3.4110 lr=5.75e-05 tok=16.406B dt=19706 wait=194 +19:22:51 step 124640 loss=3.2326 lr=5.75e-05 tok=16.409B dt=19769 wait=195 +19:23:34 pushed ckpt_w110.pt,log_w110.txt +19:24:50 waited 76s for global r=439 +19:25:00 global r=439 load 4.307 -> 4.292 +19:25:33 step 124660 loss=3.3743 lr=5.75e-05 tok=16.412B dt=19931 wait=195 +19:26:37 step 124680 loss=3.2534 lr=5.75e-05 tok=16.414B dt=19995 wait=195 +19:27:40 step 124700 loss=3.2329 lr=5.75e-05 tok=16.417B dt=20058 wait=196 +19:28:44 step 124720 loss=3.3783 lr=5.74e-05 tok=16.419B dt=20122 wait=196 +19:29:48 step 124740 loss=3.1916 lr=5.74e-05 tok=16.422B dt=20186 wait=196 +19:30:51 step 124760 loss=3.3458 lr=5.74e-05 tok=16.425B dt=20249 wait=197 +19:31:55 step 124780 loss=3.2917 lr=5.74e-05 tok=16.427B dt=20313 wait=197 +19:32:58 step 124800 loss=3.3505 lr=5.74e-05 tok=16.430B dt=20376 wait=197 +19:33:10 pushed ckpt_w110.pt,log_w110.txt +19:34:19 waited 69s for global r=440 +19:34:32 global r=440 load 4.280 -> 4.290 +19:35:37 step 124820 loss=3.3550 lr=5.73e-05 tok=16.432B dt=20535 wait=198 +19:36:40 step 124840 loss=3.3572 lr=5.73e-05 tok=16.435B dt=20598 wait=198 +19:37:44 step 124860 loss=3.2916 lr=5.73e-05 tok=16.438B dt=20662 wait=198 +19:38:47 step 124880 loss=3.2856 lr=5.73e-05 tok=16.440B dt=20725 wait=199 +19:39:51 step 124900 loss=3.3494 lr=5.73e-05 tok=16.443B dt=20789 wait=199 +19:40:54 step 124920 loss=3.4153 lr=5.72e-05 tok=16.446B dt=20853 wait=199 +19:41:58 step 124940 loss=3.2822 lr=5.72e-05 tok=16.448B dt=20916 wait=200 +19:42:46 pushed ckpt_w110.pt,log_w110.txt +19:43:47 waited 61s for global r=441 +19:43:55 global r=441 load 4.290 -> 4.284 +19:44:29 step 124960 loss=3.4118 lr=5.72e-05 tok=16.451B dt=21067 wait=200 +19:45:33 step 124980 loss=3.3245 lr=5.72e-05 tok=16.453B dt=21131 wait=200 +19:46:36 step 125000 loss=3.2848 lr=5.72e-05 tok=16.456B dt=21195 wait=201 +19:47:42 step 125020 loss=3.0654 lr=5.71e-05 tok=16.459B dt=21260 wait=201 +19:48:45 step 125040 loss=3.2442 lr=5.71e-05 tok=16.461B dt=21324 wait=201 +19:49:49 step 125060 loss=3.3237 lr=5.71e-05 tok=16.464B dt=21387 wait=202 +19:50:52 step 125080 loss=3.1201 lr=5.71e-05 tok=16.467B dt=21450 wait=202 +19:51:56 step 125100 loss=3.0957 lr=5.71e-05 tok=16.469B dt=21515 wait=202 +19:52:11 pushed ckpt_w110.pt,log_w110.txt +19:53:26 waited 76s for global r=442 +19:53:34 global r=442 load 4.302 -> 4.290 +19:54:39 step 125120 loss=3.2444 lr=5.70e-05 tok=16.472B dt=21677 wait=203 +19:55:42 step 125140 loss=3.3393 lr=5.70e-05 tok=16.474B dt=21740 wait=203 +19:56:46 step 125160 loss=3.1668 lr=5.70e-05 tok=16.477B dt=21805 wait=203 +19:57:51 step 125180 loss=3.1197 lr=5.70e-05 tok=16.480B dt=21869 wait=203 +19:58:55 step 125200 loss=3.2232 lr=5.70e-05 tok=16.482B dt=21933 wait=204 +19:59:58 step 125220 loss=3.4419 lr=5.69e-05 tok=16.485B dt=21996 wait=204 +20:01:02 step 125240 loss=3.3175 lr=5.69e-05 tok=16.488B dt=22060 wait=204 +20:01:46 pushed ckpt_w110.pt,log_w110.txt +20:02:47 waited 61s for global r=443 +20:02:57 global r=443 load 4.286 -> 4.292 +20:03:31 step 125260 loss=3.3748 lr=5.69e-05 tok=16.490B dt=22209 wait=205 +20:04:34 step 125280 loss=3.3212 lr=5.69e-05 tok=16.493B dt=22272 wait=205 +20:05:43 step 125300 loss=3.2336 lr=5.69e-05 tok=16.495B dt=22341 wait=205 +20:06:47 step 125320 loss=3.2457 lr=5.68e-05 tok=16.498B dt=22405 wait=206 +20:07:51 step 125340 loss=3.2038 lr=5.68e-05 tok=16.501B dt=22469 wait=206 +20:08:55 step 125360 loss=3.4065 lr=5.68e-05 tok=16.503B dt=22533 wait=206 +20:09:58 step 125380 loss=3.2792 lr=5.68e-05 tok=16.506B dt=22596 wait=207 +20:11:02 step 125400 loss=3.3202 lr=5.68e-05 tok=16.509B dt=22660 wait=207 +20:11:14 pushed ckpt_w110.pt,log_w110.txt +20:14:57 global r=444 load 4.295 -> 4.288 +20:16:00 step 125420 loss=3.2820 lr=5.67e-05 tok=16.512B dt=22958 wait=208 +20:17:04 step 125440 loss=3.3228 lr=5.67e-05 tok=16.515B dt=23022 wait=208 +20:18:08 step 125460 loss=3.2739 lr=5.67e-05 tok=16.518B dt=23086 wait=208 +20:19:11 step 125480 loss=3.2727 lr=5.67e-05 tok=16.520B dt=23149 wait=209 +20:20:15 step 125500 loss=3.2563 lr=5.67e-05 tok=16.523B dt=23213 wait=209 +20:21:18 step 125520 loss=3.3942 lr=5.67e-05 tok=16.526B dt=23276 wait=209 +20:22:21 step 125540 loss=3.2455 lr=5.66e-05 tok=16.528B dt=23339 wait=210 +20:23:05 pushed ckpt_w110.pt,log_w110.txt +20:24:22 waited 77s for global r=445 +20:24:35 global r=445 load 4.272 -> 4.281 +20:25:08 step 125560 loss=3.3341 lr=5.66e-05 tok=16.531B dt=23506 wait=210 +20:26:12 step 125580 loss=3.2691 lr=5.66e-05 tok=16.533B dt=23570 wait=210 +20:27:15 step 125600 loss=3.2393 lr=5.66e-05 tok=16.536B dt=23634 wait=210 +20:28:19 step 125620 loss=3.3625 lr=5.66e-05 tok=16.539B dt=23697 wait=211 +20:29:23 step 125640 loss=3.2353 lr=5.65e-05 tok=16.541B dt=23761 wait=211 +20:30:26 step 125660 loss=3.3659 lr=5.65e-05 tok=16.544B dt=23825 wait=212 +20:31:30 step 125680 loss=3.2140 lr=5.65e-05 tok=16.547B dt=23888 wait=212 +20:32:34 step 125700 loss=3.3238 lr=5.65e-05 tok=16.549B dt=23952 wait=212 +20:32:48 pushed ckpt_w110.pt,log_w110.txt +20:33:48 waited 61s for global r=446 +20:34:01 global r=446 load 4.322 -> 4.287 +20:35:06 step 125720 loss=3.2191 lr=5.65e-05 tok=16.552B dt=24104 wait=212 +20:36:09 step 125740 loss=3.4111 lr=5.64e-05 tok=16.554B dt=24168 wait=213 +20:37:13 step 125760 loss=3.2995 lr=5.64e-05 tok=16.557B dt=24231 wait=213 +20:38:17 step 125780 loss=3.1982 lr=5.64e-05 tok=16.560B dt=24295 wait=213 +20:39:20 step 125800 loss=3.2825 lr=5.64e-05 tok=16.562B dt=24358 wait=214 +20:40:23 step 125820 loss=3.4125 lr=5.64e-05 tok=16.565B dt=24422 wait=214 +20:41:27 step 125840 loss=3.2658 lr=5.63e-05 tok=16.568B dt=24485 wait=214 +20:42:11 pushed ckpt_w110.pt,log_w110.txt +20:43:12 waited 62s for global r=447 +20:43:20 global r=447 load 4.282 -> 4.273 +20:43:53 step 125860 loss=3.3186 lr=5.63e-05 tok=16.570B dt=24631 wait=215 +20:44:57 step 125880 loss=3.2941 lr=5.63e-05 tok=16.573B dt=24695 wait=215 +20:46:01 step 125900 loss=3.2971 lr=5.63e-05 tok=16.575B dt=24759 wait=215 +20:47:04 step 125920 loss=3.3622 lr=5.63e-05 tok=16.578B dt=24822 wait=216 +20:48:07 step 125940 loss=3.2942 lr=5.62e-05 tok=16.581B dt=24885 wait=216 +20:49:11 step 125960 loss=3.3752 lr=5.62e-05 tok=16.583B dt=24949 wait=216 +20:50:14 step 125980 loss=3.3039 lr=5.62e-05 tok=16.586B dt=25012 wait=217 +20:51:18 step 126000 loss=3.3405 lr=5.62e-05 tok=16.588B dt=25076 wait=217 +20:51:30 pushed ckpt_w110.pt,log_w110.txt +20:52:46 waited 76s for global r=448 +20:52:54 global r=448 load 4.286 -> 4.270 +20:53:39 mix source 1 lap 1 +20:54:02 step 126020 loss=3.2959 lr=5.62e-05 tok=16.591B dt=25240 wait=219 +20:55:05 step 126040 loss=3.3272 lr=5.61e-05 tok=16.594B dt=25304 wait=219 +20:56:09 step 126060 loss=3.3155 lr=5.61e-05 tok=16.596B dt=25367 wait=219 +20:57:13 step 126080 loss=3.2258 lr=5.61e-05 tok=16.599B dt=25432 wait=220 +20:58:17 step 126100 loss=3.3439 lr=5.61e-05 tok=16.602B dt=25495 wait=220 +20:59:20 step 126120 loss=3.4360 lr=5.61e-05 tok=16.604B dt=25559 wait=220 +21:00:24 step 126140 loss=3.2842 lr=5.60e-05 tok=16.607B dt=25622 wait=221 +21:01:13 pushed ckpt_w110.pt,log_w110.txt +21:02:14 waited 61s for global r=449 +21:02:20 global r=449 load 4.273 -> 4.272 +21:02:53 step 126160 loss=3.3855 lr=5.60e-05 tok=16.609B dt=25771 wait=221 +21:03:57 step 126180 loss=3.2794 lr=5.60e-05 tok=16.612B dt=25835 wait=221 +21:05:01 step 126200 loss=3.3165 lr=5.60e-05 tok=16.615B dt=25899 wait=222 +21:06:04 step 126220 loss=3.3951 lr=5.60e-05 tok=16.617B dt=25962 wait=222 +21:07:08 step 126240 loss=3.3133 lr=5.59e-05 tok=16.620B dt=26026 wait=222 +21:08:11 step 126260 loss=3.3664 lr=5.59e-05 tok=16.623B dt=26090 wait=223 +21:09:15 step 126280 loss=3.2414 lr=5.59e-05 tok=16.625B dt=26153 wait=223 +21:10:18 step 126300 loss=3.3192 lr=5.59e-05 tok=16.628B dt=26216 wait=223 +21:10:34 pushed ckpt_w110.pt,log_w110.txt +21:11:37 waited 63s for global r=450 +21:11:50 global r=450 load 4.279 -> 4.270 +21:12:54 step 126320 loss=3.2662 lr=5.59e-05 tok=16.630B dt=26373 wait=224 +21:13:58 step 126340 loss=3.3488 lr=5.58e-05 tok=16.633B dt=26436 wait=224 +21:15:01 step 126360 loss=3.3166 lr=5.58e-05 tok=16.636B dt=26500 wait=224 +21:16:05 step 126380 loss=3.3130 lr=5.58e-05 tok=16.638B dt=26563 wait=225 +21:17:08 step 126400 loss=3.3271 lr=5.58e-05 tok=16.641B dt=26627 wait=225 +21:18:12 step 126420 loss=3.3988 lr=5.58e-05 tok=16.644B dt=26690 wait=225 +21:19:15 step 126440 loss=3.2577 lr=5.58e-05 tok=16.646B dt=26754 wait=226 +21:20:00 pushed ckpt_w110.pt,log_w110.txt +21:21:01 waited 61s for global r=451 +21:21:11 global r=451 load 4.267 -> 4.271 +21:21:44 step 126460 loss=3.4160 lr=5.57e-05 tok=16.649B dt=26902 wait=226 +21:22:48 step 126480 loss=3.2786 lr=5.57e-05 tok=16.651B dt=26966 wait=226 +21:23:51 step 126500 loss=3.3039 lr=5.57e-05 tok=16.654B dt=27030 wait=227 +21:24:55 step 126520 loss=3.3771 lr=5.57e-05 tok=16.657B dt=27093 wait=227 +21:25:58 step 126540 loss=3.2956 lr=5.57e-05 tok=16.659B dt=27157 wait=227 +21:27:02 step 126560 loss=3.3998 lr=5.56e-05 tok=16.662B dt=27220 wait=228 +21:28:06 step 126580 loss=3.3324 lr=5.56e-05 tok=16.664B dt=27284 wait=228 +21:29:09 step 126600 loss=3.3486 lr=5.56e-05 tok=16.667B dt=27348 wait=228 +21:29:22 pushed ckpt_w110.pt,log_w110.txt +21:30:24 waited 62s for global r=452 +21:30:33 global r=452 load 4.285 -> 4.274 +21:31:38 step 126620 loss=3.3424 lr=5.56e-05 tok=16.670B dt=27496 wait=229 +21:32:41 step 126640 loss=3.3761 lr=5.56e-05 tok=16.672B dt=27559 wait=229 +21:33:44 step 126660 loss=3.3244 lr=5.55e-05 tok=16.675B dt=27623 wait=229 +21:34:48 step 126680 loss=3.3003 lr=5.55e-05 tok=16.678B dt=27686 wait=230 +21:35:52 step 126700 loss=3.3117 lr=5.55e-05 tok=16.680B dt=27750 wait=230 +21:36:55 step 126720 loss=3.4319 lr=5.55e-05 tok=16.683B dt=27814 wait=230 +21:37:59 step 126740 loss=3.2997 lr=5.55e-05 tok=16.685B dt=27877 wait=231 +21:40:21 pushed ckpt_w110.pt,log_w110.txt +21:41:22 waited 61s for global r=453 +21:41:32 global r=453 load 4.263 -> 4.274 +21:42:05 step 126760 loss=3.4211 lr=5.54e-05 tok=16.688B dt=28123 wait=231 +21:43:10 step 126780 loss=3.0126 lr=5.54e-05 tok=16.691B dt=28188 wait=231 +21:44:14 step 126800 loss=3.1797 lr=5.54e-05 tok=16.693B dt=28252 wait=232 +21:45:18 step 126820 loss=3.1497 lr=5.54e-05 tok=16.696B dt=28316 wait=232 +21:46:22 step 126840 loss=3.1982 lr=5.54e-05 tok=16.699B dt=28380 wait=232 +21:47:26 step 126860 loss=3.3666 lr=5.53e-05 tok=16.701B dt=28444 wait=233 +21:48:29 step 126880 loss=3.2850 lr=5.53e-05 tok=16.704B dt=28507 wait=233 +21:49:33 step 126900 loss=3.1552 lr=5.53e-05 tok=16.706B dt=28571 wait=233 +21:49:46 pushed ckpt_w110.pt,log_w110.txt +21:51:02 waited 76s for global r=454 +21:51:13 global r=454 load 4.300 -> 4.280 +21:52:18 step 126920 loss=3.3036 lr=5.53e-05 tok=16.709B dt=28736 wait=234 +21:53:21 step 126940 loss=3.4103 lr=5.53e-05 tok=16.712B dt=28800 wait=234 +21:54:25 step 126960 loss=3.2770 lr=5.52e-05 tok=16.714B dt=28863 wait=234 +21:55:29 step 126980 loss=3.2244 lr=5.52e-05 tok=16.717B dt=28927 wait=235 +21:56:32 step 127000 loss=3.3132 lr=5.52e-05 tok=16.720B dt=28990 wait=235 +21:57:36 step 127020 loss=3.4181 lr=5.52e-05 tok=16.722B dt=29054 wait=235 +21:58:39 step 127040 loss=3.2841 lr=5.52e-05 tok=16.725B dt=29118 wait=236 +21:59:28 pushed ckpt_w110.pt,log_w110.txt +22:00:28 waited 61s for global r=455 +22:00:36 global r=455 load 4.303 -> 4.284 +22:01:09 step 127060 loss=3.4212 lr=5.51e-05 tok=16.727B dt=29267 wait=236 +22:02:12 step 127080 loss=3.2602 lr=5.51e-05 tok=16.730B dt=29331 wait=236 +22:03:16 step 127100 loss=3.3248 lr=5.51e-05 tok=16.733B dt=29394 wait=237 +22:04:20 step 127120 loss=3.3690 lr=5.51e-05 tok=16.735B dt=29458 wait=237 +22:05:23 step 127140 loss=3.2703 lr=5.51e-05 tok=16.738B dt=29521 wait=237 +22:06:27 step 127160 loss=3.7023 lr=5.51e-05 tok=16.741B dt=29586 wait=238 +22:07:31 step 127180 loss=3.3065 lr=5.50e-05 tok=16.743B dt=29649 wait=238 +22:08:34 step 127200 loss=3.2981 lr=5.50e-05 tok=16.746B dt=29713 wait=238 +22:08:51 pushed ckpt_w110.pt,log_w110.txt +22:09:52 waited 61s for global r=456 +22:09:59 global r=456 load 4.303 -> 4.287 +22:11:05 step 127220 loss=3.2727 lr=5.50e-05 tok=16.748B dt=29863 wait=239 +22:12:08 step 127240 loss=3.2706 lr=5.50e-05 tok=16.751B dt=29927 wait=239 +22:13:12 step 127260 loss=3.2719 lr=5.50e-05 tok=16.754B dt=29990 wait=240 +22:14:16 step 127280 loss=3.1739 lr=5.49e-05 tok=16.756B dt=30054 wait=240 +22:15:20 step 127300 loss=3.3042 lr=5.49e-05 tok=16.759B dt=30118 wait=240 +22:16:23 step 127320 loss=3.3631 lr=5.49e-05 tok=16.761B dt=30182 wait=240 +22:17:27 step 127340 loss=3.3510 lr=5.49e-05 tok=16.764B dt=30246 wait=241 +22:18:15 pushed ckpt_w110.pt,log_w110.txt +22:19:15 waited 61s for global r=457 +22:19:23 global r=457 load 4.273 -> 4.291 +22:19:56 step 127360 loss=3.4873 lr=5.49e-05 tok=16.767B dt=30394 wait=241 +22:21:00 step 127380 loss=3.3314 lr=5.48e-05 tok=16.769B dt=30458 wait=241 +22:22:04 step 127400 loss=3.2920 lr=5.48e-05 tok=16.772B dt=30522 wait=242 +22:23:07 step 127420 loss=3.4119 lr=5.48e-05 tok=16.775B dt=30585 wait=242 +22:24:11 step 127440 loss=3.2857 lr=5.48e-05 tok=16.777B dt=30649 wait=242 +22:25:15 step 127460 loss=3.3498 lr=5.48e-05 tok=16.780B dt=30713 wait=243 +22:26:18 step 127480 loss=3.3396 lr=5.47e-05 tok=16.782B dt=30776 wait=243 +22:27:22 step 127500 loss=3.3544 lr=5.47e-05 tok=16.785B dt=30840 wait=243 +22:27:39 pushed ckpt_w110.pt,log_w110.txt +22:28:40 waited 61s for global r=458 +22:28:46 global r=458 load 4.275 -> 4.289 +22:29:52 step 127520 loss=3.3139 lr=5.47e-05 tok=16.788B dt=30990 wait=244 +22:30:56 step 127540 loss=3.3049 lr=5.47e-05 tok=16.790B dt=31054 wait=244 +22:31:59 step 127560 loss=3.3136 lr=5.47e-05 tok=16.793B dt=31117 wait=244 +22:33:03 step 127580 loss=3.2872 lr=5.46e-05 tok=16.796B dt=31181 wait=245 +22:34:06 step 127600 loss=3.3588 lr=5.46e-05 tok=16.798B dt=31244 wait=245 +22:35:10 step 127620 loss=3.3597 lr=5.46e-05 tok=16.801B dt=31308 wait=245 +22:36:14 step 127640 loss=3.2824 lr=5.46e-05 tok=16.803B dt=31372 wait=246 +22:37:02 pushed ckpt_w110.pt,log_w110.txt +22:38:03 waited 61s for global r=459 +22:38:12 global r=459 load 4.302 -> 4.289 +22:38:46 step 127660 loss=3.4075 lr=5.46e-05 tok=16.806B dt=31524 wait=246 +22:39:50 step 127680 loss=3.2743 lr=5.45e-05 tok=16.809B dt=31588 wait=246 +22:40:54 step 127700 loss=3.2972 lr=5.45e-05 tok=16.811B dt=31652 wait=247 +22:41:57 step 127720 loss=3.3685 lr=5.45e-05 tok=16.814B dt=31715 wait=247 +22:43:01 step 127740 loss=3.2806 lr=5.45e-05 tok=16.817B dt=31779 wait=247 +22:44:04 step 127760 loss=3.3641 lr=5.45e-05 tok=16.819B dt=31842 wait=248 +22:45:08 step 127780 loss=3.2831 lr=5.45e-05 tok=16.822B dt=31906 wait=248 +22:46:11 step 127800 loss=3.3672 lr=5.44e-05 tok=16.824B dt=31970 wait=248 +22:46:28 pushed ckpt_w110.pt,log_w110.txt +22:47:28 waited 61s for global r=460 +22:47:43 global r=460 load 4.307 -> 4.285 +22:48:47 step 127820 loss=3.2952 lr=5.44e-05 tok=16.827B dt=32126 wait=249 +22:49:51 step 127840 loss=3.3513 lr=5.44e-05 tok=16.830B dt=32189 wait=249 +22:50:54 step 127860 loss=3.3561 lr=5.44e-05 tok=16.832B dt=32252 wait=249 +22:51:57 step 127880 loss=3.2346 lr=5.44e-05 tok=16.835B dt=32316 wait=250 +22:53:01 step 127900 loss=3.2930 lr=5.43e-05 tok=16.838B dt=32379 wait=250 +22:54:05 step 127920 loss=3.4053 lr=5.43e-05 tok=16.840B dt=32443 wait=250 +22:55:08 step 127940 loss=3.2612 lr=5.43e-05 tok=16.843B dt=32507 wait=251 diff --git a/log_w111.txt b/log_w111.txt new file mode 100644 index 0000000000000000000000000000000000000000..317c18fac266d19bf13fb332586c77ab2fc1d118 --- /dev/null +++ b/log_w111.txt @@ -0,0 +1,187 @@ +launch +13:50:56 name=w111 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 58/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:51:03 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:51:03 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:51:12 params 1.08B +13:51:14 resumed step=119850 seen=15.74B round=408 +13:51:14 stream salt=119850 +13:53:02 blocks compiled +13:53:02 start: accum=4, 131072 tokens per optimiser step +13:55:44 step 119851 loss=2.9684 lr=6.23e-05 tok=15.743B dt=161 wait=113 +13:55:47 step 119852 loss=3.1167 lr=6.23e-05 tok=15.743B dt=165 wait=113 +13:56:16 step 119860 loss=3.5113 lr=6.23e-05 tok=15.744B dt=194 wait=113 +13:57:25 step 119880 loss=3.4303 lr=6.23e-05 tok=15.747B dt=263 wait=114 +13:58:29 step 119900 loss=3.4406 lr=6.22e-05 tok=15.750B dt=327 wait=114 +13:59:33 step 119920 loss=3.3927 lr=6.22e-05 tok=15.752B dt=391 wait=115 +14:00:38 step 119940 loss=3.4727 lr=6.22e-05 tok=15.755B dt=455 wait=115 +14:01:42 step 119960 loss=3.5982 lr=6.22e-05 tok=15.757B dt=520 wait=115 +14:02:46 step 119980 loss=3.4332 lr=6.22e-05 tok=15.760B dt=584 wait=116 +14:03:50 step 120000 loss=3.4597 lr=6.21e-05 tok=15.763B dt=648 wait=116 +14:04:05 pushed ckpt_w111.pt,log_w111.txt +14:07:06 waited 181s for global r=409 +14:07:16 global r=409 load 4.257 -> 4.279 +14:08:23 step 120020 loss=3.4025 lr=6.21e-05 tok=15.765B dt=921 wait=116 +14:09:27 step 120040 loss=3.4524 lr=6.21e-05 tok=15.768B dt=985 wait=117 +14:10:31 step 120060 loss=3.4617 lr=6.21e-05 tok=15.771B dt=1048 wait=117 +14:11:34 step 120080 loss=3.3602 lr=6.21e-05 tok=15.773B dt=1112 wait=117 +14:12:38 step 120100 loss=3.3651 lr=6.20e-05 tok=15.776B dt=1176 wait=117 +14:13:42 step 120120 loss=3.4140 lr=6.20e-05 tok=15.778B dt=1240 wait=118 +14:14:46 step 120140 loss=3.2987 lr=6.20e-05 tok=15.781B dt=1304 wait=118 +14:15:30 pushed ckpt_w111.pt,log_w111.txt +14:16:31 waited 61s for global r=410 +14:16:37 global r=410 load 4.301 -> 4.291 +14:17:11 step 120160 loss=3.5133 lr=6.20e-05 tok=15.784B dt=1449 wait=118 +14:18:15 step 120180 loss=3.3560 lr=6.20e-05 tok=15.786B dt=1513 wait=119 +14:19:19 step 120200 loss=3.3767 lr=6.19e-05 tok=15.789B dt=1577 wait=119 +14:20:23 step 120220 loss=3.4440 lr=6.19e-05 tok=15.792B dt=1641 wait=119 +14:21:27 step 120240 loss=3.3387 lr=6.19e-05 tok=15.794B dt=1704 wait=120 +14:22:31 step 120260 loss=3.4153 lr=6.19e-05 tok=15.797B dt=1768 wait=120 +14:23:34 step 120280 loss=3.2747 lr=6.19e-05 tok=15.799B dt=1832 wait=120 +14:24:38 step 120300 loss=3.4090 lr=6.18e-05 tok=15.802B dt=1896 wait=121 +14:24:51 pushed ckpt_w111.pt,log_w111.txt +14:26:07 waited 76s for global r=411 +14:26:12 global r=411 load 4.291 -> 4.277 +14:27:17 step 120320 loss=3.3300 lr=6.18e-05 tok=15.805B dt=2055 wait=121 +14:28:21 step 120340 loss=3.4256 lr=6.18e-05 tok=15.807B dt=2119 wait=121 +14:29:25 step 120360 loss=3.3351 lr=6.18e-05 tok=15.810B dt=2183 wait=122 +14:30:30 step 120380 loss=3.3588 lr=6.18e-05 tok=15.813B dt=2247 wait=122 +14:31:33 step 120400 loss=3.4288 lr=6.17e-05 tok=15.815B dt=2311 wait=122 +14:32:38 step 120420 loss=3.5183 lr=6.17e-05 tok=15.818B dt=2375 wait=123 +14:33:42 step 120440 loss=3.4013 lr=6.17e-05 tok=15.820B dt=2439 wait=123 +14:34:25 pushed ckpt_w111.pt,log_w111.txt +14:35:41 waited 76s for global r=412 +14:35:46 global r=412 load 4.287 -> 4.284 +14:36:19 step 120460 loss=3.4094 lr=6.17e-05 tok=15.823B dt=2597 wait=123 +14:37:23 step 120480 loss=3.3693 lr=6.17e-05 tok=15.826B dt=2661 wait=124 +14:38:28 step 120500 loss=3.4171 lr=6.16e-05 tok=15.828B dt=2725 wait=124 +14:39:32 step 120520 loss=3.4375 lr=6.16e-05 tok=15.831B dt=2790 wait=124 +14:40:36 step 120540 loss=3.3138 lr=6.16e-05 tok=15.833B dt=2854 wait=125 +14:41:40 step 120560 loss=3.4902 lr=6.16e-05 tok=15.836B dt=2918 wait=125 +14:42:44 step 120580 loss=3.3804 lr=6.16e-05 tok=15.839B dt=2982 wait=125 +14:43:49 step 120600 loss=3.4507 lr=6.15e-05 tok=15.841B dt=3046 wait=126 +14:44:01 pushed ckpt_w111.pt,log_w111.txt +14:45:16 waited 76s for global r=413 +14:45:23 global r=413 load 4.311 -> 4.292 +14:46:28 step 120620 loss=3.3207 lr=6.15e-05 tok=15.844B dt=3206 wait=126 +14:47:32 step 120640 loss=3.4656 lr=6.15e-05 tok=15.847B dt=3269 wait=126 +14:48:36 step 120660 loss=3.4669 lr=6.15e-05 tok=15.849B dt=3334 wait=127 +14:49:41 step 120680 loss=3.3233 lr=6.15e-05 tok=15.852B dt=3399 wait=127 +14:50:45 step 120700 loss=3.4491 lr=6.14e-05 tok=15.854B dt=3463 wait=127 +14:51:49 step 120720 loss=3.4542 lr=6.14e-05 tok=15.857B dt=3527 wait=128 +14:52:53 step 120740 loss=3.3933 lr=6.14e-05 tok=15.860B dt=3591 wait=128 +14:53:38 pushed ckpt_w111.pt,log_w111.txt +14:54:53 waited 76s for global r=414 +14:54:58 global r=414 load 4.296 -> 4.279 +14:55:31 step 120760 loss=3.5310 lr=6.14e-05 tok=15.862B dt=3749 wait=128 +14:56:35 step 120780 loss=3.3855 lr=6.14e-05 tok=15.865B dt=3813 wait=129 +14:57:39 step 120800 loss=3.3975 lr=6.13e-05 tok=15.868B dt=3877 wait=129 +14:58:43 step 120820 loss=3.4638 lr=6.13e-05 tok=15.870B dt=3941 wait=129 +14:59:47 step 120840 loss=3.3252 lr=6.13e-05 tok=15.873B dt=4004 wait=130 +15:00:50 step 120860 loss=3.5143 lr=6.13e-05 tok=15.875B dt=4068 wait=130 +15:01:55 step 120880 loss=3.3759 lr=6.13e-05 tok=15.878B dt=4132 wait=130 +15:02:58 step 120900 loss=3.4105 lr=6.12e-05 tok=15.881B dt=4196 wait=130 +15:03:10 pushed ckpt_w111.pt,log_w111.txt +15:04:26 waited 76s for global r=415 +15:04:33 global r=415 load 4.275 -> 4.273 +15:05:37 step 120920 loss=3.3895 lr=6.12e-05 tok=15.883B dt=4355 wait=131 +15:06:41 step 120940 loss=3.4233 lr=6.12e-05 tok=15.886B dt=4419 wait=131 +15:07:45 step 120960 loss=3.4327 lr=6.12e-05 tok=15.889B dt=4483 wait=131 +15:08:48 step 120980 loss=3.3576 lr=6.12e-05 tok=15.891B dt=4546 wait=132 +15:09:52 step 121000 loss=3.3555 lr=6.11e-05 tok=15.894B dt=4609 wait=132 +15:10:56 step 121020 loss=3.4469 lr=6.11e-05 tok=15.896B dt=4673 wait=132 +15:11:59 step 121040 loss=3.2976 lr=6.11e-05 tok=15.899B dt=4737 wait=133 +15:12:43 pushed ckpt_w111.pt,log_w111.txt +15:13:44 waited 61s for global r=416 +15:13:50 global r=416 load 4.276 -> 4.281 +15:14:23 step 121060 loss=3.4184 lr=6.11e-05 tok=15.902B dt=4880 wait=133 +15:15:26 step 121080 loss=3.2932 lr=6.11e-05 tok=15.904B dt=4944 wait=133 +15:16:30 step 121100 loss=3.2989 lr=6.10e-05 tok=15.907B dt=5007 wait=134 +15:17:33 step 121120 loss=3.3897 lr=6.10e-05 tok=15.910B dt=5071 wait=134 +15:18:36 step 121140 loss=3.3146 lr=6.10e-05 tok=15.912B dt=5134 wait=134 +15:19:39 step 121160 loss=3.3938 lr=6.10e-05 tok=15.915B dt=5197 wait=135 +15:20:44 step 121180 loss=3.1906 lr=6.10e-05 tok=15.917B dt=5262 wait=135 +15:21:47 step 121200 loss=3.3661 lr=6.09e-05 tok=15.920B dt=5325 wait=135 +15:22:00 pushed ckpt_w111.pt,log_w111.txt +15:23:15 waited 76s for global r=417 +15:23:24 global r=417 load 4.288 -> 4.286 +15:24:28 step 121220 loss=3.3123 lr=6.09e-05 tok=15.923B dt=5486 wait=136 +15:25:32 step 121240 loss=3.3734 lr=6.09e-05 tok=15.925B dt=5549 wait=136 +15:26:35 step 121260 loss=3.3478 lr=6.09e-05 tok=15.928B dt=5613 wait=136 +15:27:39 step 121280 loss=3.3305 lr=6.09e-05 tok=15.930B dt=5676 wait=137 +15:28:42 step 121300 loss=3.3025 lr=6.08e-05 tok=15.933B dt=5739 wait=137 +15:29:45 step 121320 loss=3.4019 lr=6.08e-05 tok=15.936B dt=5803 wait=137 +15:30:49 step 121340 loss=3.2836 lr=6.08e-05 tok=15.938B dt=5866 wait=137 +15:31:33 pushed ckpt_w111.pt,log_w111.txt +15:32:48 waited 76s for global r=418 +15:32:54 global r=418 load 4.290 -> 4.291 +15:33:26 step 121360 loss=3.4319 lr=6.08e-05 tok=15.941B dt=6024 wait=138 +15:34:30 step 121380 loss=3.3468 lr=6.08e-05 tok=15.944B dt=6088 wait=138 +15:35:33 step 121400 loss=3.3191 lr=6.07e-05 tok=15.946B dt=6151 wait=138 +15:36:37 step 121420 loss=3.4046 lr=6.07e-05 tok=15.949B dt=6214 wait=139 +15:37:40 step 121440 loss=3.3067 lr=6.07e-05 tok=15.951B dt=6278 wait=139 +15:38:43 step 121460 loss=3.3979 lr=6.07e-05 tok=15.954B dt=6341 wait=139 +15:39:47 step 121480 loss=3.3905 lr=6.07e-05 tok=15.957B dt=6404 wait=140 +15:40:50 step 121500 loss=3.3320 lr=6.06e-05 tok=15.959B dt=6468 wait=140 +15:41:07 pushed ckpt_w111.pt,log_w111.txt +15:42:22 waited 76s for global r=419 +15:42:28 global r=419 load 4.328 -> 4.294 +15:43:33 step 121520 loss=3.3211 lr=6.06e-05 tok=15.962B dt=6630 wait=140 +15:44:36 step 121540 loss=3.4068 lr=6.06e-05 tok=15.965B dt=6694 wait=141 +15:45:39 step 121560 loss=3.3041 lr=6.06e-05 tok=15.967B dt=6757 wait=141 +15:46:43 step 121580 loss=3.3011 lr=6.06e-05 tok=15.970B dt=6820 wait=141 +15:47:46 step 121600 loss=3.2581 lr=6.05e-05 tok=15.972B dt=6883 wait=141 +15:48:49 step 121620 loss=3.3676 lr=6.05e-05 tok=15.975B dt=6947 wait=142 +15:49:52 step 121640 loss=3.2741 lr=6.05e-05 tok=15.978B dt=7010 wait=142 +15:50:38 pushed ckpt_w111.pt,log_w111.txt +15:51:54 waited 76s for global r=420 +15:52:00 global r=420 load 4.305 -> 4.302 +15:52:33 step 121660 loss=3.3642 lr=6.05e-05 tok=15.980B dt=7171 wait=142 +15:53:36 step 121680 loss=3.2984 lr=6.05e-05 tok=15.983B dt=7234 wait=143 +15:54:40 step 121700 loss=3.2969 lr=6.04e-05 tok=15.986B dt=7297 wait=143 +15:55:43 step 121720 loss=3.4246 lr=6.04e-05 tok=15.988B dt=7361 wait=143 +15:56:47 step 121740 loss=3.3146 lr=6.04e-05 tok=15.991B dt=7424 wait=144 +15:57:50 step 121760 loss=3.4335 lr=6.04e-05 tok=15.993B dt=7488 wait=144 +15:58:53 step 121780 loss=3.3674 lr=6.04e-05 tok=15.996B dt=7551 wait=144 +15:59:57 step 121800 loss=3.3831 lr=6.03e-05 tok=15.999B dt=7614 wait=145 +16:00:09 pushed ckpt_w111.pt,log_w111.txt +16:01:39 waited 91s for global r=421 +16:01:44 global r=421 load 4.304 -> 4.298 +16:02:49 step 121820 loss=3.3625 lr=6.03e-05 tok=16.001B dt=7786 wait=145 +16:03:52 step 121840 loss=3.3590 lr=6.03e-05 tok=16.004B dt=7850 wait=145 +16:04:55 step 121860 loss=3.3547 lr=6.03e-05 tok=16.007B dt=7913 wait=146 +16:05:59 step 121880 loss=3.2305 lr=6.03e-05 tok=16.009B dt=7977 wait=146 +16:07:02 step 121900 loss=3.2666 lr=6.02e-05 tok=16.012B dt=8040 wait=146 +16:08:06 step 121920 loss=3.4118 lr=6.02e-05 tok=16.014B dt=8103 wait=147 +16:09:08 step 121940 loss=3.2510 lr=6.02e-05 tok=16.017B dt=8166 wait=147 +16:09:52 pushed ckpt_w111.pt,log_w111.txt +16:11:08 waited 76s for global r=422 +16:11:14 global r=422 load 4.301 -> 4.295 +16:11:47 step 121960 loss=3.4008 lr=6.02e-05 tok=16.020B dt=8325 wait=147 +16:12:51 step 121980 loss=3.2545 lr=6.02e-05 tok=16.022B dt=8389 wait=147 +16:13:55 step 122000 loss=3.2389 lr=6.01e-05 tok=16.025B dt=8452 wait=148 +16:14:58 step 122020 loss=3.4399 lr=6.01e-05 tok=16.027B dt=8516 wait=148 +16:16:02 step 122040 loss=3.2937 lr=6.01e-05 tok=16.030B dt=8579 wait=149 +16:17:05 step 122060 loss=3.2513 lr=6.01e-05 tok=16.033B dt=8643 wait=149 +16:18:08 step 122080 loss=3.3372 lr=6.01e-05 tok=16.035B dt=8706 wait=149 +16:19:12 step 122100 loss=3.4219 lr=6.00e-05 tok=16.038B dt=8769 wait=149 +16:19:23 pushed ckpt_w111.pt,log_w111.txt +16:20:54 waited 91s for global r=423 +16:21:01 global r=423 load 4.296 -> 4.291 +16:22:06 step 122120 loss=3.3661 lr=6.00e-05 tok=16.041B dt=8944 wait=150 +16:23:10 step 122140 loss=3.3586 lr=6.00e-05 tok=16.043B dt=9007 wait=150 +16:24:13 step 122160 loss=3.3649 lr=6.00e-05 tok=16.046B dt=9071 wait=150 +16:25:17 step 122180 loss=3.2317 lr=6.00e-05 tok=16.048B dt=9135 wait=151 +16:26:20 step 122200 loss=3.3092 lr=5.99e-05 tok=16.051B dt=9198 wait=151 +16:27:24 step 122220 loss=3.3864 lr=5.99e-05 tok=16.054B dt=9262 wait=151 +16:28:28 step 122240 loss=3.2241 lr=5.99e-05 tok=16.056B dt=9325 wait=152 +16:29:11 pushed ckpt_w111.pt,log_w111.txt +16:30:27 waited 76s for global r=424 +16:30:34 global r=424 load 4.304 -> 4.289 +16:31:07 step 122260 loss=3.4023 lr=5.99e-05 tok=16.059B dt=9484 wait=152 +16:32:10 step 122280 loss=3.2812 lr=5.99e-05 tok=16.062B dt=9548 wait=152 +16:33:13 step 122300 loss=3.2980 lr=5.98e-05 tok=16.064B dt=9611 wait=153 +16:34:17 step 122320 loss=3.3797 lr=5.98e-05 tok=16.067B dt=9675 wait=153 +16:35:20 step 122340 loss=3.3067 lr=5.98e-05 tok=16.069B dt=9738 wait=153 +16:36:25 step 122360 loss=2.9049 lr=5.98e-05 tok=16.072B dt=9803 wait=153 +16:37:28 step 122380 loss=3.3153 lr=5.98e-05 tok=16.075B dt=9866 wait=154 +16:38:32 step 122400 loss=3.3544 lr=5.97e-05 tok=16.077B dt=9929 wait=154 diff --git a/log_w112.txt b/log_w112.txt new file mode 100644 index 0000000000000000000000000000000000000000..7d470ea77f015ae1537addc8a6429eac4dabfbff --- /dev/null +++ b/log_w112.txt @@ -0,0 +1,186 @@ +launch +13:51:32 name=w112 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 59/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:51:38 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:51:38 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:51:48 params 1.08B +13:51:49 resumed step=119850 seen=15.74B round=408 +13:51:49 stream salt=119850 +13:53:32 blocks compiled +13:53:32 start: accum=4, 131072 tokens per optimiser step +13:55:26 step 119851 loss=3.1721 lr=6.23e-05 tok=15.743B dt=114 wait=75 +13:55:29 step 119852 loss=3.1546 lr=6.23e-05 tok=15.743B dt=117 wait=75 +13:55:58 step 119860 loss=3.4168 lr=6.23e-05 tok=15.744B dt=146 wait=75 +13:57:08 step 119880 loss=3.3070 lr=6.23e-05 tok=15.747B dt=216 wait=76 +13:58:14 step 119900 loss=3.4177 lr=6.22e-05 tok=15.750B dt=282 wait=76 +13:59:18 step 119920 loss=3.3433 lr=6.22e-05 tok=15.752B dt=346 wait=76 +14:00:22 step 119940 loss=3.4156 lr=6.22e-05 tok=15.755B dt=410 wait=77 +14:01:26 step 119960 loss=3.5200 lr=6.22e-05 tok=15.757B dt=474 wait=77 +14:02:30 step 119980 loss=3.3447 lr=6.22e-05 tok=15.760B dt=538 wait=77 +14:03:34 step 120000 loss=3.4038 lr=6.21e-05 tok=15.763B dt=602 wait=78 +14:03:49 pushed ckpt_w112.pt,log_w112.txt +14:07:36 global r=409 load 4.281 -> 4.279 +14:08:39 step 120020 loss=3.3378 lr=6.21e-05 tok=15.767B dt=907 wait=78 +14:09:43 step 120040 loss=3.3754 lr=6.21e-05 tok=15.769B dt=971 wait=78 +14:10:46 step 120060 loss=3.4077 lr=6.21e-05 tok=15.772B dt=1034 wait=79 +14:11:50 step 120080 loss=3.3264 lr=6.21e-05 tok=15.775B dt=1098 wait=79 +14:12:54 step 120100 loss=3.3655 lr=6.20e-05 tok=15.777B dt=1162 wait=79 +14:13:58 step 120120 loss=3.4382 lr=6.20e-05 tok=15.780B dt=1226 wait=80 +14:15:02 step 120140 loss=3.3136 lr=6.20e-05 tok=15.782B dt=1290 wait=80 +14:15:45 pushed ckpt_w112.pt,log_w112.txt +14:16:31 waited 45s for global r=410 +14:16:36 global r=410 load 4.298 -> 4.291 +14:17:09 step 120160 loss=3.4109 lr=6.20e-05 tok=15.785B dt=1417 wait=80 +14:18:13 step 120180 loss=3.3162 lr=6.20e-05 tok=15.788B dt=1481 wait=81 +14:19:17 step 120200 loss=3.3474 lr=6.19e-05 tok=15.790B dt=1545 wait=81 +14:20:21 step 120220 loss=3.4102 lr=6.19e-05 tok=15.793B dt=1609 wait=81 +14:21:24 step 120240 loss=3.2981 lr=6.19e-05 tok=15.795B dt=1672 wait=82 +14:22:28 step 120260 loss=3.4465 lr=6.19e-05 tok=15.798B dt=1736 wait=82 +14:23:32 step 120280 loss=3.3095 lr=6.19e-05 tok=15.801B dt=1800 wait=82 +14:24:36 step 120300 loss=3.3779 lr=6.18e-05 tok=15.803B dt=1864 wait=83 +14:24:51 pushed ckpt_w112.pt,log_w112.txt +14:26:06 waited 76s for global r=411 +14:26:14 global r=411 load 4.293 -> 4.277 +14:27:19 step 120320 loss=3.3431 lr=6.18e-05 tok=15.806B dt=2027 wait=83 +14:28:23 step 120340 loss=3.3600 lr=6.18e-05 tok=15.809B dt=2091 wait=83 +14:29:26 step 120360 loss=3.3731 lr=6.18e-05 tok=15.811B dt=2154 wait=84 +14:30:30 step 120380 loss=3.3342 lr=6.18e-05 tok=15.814B dt=2218 wait=84 +14:31:33 step 120400 loss=3.3287 lr=6.17e-05 tok=15.816B dt=2281 wait=84 +14:32:37 step 120420 loss=3.4423 lr=6.17e-05 tok=15.819B dt=2345 wait=85 +14:33:41 step 120440 loss=3.3322 lr=6.17e-05 tok=15.822B dt=2409 wait=85 +14:34:26 pushed ckpt_w112.pt,log_w112.txt +14:35:42 waited 76s for global r=412 +14:35:47 global r=412 load 4.311 -> 4.284 +14:36:20 step 120460 loss=3.3845 lr=6.17e-05 tok=15.824B dt=2568 wait=85 +14:37:24 step 120480 loss=3.2952 lr=6.17e-05 tok=15.827B dt=2632 wait=86 +14:38:28 step 120500 loss=3.3530 lr=6.16e-05 tok=15.830B dt=2696 wait=86 +14:39:32 step 120520 loss=3.4537 lr=6.16e-05 tok=15.832B dt=2760 wait=86 +14:40:36 step 120540 loss=3.3396 lr=6.16e-05 tok=15.835B dt=2824 wait=86 +14:41:40 step 120560 loss=3.4129 lr=6.16e-05 tok=15.837B dt=2888 wait=87 +14:42:44 step 120580 loss=3.3783 lr=6.16e-05 tok=15.840B dt=2952 wait=87 +14:43:48 step 120600 loss=3.3242 lr=6.15e-05 tok=15.843B dt=3016 wait=87 +14:44:02 pushed ckpt_w112.pt,log_w112.txt +14:45:18 waited 76s for global r=413 +14:45:24 global r=413 load 4.292 -> 4.292 +14:46:29 step 120620 loss=3.3117 lr=6.15e-05 tok=15.845B dt=3177 wait=88 +14:47:33 step 120640 loss=3.3758 lr=6.15e-05 tok=15.848B dt=3241 wait=88 +14:48:36 step 120660 loss=3.3819 lr=6.15e-05 tok=15.851B dt=3304 wait=88 +14:49:40 step 120680 loss=3.3268 lr=6.15e-05 tok=15.853B dt=3368 wait=89 +14:50:44 step 120700 loss=3.3528 lr=6.14e-05 tok=15.856B dt=3432 wait=89 +14:51:48 step 120720 loss=3.4341 lr=6.14e-05 tok=15.858B dt=3496 wait=89 +14:52:52 step 120740 loss=3.2943 lr=6.14e-05 tok=15.861B dt=3560 wait=89 +14:53:36 pushed ckpt_w112.pt,log_w112.txt +14:54:52 waited 76s for global r=414 +14:55:00 global r=414 load 4.271 -> 4.279 +14:55:33 step 120760 loss=3.4988 lr=6.14e-05 tok=15.864B dt=3721 wait=90 +14:56:37 step 120780 loss=3.2409 lr=6.14e-05 tok=15.866B dt=3784 wait=90 +14:57:41 step 120800 loss=3.2856 lr=6.13e-05 tok=15.869B dt=3849 wait=90 +14:58:44 step 120820 loss=3.4876 lr=6.13e-05 tok=15.872B dt=3912 wait=91 +14:59:48 step 120840 loss=3.3056 lr=6.13e-05 tok=15.874B dt=3976 wait=91 +15:00:52 step 120860 loss=3.4086 lr=6.13e-05 tok=15.877B dt=4040 wait=91 +15:01:56 step 120880 loss=3.3656 lr=6.13e-05 tok=15.879B dt=4104 wait=92 +15:02:59 step 120900 loss=3.4415 lr=6.12e-05 tok=15.882B dt=4167 wait=92 +15:03:11 pushed ckpt_w112.pt,log_w112.txt +15:04:27 waited 76s for global r=415 +15:04:33 global r=415 load 4.283 -> 4.273 +15:05:37 step 120920 loss=3.3598 lr=6.12e-05 tok=15.885B dt=4325 wait=92 +15:06:41 step 120940 loss=3.3219 lr=6.12e-05 tok=15.887B dt=4389 wait=93 +15:07:45 step 120960 loss=3.3549 lr=6.12e-05 tok=15.890B dt=4453 wait=93 +15:08:49 step 120980 loss=3.2909 lr=6.12e-05 tok=15.892B dt=4517 wait=93 +15:09:52 step 121000 loss=3.3418 lr=6.11e-05 tok=15.895B dt=4580 wait=94 +15:10:55 step 121020 loss=3.4322 lr=6.11e-05 tok=15.898B dt=4643 wait=94 +15:11:59 step 121040 loss=3.3034 lr=6.11e-05 tok=15.900B dt=4707 wait=94 +15:12:45 pushed ckpt_w112.pt,log_w112.txt +15:13:46 waited 61s for global r=416 +15:13:53 global r=416 load 4.291 -> 4.281 +15:14:26 step 121060 loss=3.4012 lr=6.11e-05 tok=15.903B dt=4854 wait=95 +15:15:30 step 121080 loss=3.2998 lr=6.11e-05 tok=15.906B dt=4918 wait=95 +15:16:34 step 121100 loss=3.2941 lr=6.10e-05 tok=15.908B dt=4981 wait=95 +15:17:37 step 121120 loss=3.3871 lr=6.10e-05 tok=15.911B dt=5045 wait=95 +15:18:41 step 121140 loss=3.3286 lr=6.10e-05 tok=15.913B dt=5109 wait=96 +15:19:45 step 121160 loss=3.4190 lr=6.10e-05 tok=15.916B dt=5172 wait=96 +15:20:48 step 121180 loss=3.3464 lr=6.10e-05 tok=15.919B dt=5236 wait=96 +15:21:52 step 121200 loss=3.2850 lr=6.09e-05 tok=15.921B dt=5300 wait=97 +15:22:06 pushed ckpt_w112.pt,log_w112.txt +15:23:21 waited 76s for global r=417 +15:23:28 global r=417 load 4.301 -> 4.286 +15:24:33 step 121220 loss=3.3235 lr=6.09e-05 tok=15.924B dt=5461 wait=97 +15:25:37 step 121240 loss=3.3677 lr=6.09e-05 tok=15.927B dt=5525 wait=97 +15:26:41 step 121260 loss=3.3934 lr=6.09e-05 tok=15.929B dt=5589 wait=98 +15:27:45 step 121280 loss=3.3075 lr=6.09e-05 tok=15.932B dt=5653 wait=98 +15:28:48 step 121300 loss=3.3440 lr=6.08e-05 tok=15.934B dt=5716 wait=98 +15:29:52 step 121320 loss=3.4116 lr=6.08e-05 tok=15.937B dt=5780 wait=99 +15:30:56 step 121340 loss=3.2655 lr=6.08e-05 tok=15.940B dt=5844 wait=99 +15:31:41 pushed ckpt_w112.pt,log_w112.txt +15:32:56 waited 76s for global r=418 +15:33:02 global r=418 load 4.293 -> 4.291 +15:33:35 step 121360 loss=3.4669 lr=6.08e-05 tok=15.942B dt=6003 wait=99 +15:34:38 step 121380 loss=3.3064 lr=6.08e-05 tok=15.945B dt=6066 wait=100 +15:35:43 step 121400 loss=3.2152 lr=6.07e-05 tok=15.948B dt=6130 wait=100 +15:36:47 step 121420 loss=3.3065 lr=6.07e-05 tok=15.950B dt=6195 wait=100 +15:37:51 step 121440 loss=3.3348 lr=6.07e-05 tok=15.953B dt=6259 wait=100 +15:38:55 step 121460 loss=3.3979 lr=6.07e-05 tok=15.955B dt=6323 wait=101 +15:39:59 step 121480 loss=3.3826 lr=6.07e-05 tok=15.958B dt=6387 wait=101 +15:41:03 step 121500 loss=3.4316 lr=6.06e-05 tok=15.961B dt=6451 wait=101 +15:41:21 pushed ckpt_w112.pt,log_w112.txt +15:42:21 waited 60s for global r=419 +15:42:29 global r=419 load 4.270 -> 4.294 +15:43:34 step 121520 loss=3.1800 lr=6.06e-05 tok=15.963B dt=6602 wait=102 +15:44:38 step 121540 loss=3.3825 lr=6.06e-05 tok=15.966B dt=6666 wait=102 +15:45:41 step 121560 loss=3.3811 lr=6.06e-05 tok=15.969B dt=6729 wait=102 +15:46:45 step 121580 loss=3.3270 lr=6.06e-05 tok=15.971B dt=6793 wait=103 +15:47:49 step 121600 loss=3.2881 lr=6.05e-05 tok=15.974B dt=6857 wait=103 +15:48:53 step 121620 loss=3.4157 lr=6.05e-05 tok=15.976B dt=6920 wait=103 +15:49:56 step 121640 loss=3.2860 lr=6.05e-05 tok=15.979B dt=6984 wait=104 +15:50:40 pushed ckpt_w112.pt,log_w112.txt +15:51:55 waited 76s for global r=420 +15:52:01 global r=420 load 4.340 -> 4.302 +15:52:34 step 121660 loss=3.4569 lr=6.05e-05 tok=15.982B dt=7142 wait=104 +15:53:38 step 121680 loss=3.3776 lr=6.05e-05 tok=15.984B dt=7206 wait=104 +15:54:42 step 121700 loss=3.3076 lr=6.04e-05 tok=15.987B dt=7269 wait=105 +15:55:45 step 121720 loss=3.4283 lr=6.04e-05 tok=15.989B dt=7333 wait=105 +15:56:49 step 121740 loss=3.3666 lr=6.04e-05 tok=15.992B dt=7397 wait=105 +15:57:53 step 121760 loss=3.4062 lr=6.04e-05 tok=15.995B dt=7461 wait=106 +15:58:57 step 121780 loss=3.0390 lr=6.04e-05 tok=15.997B dt=7525 wait=106 +16:00:01 step 121800 loss=3.3536 lr=6.03e-05 tok=16.000B dt=7589 wait=106 +16:00:13 pushed ckpt_w112.pt,log_w112.txt +16:01:44 waited 91s for global r=421 +16:01:51 global r=421 load 4.290 -> 4.298 +16:02:56 step 121820 loss=3.2762 lr=6.03e-05 tok=16.003B dt=7764 wait=107 +16:04:00 step 121840 loss=3.3454 lr=6.03e-05 tok=16.005B dt=7828 wait=107 +16:05:04 step 121860 loss=3.3765 lr=6.03e-05 tok=16.008B dt=7892 wait=107 +16:06:07 step 121880 loss=3.2439 lr=6.03e-05 tok=16.010B dt=7955 wait=108 +16:07:11 step 121900 loss=3.3471 lr=6.02e-05 tok=16.013B dt=8019 wait=108 +16:08:15 step 121920 loss=3.4009 lr=6.02e-05 tok=16.016B dt=8083 wait=108 +16:09:19 step 121940 loss=3.2973 lr=6.02e-05 tok=16.018B dt=8147 wait=109 +16:10:02 pushed ckpt_w112.pt,log_w112.txt +16:11:18 waited 76s for global r=422 +16:11:24 global r=422 load 4.299 -> 4.295 +16:11:57 step 121960 loss=3.4130 lr=6.02e-05 tok=16.021B dt=8305 wait=109 +16:13:01 step 121980 loss=3.3404 lr=6.02e-05 tok=16.024B dt=8369 wait=109 +16:14:05 step 122000 loss=3.3643 lr=6.01e-05 tok=16.026B dt=8433 wait=109 +16:15:09 step 122020 loss=3.4424 lr=6.01e-05 tok=16.029B dt=8497 wait=110 +16:16:12 step 122040 loss=3.3322 lr=6.01e-05 tok=16.031B dt=8560 wait=110 +16:17:16 step 122060 loss=3.3217 lr=6.01e-05 tok=16.034B dt=8624 wait=110 +16:18:20 step 122080 loss=3.3566 lr=6.01e-05 tok=16.037B dt=8688 wait=111 +16:19:24 step 122100 loss=3.4583 lr=6.00e-05 tok=16.039B dt=8752 wait=111 +16:19:36 pushed ckpt_w112.pt,log_w112.txt +16:20:51 waited 76s for global r=423 +16:20:58 global r=423 load 4.295 -> 4.291 +16:22:03 step 122120 loss=3.3068 lr=6.00e-05 tok=16.042B dt=8911 wait=111 +16:23:07 step 122140 loss=3.3610 lr=6.00e-05 tok=16.045B dt=8975 wait=112 +16:24:11 step 122160 loss=3.3863 lr=6.00e-05 tok=16.047B dt=9039 wait=112 +16:25:15 step 122180 loss=3.3302 lr=6.00e-05 tok=16.050B dt=9103 wait=112 +16:26:19 step 122200 loss=3.3011 lr=5.99e-05 tok=16.052B dt=9167 wait=113 +16:27:23 step 122220 loss=3.4094 lr=5.99e-05 tok=16.055B dt=9231 wait=113 +16:28:27 step 122240 loss=3.3466 lr=5.99e-05 tok=16.058B dt=9294 wait=113 +16:29:10 pushed ckpt_w112.pt,log_w112.txt +16:30:25 waited 76s for global r=424 +16:30:32 global r=424 load 4.299 -> 4.289 +16:31:05 step 122260 loss=3.4498 lr=5.99e-05 tok=16.060B dt=9453 wait=114 +16:32:09 step 122280 loss=3.3146 lr=5.99e-05 tok=16.063B dt=9516 wait=114 +16:33:12 step 122300 loss=3.3447 lr=5.98e-05 tok=16.065B dt=9580 wait=114 +16:34:17 step 122320 loss=3.4449 lr=5.98e-05 tok=16.068B dt=9645 wait=115 +16:35:21 step 122340 loss=3.3234 lr=5.98e-05 tok=16.071B dt=9709 wait=115 +16:36:25 step 122360 loss=3.4542 lr=5.98e-05 tok=16.073B dt=9773 wait=115 +16:37:29 step 122380 loss=3.3733 lr=5.98e-05 tok=16.076B dt=9837 wait=115 +16:38:33 step 122400 loss=3.1996 lr=5.97e-05 tok=16.079B dt=9901 wait=116 diff --git a/log_w113.txt b/log_w113.txt new file mode 100644 index 0000000000000000000000000000000000000000..7d073119fb14c16b0ae7345fef13c377e57d8cc4 --- /dev/null +++ b/log_w113.txt @@ -0,0 +1,18 @@ +launch +16:55:08 name=w113 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 60/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:55:15 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:55:15 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:55:24 params 1.08B +16:55:26 resumed step=122400 seen=16.08B round=425 +16:55:26 stream salt=122400 +16:57:18 blocks compiled +16:57:18 start: accum=4, 131072 tokens per optimiser step +17:00:21 step 122401 loss=2.8225 lr=5.97e-05 tok=16.079B dt=183 wait=147 +17:00:24 step 122402 loss=2.9789 lr=5.97e-05 tok=16.079B dt=186 wait=147 +17:01:25 step 122420 loss=3.3340 lr=5.97e-05 tok=16.081B dt=247 wait=147 +17:02:33 step 122440 loss=3.3835 lr=5.97e-05 tok=16.084B dt=315 wait=148 +17:03:36 step 122460 loss=3.4459 lr=5.97e-05 tok=16.086B dt=378 wait=148 +17:04:40 step 122480 loss=3.3921 lr=5.97e-05 tok=16.089B dt=442 wait=149 +17:05:44 step 122500 loss=3.4389 lr=5.96e-05 tok=16.092B dt=506 wait=149 +17:06:47 step 122520 loss=3.3834 lr=5.96e-05 tok=16.094B dt=569 wait=149 +17:07:51 step 122540 loss=3.3612 lr=5.96e-05 tok=16.097B dt=633 wait=150 diff --git a/log_w115.txt b/log_w115.txt new file mode 100644 index 0000000000000000000000000000000000000000..21c0f9b7540f340a5b15a68c9309e05c9aeeea01 --- /dev/null +++ b/log_w115.txt @@ -0,0 +1,195 @@ +launch +16:55:09 name=w115 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 62/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:55:15 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:55:15 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:55:25 params 1.08B +16:55:27 resumed step=122400 seen=16.08B round=425 +16:55:27 stream salt=122400 +16:57:21 blocks compiled +16:57:21 start: accum=4, 131072 tokens per optimiser step +17:01:02 step 122401 loss=3.0337 lr=5.97e-05 tok=16.079B dt=221 wait=174 +17:01:08 step 122402 loss=2.9563 lr=5.97e-05 tok=16.079B dt=227 wait=175 +17:02:17 step 122420 loss=3.3527 lr=5.97e-05 tok=16.081B dt=296 wait=175 +17:03:21 step 122440 loss=3.3361 lr=5.97e-05 tok=16.084B dt=360 wait=176 +17:04:24 step 122460 loss=3.3453 lr=5.97e-05 tok=16.086B dt=423 wait=176 +17:05:28 step 122480 loss=3.2931 lr=5.97e-05 tok=16.089B dt=487 wait=176 +17:06:31 step 122500 loss=3.4376 lr=5.96e-05 tok=16.092B dt=550 wait=177 +17:07:35 step 122520 loss=3.3394 lr=5.96e-05 tok=16.094B dt=614 wait=177 +17:08:38 step 122540 loss=3.3154 lr=5.96e-05 tok=16.097B dt=678 wait=178 +17:09:24 pushed ckpt_w115.pt,log_w115.txt +17:09:25 waited 0s for global r=426 +17:09:42 global r=426 load 4.271 -> 4.287 +17:10:25 step 122560 loss=3.4207 lr=5.96e-05 tok=16.100B dt=784 wait=178 +17:11:40 step 122580 loss=3.2270 lr=5.96e-05 tok=16.102B dt=859 wait=178 +17:12:44 step 122600 loss=3.3137 lr=5.95e-05 tok=16.105B dt=923 wait=179 +17:13:47 step 122620 loss=3.4482 lr=5.95e-05 tok=16.107B dt=986 wait=179 +17:14:51 step 122640 loss=3.3056 lr=5.95e-05 tok=16.110B dt=1050 wait=179 +17:15:54 step 122660 loss=3.4171 lr=5.95e-05 tok=16.113B dt=1113 wait=180 +17:16:58 step 122680 loss=3.3575 lr=5.95e-05 tok=16.115B dt=1177 wait=180 +17:18:01 step 122700 loss=3.3543 lr=5.94e-05 tok=16.118B dt=1240 wait=180 +17:18:16 pushed ckpt_w115.pt,log_w115.txt +17:19:16 waited 60s for global r=427 +17:19:23 global r=427 load 4.276 -> 4.284 +17:20:29 step 122720 loss=3.3596 lr=5.94e-05 tok=16.121B dt=1388 wait=181 +17:21:39 step 122740 loss=3.4000 lr=5.94e-05 tok=16.123B dt=1459 wait=181 +17:22:55 step 122760 loss=3.3181 lr=5.94e-05 tok=16.126B dt=1534 wait=181 +17:23:59 step 122780 loss=3.2765 lr=5.94e-05 tok=16.128B dt=1598 wait=182 +17:25:02 step 122800 loss=3.2653 lr=5.93e-05 tok=16.131B dt=1661 wait=182 +17:26:06 step 122820 loss=3.3773 lr=5.93e-05 tok=16.134B dt=1725 wait=182 +17:27:10 step 122840 loss=3.2312 lr=5.93e-05 tok=16.136B dt=1789 wait=183 +17:27:55 pushed ckpt_w115.pt,log_w115.txt +17:28:56 waited 61s for global r=428 +17:29:02 global r=428 load 4.285 -> 4.284 +17:29:36 step 122860 loss=3.3494 lr=5.93e-05 tok=16.139B dt=1935 wait=183 +17:30:39 step 122880 loss=3.2778 lr=5.93e-05 tok=16.142B dt=1998 wait=184 +17:32:01 step 122900 loss=3.2593 lr=5.92e-05 tok=16.144B dt=2080 wait=184 +17:33:07 step 122920 loss=3.3944 lr=5.92e-05 tok=16.147B dt=2146 wait=184 +17:34:10 step 122940 loss=3.3211 lr=5.92e-05 tok=16.149B dt=2209 wait=185 +17:35:14 step 122960 loss=3.4123 lr=5.92e-05 tok=16.152B dt=2273 wait=185 +17:36:17 step 122980 loss=3.3137 lr=5.92e-05 tok=16.155B dt=2336 wait=185 +17:37:20 step 123000 loss=3.3341 lr=5.91e-05 tok=16.157B dt=2399 wait=186 +17:37:33 pushed ckpt_w115.pt,log_w115.txt +17:38:33 waited 61s for global r=429 +17:38:42 global r=429 load 4.279 -> 4.277 +17:39:47 step 123020 loss=3.2712 lr=5.91e-05 tok=16.160B dt=2546 wait=186 +17:40:59 step 123040 loss=3.2801 lr=5.91e-05 tok=16.162B dt=2618 wait=186 +17:42:14 step 123060 loss=3.3617 lr=5.91e-05 tok=16.165B dt=2693 wait=187 +17:43:17 step 123080 loss=3.3140 lr=5.91e-05 tok=16.168B dt=2756 wait=187 +17:44:21 step 123100 loss=3.2763 lr=5.90e-05 tok=16.170B dt=2820 wait=187 +17:45:25 step 123120 loss=3.3854 lr=5.90e-05 tok=16.173B dt=2884 wait=188 +17:46:28 step 123140 loss=3.3058 lr=5.90e-05 tok=16.176B dt=2947 wait=188 +17:47:12 pushed ckpt_w115.pt,log_w115.txt +17:48:12 waited 60s for global r=430 +17:48:18 global r=430 load 4.309 -> 4.288 +17:48:51 step 123160 loss=3.3821 lr=5.90e-05 tok=16.178B dt=3090 wait=188 +17:49:54 step 123180 loss=3.2686 lr=5.90e-05 tok=16.181B dt=3153 wait=189 +17:51:17 step 123200 loss=3.2789 lr=5.89e-05 tok=16.183B dt=3236 wait=189 +17:52:21 step 123220 loss=3.4190 lr=5.89e-05 tok=16.186B dt=3300 wait=189 +17:53:25 step 123240 loss=3.2490 lr=5.89e-05 tok=16.189B dt=3364 wait=190 +17:54:29 step 123260 loss=3.3193 lr=5.89e-05 tok=16.191B dt=3428 wait=190 +17:55:32 step 123280 loss=3.2958 lr=5.89e-05 tok=16.194B dt=3491 wait=190 +17:56:36 step 123300 loss=3.3581 lr=5.88e-05 tok=16.197B dt=3555 wait=191 +17:56:48 pushed ckpt_w115.pt,log_w115.txt +17:57:33 waited 46s for global r=431 +17:57:39 global r=431 load 4.317 -> 4.295 +17:58:44 step 123320 loss=3.2409 lr=5.88e-05 tok=16.199B dt=3683 wait=191 +17:59:54 step 123340 loss=3.3778 lr=5.88e-05 tok=16.202B dt=3753 wait=191 +18:01:11 step 123360 loss=3.3116 lr=5.88e-05 tok=16.204B dt=3830 wait=192 +18:02:14 step 123380 loss=3.3159 lr=5.88e-05 tok=16.207B dt=3893 wait=192 +18:03:18 step 123400 loss=3.3084 lr=5.87e-05 tok=16.210B dt=3957 wait=192 +18:04:21 step 123420 loss=3.3852 lr=5.87e-05 tok=16.212B dt=4020 wait=193 +18:05:24 step 123440 loss=3.2423 lr=5.87e-05 tok=16.215B dt=4083 wait=193 +18:06:10 pushed ckpt_w115.pt,log_w115.txt +18:07:10 waited 61s for global r=432 +18:07:18 global r=432 load 4.316 -> 4.294 +18:07:51 step 123460 loss=3.4011 lr=5.87e-05 tok=16.218B dt=4230 wait=194 +18:08:55 step 123480 loss=3.3204 lr=5.87e-05 tok=16.220B dt=4294 wait=194 +18:10:16 step 123500 loss=3.2899 lr=5.86e-05 tok=16.223B dt=4375 wait=194 +18:11:22 step 123520 loss=3.3618 lr=5.86e-05 tok=16.225B dt=4441 wait=195 +18:12:25 step 123540 loss=3.3014 lr=5.86e-05 tok=16.228B dt=4504 wait=195 +18:13:28 step 123560 loss=3.3395 lr=5.86e-05 tok=16.231B dt=4567 wait=195 +18:14:32 step 123580 loss=3.3229 lr=5.86e-05 tok=16.233B dt=4631 wait=196 +18:15:35 step 123600 loss=3.2597 lr=5.85e-05 tok=16.236B dt=4694 wait=196 +18:15:48 pushed ckpt_w115.pt,log_w115.txt +18:16:34 waited 46s for global r=433 +18:16:42 global r=433 load 4.297 -> 4.281 +18:17:47 step 123620 loss=3.2880 lr=5.85e-05 tok=16.239B dt=4826 wait=196 +18:18:56 step 123640 loss=3.3468 lr=5.85e-05 tok=16.241B dt=4895 wait=197 +18:20:13 step 123660 loss=3.3364 lr=5.85e-05 tok=16.244B dt=4972 wait=197 +18:21:16 step 123680 loss=3.2644 lr=5.85e-05 tok=16.246B dt=5035 wait=198 +18:22:20 step 123700 loss=3.3222 lr=5.84e-05 tok=16.249B dt=5099 wait=198 +18:23:23 step 123720 loss=3.4035 lr=5.84e-05 tok=16.252B dt=5162 wait=198 +18:24:27 step 123740 loss=3.3016 lr=5.84e-05 tok=16.254B dt=5226 wait=199 +18:25:12 pushed ckpt_w115.pt,log_w115.txt +18:26:12 waited 60s for global r=434 +18:26:22 global r=434 load 4.286 -> 4.290 +18:26:55 step 123760 loss=3.4285 lr=5.84e-05 tok=16.257B dt=5374 wait=199 +18:27:58 step 123780 loss=3.2377 lr=5.84e-05 tok=16.259B dt=5438 wait=199 +18:29:18 step 123800 loss=3.2550 lr=5.83e-05 tok=16.262B dt=5517 wait=200 +18:30:25 step 123820 loss=3.3895 lr=5.83e-05 tok=16.265B dt=5584 wait=200 +18:31:29 step 123840 loss=3.2380 lr=5.83e-05 tok=16.267B dt=5648 wait=200 +18:32:32 step 123860 loss=3.3709 lr=5.83e-05 tok=16.270B dt=5712 wait=201 +18:33:36 step 123880 loss=3.2828 lr=5.83e-05 tok=16.273B dt=5775 wait=201 +18:34:40 step 123900 loss=3.4023 lr=5.82e-05 tok=16.275B dt=5839 wait=201 +18:34:52 pushed ckpt_w115.pt,log_w115.txt +18:38:59 step 123920 loss=3.3905 lr=5.82e-05 tok=16.278B dt=6098 wait=202 +18:40:02 step 123940 loss=3.3189 lr=5.82e-05 tok=16.280B dt=6161 wait=202 +18:41:05 step 123960 loss=3.2242 lr=5.82e-05 tok=16.283B dt=6224 wait=202 +18:42:09 step 123980 loss=3.2905 lr=5.82e-05 tok=16.286B dt=6288 wait=203 +18:43:12 step 124000 loss=3.2093 lr=5.81e-05 tok=16.288B dt=6351 wait=203 +18:44:16 step 124020 loss=3.3838 lr=5.81e-05 tok=16.291B dt=6415 wait=203 +18:45:19 step 124040 loss=3.3338 lr=5.81e-05 tok=16.294B dt=6478 wait=204 +18:46:06 pushed ckpt_w115.pt,log_w115.txt +18:46:51 waited 45s for global r=435 +18:47:01 global r=435 load 4.310 -> 4.286 +18:47:34 step 124060 loss=3.4452 lr=5.81e-05 tok=16.296B dt=6613 wait=204 +18:48:59 step 124080 loss=3.2670 lr=5.81e-05 tok=16.299B dt=6698 wait=204 +18:50:06 step 124100 loss=3.3042 lr=5.80e-05 tok=16.301B dt=6765 wait=205 +18:51:09 step 124120 loss=3.3961 lr=5.80e-05 tok=16.304B dt=6828 wait=205 +18:52:12 step 124140 loss=3.2733 lr=5.80e-05 tok=16.307B dt=6891 wait=205 +18:53:16 step 124160 loss=3.3371 lr=5.80e-05 tok=16.309B dt=6955 wait=206 +18:54:20 step 124180 loss=3.3114 lr=5.80e-05 tok=16.312B dt=7019 wait=206 +18:55:23 step 124200 loss=3.3816 lr=5.79e-05 tok=16.315B dt=7082 wait=206 +18:55:37 pushed ckpt_w115.pt,log_w115.txt +18:56:23 waited 45s for global r=436 +18:56:31 global r=436 load 4.277 -> 4.277 +18:57:42 step 124220 loss=3.3362 lr=5.79e-05 tok=16.317B dt=7221 wait=207 +18:58:58 step 124240 loss=3.3394 lr=5.79e-05 tok=16.320B dt=7297 wait=207 +19:00:02 step 124260 loss=3.3393 lr=5.79e-05 tok=16.322B dt=7361 wait=207 +19:01:05 step 124280 loss=3.3617 lr=5.79e-05 tok=16.325B dt=7424 wait=208 +19:02:09 step 124300 loss=3.2871 lr=5.78e-05 tok=16.328B dt=7488 wait=208 +19:03:12 step 124320 loss=3.3769 lr=5.78e-05 tok=16.330B dt=7551 wait=209 +19:04:15 step 124340 loss=3.2881 lr=5.78e-05 tok=16.333B dt=7614 wait=209 +19:05:01 pushed ckpt_w115.pt,log_w115.txt +19:06:01 waited 61s for global r=437 +19:06:07 global r=437 load 4.288 -> 4.282 +19:06:41 step 124360 loss=3.4491 lr=5.78e-05 tok=16.336B dt=7760 wait=209 +19:08:00 step 124380 loss=3.3046 lr=5.78e-05 tok=16.338B dt=7839 wait=210 +19:09:07 step 124400 loss=3.3111 lr=5.78e-05 tok=16.341B dt=7906 wait=210 +19:10:11 step 124420 loss=3.3102 lr=5.77e-05 tok=16.343B dt=7970 wait=210 +19:11:15 step 124440 loss=3.1710 lr=5.77e-05 tok=16.346B dt=8034 wait=211 +19:12:19 step 124460 loss=3.3272 lr=5.77e-05 tok=16.349B dt=8098 wait=211 +19:13:22 step 124480 loss=3.3272 lr=5.77e-05 tok=16.351B dt=8161 wait=211 +19:14:26 step 124500 loss=3.2979 lr=5.77e-05 tok=16.354B dt=8225 wait=212 +19:14:40 pushed ckpt_w115.pt,log_w115.txt +19:15:10 waited 30s for global r=438 +19:15:17 global r=438 load 4.299 -> 4.283 +19:16:22 step 124520 loss=3.3164 lr=5.76e-05 tok=16.356B dt=8341 wait=212 +19:17:45 step 124540 loss=3.3954 lr=5.76e-05 tok=16.359B dt=8424 wait=212 +19:18:48 step 124560 loss=3.3005 lr=5.76e-05 tok=16.362B dt=8487 wait=213 +19:19:52 step 124580 loss=3.3268 lr=5.76e-05 tok=16.364B dt=8551 wait=213 +19:20:55 step 124600 loss=3.3193 lr=5.76e-05 tok=16.367B dt=8614 wait=213 +19:21:59 step 124620 loss=3.4583 lr=5.75e-05 tok=16.370B dt=8678 wait=214 +19:23:02 step 124640 loss=3.2577 lr=5.75e-05 tok=16.372B dt=8741 wait=214 +19:23:48 pushed ckpt_w115.pt,log_w115.txt +19:24:49 waited 61s for global r=439 +19:24:56 global r=439 load 4.286 -> 4.292 +19:25:29 step 124660 loss=3.3814 lr=5.75e-05 tok=16.375B dt=8888 wait=214 +19:26:41 step 124680 loss=3.2942 lr=5.75e-05 tok=16.377B dt=8960 wait=215 +19:27:56 step 124700 loss=3.2784 lr=5.75e-05 tok=16.380B dt=9035 wait=215 +19:28:59 step 124720 loss=3.3898 lr=5.74e-05 tok=16.383B dt=9099 wait=215 +19:30:03 step 124740 loss=3.3250 lr=5.74e-05 tok=16.385B dt=9162 wait=216 +19:31:06 step 124760 loss=3.3745 lr=5.74e-05 tok=16.388B dt=9225 wait=216 +19:32:10 step 124780 loss=3.3224 lr=5.74e-05 tok=16.391B dt=9289 wait=216 +19:33:13 step 124800 loss=3.3491 lr=5.74e-05 tok=16.393B dt=9352 wait=217 +19:33:25 pushed ckpt_w115.pt,log_w115.txt +19:34:25 waited 61s for global r=440 +19:34:32 global r=440 load 4.299 -> 4.290 +19:35:40 step 124820 loss=3.3167 lr=5.73e-05 tok=16.396B dt=9499 wait=217 +19:37:00 step 124840 loss=3.3142 lr=5.73e-05 tok=16.398B dt=9579 wait=217 +19:38:04 step 124860 loss=3.3239 lr=5.73e-05 tok=16.401B dt=9643 wait=218 +19:39:07 step 124880 loss=3.3014 lr=5.73e-05 tok=16.404B dt=9706 wait=218 +19:40:10 step 124900 loss=3.1802 lr=5.73e-05 tok=16.406B dt=9769 wait=219 +19:41:14 step 124920 loss=3.3952 lr=5.72e-05 tok=16.409B dt=9833 wait=219 +19:42:18 step 124940 loss=3.3270 lr=5.72e-05 tok=16.412B dt=9897 wait=219 +19:43:02 pushed ckpt_w115.pt,log_w115.txt +19:43:47 waited 45s for global r=441 +19:43:54 global r=441 load 4.297 -> 4.284 +19:44:28 step 124960 loss=3.4603 lr=5.72e-05 tok=16.414B dt=10027 wait=220 +19:45:39 step 124980 loss=3.3038 lr=5.72e-05 tok=16.417B dt=10098 wait=220 +19:46:54 step 125000 loss=3.2973 lr=5.72e-05 tok=16.419B dt=10173 wait=220 +19:47:58 step 125020 loss=3.3980 lr=5.71e-05 tok=16.422B dt=10237 wait=221 +19:49:01 step 125040 loss=3.3304 lr=5.71e-05 tok=16.425B dt=10300 wait=221 +19:50:05 step 125060 loss=3.3758 lr=5.71e-05 tok=16.427B dt=10364 wait=221 +19:51:08 step 125080 loss=3.3642 lr=5.71e-05 tok=16.430B dt=10427 wait=222 +19:52:11 step 125100 loss=3.3529 lr=5.71e-05 tok=16.432B dt=10490 wait=222 diff --git a/log_w116.txt b/log_w116.txt new file mode 100644 index 0000000000000000000000000000000000000000..7c7df5ba456ab69777cc969d26a2aeb8f902c9cc --- /dev/null +++ b/log_w116.txt @@ -0,0 +1,195 @@ +launch +16:55:11 name=w116 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 63/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:55:17 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:55:17 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:55:26 params 1.08B +16:55:28 resumed step=122400 seen=16.08B round=425 +16:55:28 stream salt=122400 +16:57:18 blocks compiled +16:57:18 start: accum=4, 131072 tokens per optimiser step +17:00:47 step 122401 loss=3.2079 lr=5.97e-05 tok=16.079B dt=209 wait=173 +17:00:51 step 122402 loss=3.1863 lr=5.97e-05 tok=16.079B dt=213 wait=173 +17:01:55 step 122420 loss=3.4047 lr=5.97e-05 tok=16.081B dt=277 wait=174 +17:03:02 step 122440 loss=3.4332 lr=5.97e-05 tok=16.084B dt=344 wait=174 +17:04:06 step 122460 loss=3.3975 lr=5.97e-05 tok=16.086B dt=409 wait=175 +17:05:10 step 122480 loss=3.4040 lr=5.97e-05 tok=16.089B dt=473 wait=175 +17:06:15 step 122500 loss=3.4655 lr=5.96e-05 tok=16.092B dt=537 wait=175 +17:07:19 step 122520 loss=3.4005 lr=5.96e-05 tok=16.094B dt=601 wait=176 +17:08:23 step 122540 loss=3.3881 lr=5.96e-05 tok=16.097B dt=665 wait=176 +17:09:11 pushed ckpt_w116.pt,log_w116.txt +17:09:26 waited 15s for global r=426 +17:09:44 global r=426 load 4.285 -> 4.287 +17:10:19 step 122560 loss=3.4918 lr=5.96e-05 tok=16.100B dt=781 wait=176 +17:11:23 step 122580 loss=3.3246 lr=5.96e-05 tok=16.102B dt=845 wait=177 +17:12:27 step 122600 loss=3.3914 lr=5.95e-05 tok=16.105B dt=909 wait=177 +17:13:31 step 122620 loss=3.4540 lr=5.95e-05 tok=16.107B dt=974 wait=177 +17:14:36 step 122640 loss=3.3237 lr=5.95e-05 tok=16.110B dt=1038 wait=177 +17:15:40 step 122660 loss=3.4372 lr=5.95e-05 tok=16.113B dt=1102 wait=178 +17:16:44 step 122680 loss=3.3847 lr=5.95e-05 tok=16.115B dt=1166 wait=178 +17:17:48 step 122700 loss=3.3269 lr=5.94e-05 tok=16.118B dt=1230 wait=178 +17:18:01 pushed ckpt_w116.pt,log_w116.txt +17:19:16 waited 76s for global r=427 +17:19:24 global r=427 load 4.318 -> 4.284 +17:20:30 step 122720 loss=3.3549 lr=5.94e-05 tok=16.121B dt=1393 wait=179 +17:21:35 step 122740 loss=3.4282 lr=5.94e-05 tok=16.123B dt=1457 wait=179 +17:22:38 step 122760 loss=3.3737 lr=5.94e-05 tok=16.126B dt=1521 wait=179 +17:23:43 step 122780 loss=3.3808 lr=5.94e-05 tok=16.128B dt=1585 wait=180 +17:24:47 step 122800 loss=3.3711 lr=5.93e-05 tok=16.131B dt=1649 wait=180 +17:25:51 step 122820 loss=3.4346 lr=5.93e-05 tok=16.134B dt=1713 wait=180 +17:26:55 step 122840 loss=3.3060 lr=5.93e-05 tok=16.136B dt=1777 wait=181 +17:27:41 pushed ckpt_w116.pt,log_w116.txt +17:28:56 waited 76s for global r=428 +17:29:04 global r=428 load 4.297 -> 4.284 +17:29:37 step 122860 loss=3.4775 lr=5.93e-05 tok=16.139B dt=1940 wait=181 +17:30:41 step 122880 loss=3.3132 lr=5.93e-05 tok=16.142B dt=2003 wait=181 +17:31:45 step 122900 loss=3.3431 lr=5.92e-05 tok=16.144B dt=2067 wait=182 +17:32:49 step 122920 loss=3.4774 lr=5.92e-05 tok=16.147B dt=2131 wait=182 +17:33:53 step 122940 loss=3.4084 lr=5.92e-05 tok=16.149B dt=2196 wait=182 +17:34:58 step 122960 loss=3.4505 lr=5.92e-05 tok=16.152B dt=2260 wait=183 +17:36:02 step 122980 loss=3.3678 lr=5.92e-05 tok=16.155B dt=2324 wait=183 +17:37:06 step 123000 loss=3.3878 lr=5.91e-05 tok=16.157B dt=2388 wait=183 +17:37:18 pushed ckpt_w116.pt,log_w116.txt +17:38:33 waited 76s for global r=429 +17:38:39 global r=429 load 4.285 -> 4.277 +17:39:45 step 123020 loss=3.3652 lr=5.91e-05 tok=16.160B dt=2547 wait=184 +17:40:49 step 123040 loss=3.4029 lr=5.91e-05 tok=16.162B dt=2611 wait=184 +17:41:53 step 123060 loss=3.4080 lr=5.91e-05 tok=16.165B dt=2675 wait=184 +17:42:57 step 123080 loss=3.3232 lr=5.91e-05 tok=16.168B dt=2739 wait=184 +17:44:01 step 123100 loss=3.3754 lr=5.90e-05 tok=16.170B dt=2804 wait=185 +17:45:07 step 123120 loss=3.4564 lr=5.90e-05 tok=16.173B dt=2869 wait=185 +17:46:12 step 123140 loss=3.2717 lr=5.90e-05 tok=16.176B dt=2934 wait=185 +17:46:56 pushed ckpt_w116.pt,log_w116.txt +17:48:12 waited 76s for global r=430 +17:48:18 global r=430 load 4.293 -> 4.288 +17:48:51 step 123160 loss=3.4275 lr=5.90e-05 tok=16.178B dt=3094 wait=186 +17:49:56 step 123180 loss=3.3301 lr=5.90e-05 tok=16.181B dt=3158 wait=186 +17:51:00 step 123200 loss=3.3782 lr=5.89e-05 tok=16.183B dt=3222 wait=186 +17:52:04 step 123220 loss=3.4226 lr=5.89e-05 tok=16.186B dt=3286 wait=187 +17:53:10 step 123240 loss=3.2427 lr=5.89e-05 tok=16.189B dt=3352 wait=187 +17:54:15 step 123260 loss=3.4229 lr=5.89e-05 tok=16.191B dt=3417 wait=187 +17:55:19 step 123280 loss=3.3312 lr=5.89e-05 tok=16.194B dt=3481 wait=188 +17:56:23 step 123300 loss=3.4077 lr=5.88e-05 tok=16.197B dt=3545 wait=188 +17:56:35 pushed ckpt_w116.pt,log_w116.txt +17:57:36 waited 61s for global r=431 +17:57:41 global r=431 load 4.287 -> 4.295 +17:58:47 step 123320 loss=3.3556 lr=5.88e-05 tok=16.199B dt=3689 wait=188 +17:59:52 step 123340 loss=3.3796 lr=5.88e-05 tok=16.202B dt=3754 wait=189 +18:00:56 step 123360 loss=3.3938 lr=5.88e-05 tok=16.204B dt=3818 wait=189 +18:02:00 step 123380 loss=3.3370 lr=5.88e-05 tok=16.207B dt=3882 wait=189 +18:03:04 step 123400 loss=3.3403 lr=5.87e-05 tok=16.210B dt=3946 wait=190 +18:04:09 step 123420 loss=3.4602 lr=5.87e-05 tok=16.212B dt=4011 wait=190 +18:05:13 step 123440 loss=3.2946 lr=5.87e-05 tok=16.215B dt=4075 wait=190 +18:05:58 pushed ckpt_w116.pt,log_w116.txt +18:07:13 waited 76s for global r=432 +18:07:19 global r=432 load 4.306 -> 4.294 +18:07:52 step 123460 loss=3.4745 lr=5.87e-05 tok=16.218B dt=4234 wait=191 +18:08:56 step 123480 loss=3.3278 lr=5.87e-05 tok=16.220B dt=4298 wait=191 +18:10:00 step 123500 loss=3.3346 lr=5.86e-05 tok=16.223B dt=4362 wait=191 +18:11:04 step 123520 loss=3.4400 lr=5.86e-05 tok=16.225B dt=4426 wait=192 +18:12:09 step 123540 loss=3.3367 lr=5.86e-05 tok=16.228B dt=4491 wait=192 +18:13:12 step 123560 loss=3.3774 lr=5.86e-05 tok=16.231B dt=4554 wait=192 +18:14:16 step 123580 loss=3.3569 lr=5.86e-05 tok=16.233B dt=4618 wait=193 +18:15:20 step 123600 loss=3.3637 lr=5.85e-05 tok=16.236B dt=4682 wait=193 +18:15:34 pushed ckpt_w116.pt,log_w116.txt +18:16:35 waited 61s for global r=433 +18:16:41 global r=433 load 4.294 -> 4.281 +18:17:46 step 123620 loss=3.3431 lr=5.85e-05 tok=16.239B dt=4828 wait=193 +18:18:50 step 123640 loss=3.3619 lr=5.85e-05 tok=16.241B dt=4892 wait=194 +18:19:53 step 123660 loss=3.3321 lr=5.85e-05 tok=16.244B dt=4955 wait=194 +18:20:57 step 123680 loss=3.3024 lr=5.85e-05 tok=16.246B dt=5019 wait=194 +18:22:01 step 123700 loss=3.2602 lr=5.84e-05 tok=16.249B dt=5083 wait=195 +18:23:05 step 123720 loss=3.3456 lr=5.84e-05 tok=16.252B dt=5147 wait=195 +18:24:08 step 123740 loss=3.2689 lr=5.84e-05 tok=16.254B dt=5210 wait=195 +18:24:54 pushed ckpt_w116.pt,log_w116.txt +18:26:09 waited 76s for global r=434 +18:26:16 global r=434 load 4.295 -> 4.290 +18:26:49 step 123760 loss=3.3212 lr=5.84e-05 tok=16.257B dt=5371 wait=196 +18:27:53 step 123780 loss=3.2278 lr=5.84e-05 tok=16.259B dt=5435 wait=196 +18:28:56 step 123800 loss=3.2836 lr=5.83e-05 tok=16.262B dt=5498 wait=196 +18:30:00 step 123820 loss=3.3556 lr=5.83e-05 tok=16.265B dt=5562 wait=196 +18:31:03 step 123840 loss=3.3171 lr=5.83e-05 tok=16.267B dt=5626 wait=197 +18:32:07 step 123860 loss=3.4056 lr=5.83e-05 tok=16.270B dt=5689 wait=197 +18:33:10 step 123880 loss=3.3252 lr=5.83e-05 tok=16.273B dt=5753 wait=197 +18:34:14 step 123900 loss=3.3182 lr=5.82e-05 tok=16.275B dt=5816 wait=198 +18:34:29 pushed ckpt_w116.pt,log_w116.txt +18:38:36 step 123920 loss=3.3547 lr=5.82e-05 tok=16.278B dt=6078 wait=198 +18:39:39 step 123940 loss=3.2961 lr=5.82e-05 tok=16.280B dt=6142 wait=199 +18:40:43 step 123960 loss=3.2349 lr=5.82e-05 tok=16.283B dt=6205 wait=199 +18:41:46 step 123980 loss=3.2853 lr=5.82e-05 tok=16.286B dt=6268 wait=199 +18:42:49 step 124000 loss=3.1964 lr=5.81e-05 tok=16.288B dt=6332 wait=199 +18:43:53 step 124020 loss=3.4137 lr=5.81e-05 tok=16.291B dt=6395 wait=200 +18:44:57 step 124040 loss=3.2830 lr=5.81e-05 tok=16.294B dt=6459 wait=200 +18:45:41 pushed ckpt_w116.pt,log_w116.txt +18:46:56 waited 76s for global r=435 +18:47:03 global r=435 load 4.305 -> 4.286 +18:47:37 step 124060 loss=3.4058 lr=5.81e-05 tok=16.296B dt=6619 wait=200 +18:48:40 step 124080 loss=3.3503 lr=5.81e-05 tok=16.299B dt=6682 wait=201 +18:49:44 step 124100 loss=3.3226 lr=5.80e-05 tok=16.301B dt=6746 wait=201 +18:50:48 step 124120 loss=3.3799 lr=5.80e-05 tok=16.304B dt=6810 wait=202 +18:51:52 step 124140 loss=3.2460 lr=5.80e-05 tok=16.307B dt=6874 wait=202 +18:52:55 step 124160 loss=3.3991 lr=5.80e-05 tok=16.309B dt=6938 wait=202 +18:53:59 step 124180 loss=3.3115 lr=5.80e-05 tok=16.312B dt=7001 wait=203 +18:55:03 step 124200 loss=3.3068 lr=5.79e-05 tok=16.315B dt=7065 wait=203 +18:55:15 pushed ckpt_w116.pt,log_w116.txt +18:56:30 waited 76s for global r=436 +18:56:38 global r=436 load 4.290 -> 4.277 +18:57:43 step 124220 loss=3.2902 lr=5.79e-05 tok=16.317B dt=7225 wait=203 +18:58:46 step 124240 loss=3.3777 lr=5.79e-05 tok=16.320B dt=7289 wait=204 +18:59:50 step 124260 loss=3.3025 lr=5.79e-05 tok=16.322B dt=7352 wait=204 +19:00:54 step 124280 loss=3.3026 lr=5.79e-05 tok=16.325B dt=7416 wait=204 +19:01:57 step 124300 loss=3.3808 lr=5.78e-05 tok=16.328B dt=7479 wait=204 +19:03:01 step 124320 loss=3.4420 lr=5.78e-05 tok=16.330B dt=7543 wait=205 +19:04:05 step 124340 loss=3.2986 lr=5.78e-05 tok=16.333B dt=7607 wait=205 +19:04:53 pushed ckpt_w116.pt,log_w116.txt +19:05:54 waited 60s for global r=437 +19:06:01 global r=437 load 4.291 -> 4.282 +19:06:34 step 124360 loss=3.4181 lr=5.78e-05 tok=16.336B dt=7756 wait=205 +19:07:38 step 124380 loss=3.2848 lr=5.78e-05 tok=16.338B dt=7820 wait=206 +19:08:42 step 124400 loss=3.2934 lr=5.78e-05 tok=16.341B dt=7884 wait=206 +19:09:46 step 124420 loss=3.3600 lr=5.77e-05 tok=16.343B dt=7948 wait=206 +19:10:50 step 124440 loss=3.0572 lr=5.77e-05 tok=16.346B dt=8012 wait=207 +19:11:53 step 124460 loss=3.3600 lr=5.77e-05 tok=16.349B dt=8076 wait=207 +19:12:57 step 124480 loss=3.3179 lr=5.77e-05 tok=16.351B dt=8139 wait=207 +19:14:00 step 124500 loss=3.3354 lr=5.77e-05 tok=16.354B dt=8202 wait=208 +19:14:13 pushed ckpt_w116.pt,log_w116.txt +19:15:13 waited 60s for global r=438 +19:15:19 global r=438 load 4.277 -> 4.283 +19:16:25 step 124520 loss=3.1796 lr=5.76e-05 tok=16.356B dt=8347 wait=208 +19:17:29 step 124540 loss=3.3848 lr=5.76e-05 tok=16.359B dt=8411 wait=208 +19:18:33 step 124560 loss=3.2956 lr=5.76e-05 tok=16.362B dt=8475 wait=209 +19:19:37 step 124580 loss=3.2597 lr=5.76e-05 tok=16.364B dt=8539 wait=209 +19:20:41 step 124600 loss=3.2584 lr=5.76e-05 tok=16.367B dt=8603 wait=209 +19:21:44 step 124620 loss=3.3691 lr=5.75e-05 tok=16.370B dt=8667 wait=210 +19:22:48 step 124640 loss=3.2865 lr=5.75e-05 tok=16.372B dt=8730 wait=210 +19:23:33 pushed ckpt_w116.pt,log_w116.txt +19:24:49 waited 76s for global r=439 +19:24:55 global r=439 load 4.303 -> 4.292 +19:25:28 step 124660 loss=3.4247 lr=5.75e-05 tok=16.375B dt=8890 wait=210 +19:26:32 step 124680 loss=3.2755 lr=5.75e-05 tok=16.377B dt=8955 wait=211 +19:27:36 step 124700 loss=3.3546 lr=5.75e-05 tok=16.380B dt=9018 wait=211 +19:28:40 step 124720 loss=3.4156 lr=5.74e-05 tok=16.383B dt=9082 wait=211 +19:29:43 step 124740 loss=3.2731 lr=5.74e-05 tok=16.385B dt=9146 wait=212 +19:30:48 step 124760 loss=3.3828 lr=5.74e-05 tok=16.388B dt=9210 wait=212 +19:31:53 step 124780 loss=2.8064 lr=5.74e-05 tok=16.391B dt=9275 wait=212 +19:32:56 step 124800 loss=3.3904 lr=5.74e-05 tok=16.393B dt=9339 wait=213 +19:33:09 pushed ckpt_w116.pt,log_w116.txt +19:34:24 waited 76s for global r=440 +19:34:33 global r=440 load 4.314 -> 4.290 +19:35:38 step 124820 loss=3.3228 lr=5.73e-05 tok=16.396B dt=9500 wait=213 +19:36:41 step 124840 loss=3.3813 lr=5.73e-05 tok=16.398B dt=9563 wait=213 +19:37:45 step 124860 loss=3.3499 lr=5.73e-05 tok=16.401B dt=9627 wait=214 +19:38:48 step 124880 loss=3.2797 lr=5.73e-05 tok=16.404B dt=9690 wait=214 +19:39:52 step 124900 loss=3.2128 lr=5.73e-05 tok=16.406B dt=9754 wait=214 +19:41:00 step 124920 loss=3.3945 lr=5.72e-05 tok=16.409B dt=9822 wait=215 +19:42:03 step 124940 loss=3.3135 lr=5.72e-05 tok=16.412B dt=9886 wait=215 +19:42:47 pushed ckpt_w116.pt,log_w116.txt +19:43:48 waited 61s for global r=441 +19:43:53 global r=441 load 4.293 -> 4.284 +19:44:27 step 124960 loss=3.3018 lr=5.72e-05 tok=16.414B dt=10029 wait=215 +19:45:30 step 124980 loss=3.1880 lr=5.72e-05 tok=16.417B dt=10092 wait=216 +19:46:34 step 125000 loss=3.3112 lr=5.72e-05 tok=16.419B dt=10156 wait=216 +19:47:37 step 125020 loss=3.3382 lr=5.71e-05 tok=16.422B dt=10219 wait=216 +19:48:41 step 125040 loss=3.2075 lr=5.71e-05 tok=16.425B dt=10283 wait=217 +19:49:44 step 125060 loss=3.4017 lr=5.71e-05 tok=16.427B dt=10347 wait=217 +19:50:48 step 125080 loss=3.3084 lr=5.71e-05 tok=16.430B dt=10410 wait=217 +19:51:52 step 125100 loss=3.3054 lr=5.71e-05 tok=16.432B dt=10474 wait=218 diff --git a/log_w117.txt b/log_w117.txt new file mode 100644 index 0000000000000000000000000000000000000000..3f8b335a24d64c13f4ecdde35fa99d8192862cc1 --- /dev/null +++ b/log_w117.txt @@ -0,0 +1,393 @@ +launch +16:55:09 name=w117 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 64/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:55:16 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:55:16 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:55:25 params 1.08B +16:55:27 resumed step=122400 seen=16.08B round=425 +16:55:27 stream salt=122400 +16:57:18 blocks compiled +16:57:18 start: accum=4, 131072 tokens per optimiser step +17:00:25 step 122401 loss=3.2614 lr=5.97e-05 tok=16.079B dt=187 wait=145 +17:00:28 step 122402 loss=3.1959 lr=5.97e-05 tok=16.079B dt=190 wait=145 +17:01:30 step 122420 loss=3.3600 lr=5.97e-05 tok=16.081B dt=252 wait=146 +17:02:43 step 122440 loss=3.2910 lr=5.97e-05 tok=16.084B dt=325 wait=146 +17:03:46 step 122460 loss=3.3403 lr=5.97e-05 tok=16.086B dt=388 wait=147 +17:04:50 step 122480 loss=3.3367 lr=5.97e-05 tok=16.089B dt=452 wait=147 +17:05:54 step 122500 loss=3.4830 lr=5.96e-05 tok=16.092B dt=516 wait=148 +17:06:58 step 122520 loss=3.3635 lr=5.96e-05 tok=16.094B dt=580 wait=148 +17:08:01 step 122540 loss=3.3331 lr=5.96e-05 tok=16.097B dt=643 wait=148 +17:08:52 pushed ckpt_w117.pt,log_w117.txt +17:09:22 waited 30s for global r=426 +17:09:38 global r=426 load 4.264 -> 4.287 +17:10:12 step 122560 loss=3.4249 lr=5.96e-05 tok=16.100B dt=774 wait=148 +17:11:16 step 122580 loss=3.3017 lr=5.96e-05 tok=16.102B dt=838 wait=149 +17:12:20 step 122600 loss=3.2889 lr=5.95e-05 tok=16.105B dt=902 wait=149 +17:13:23 step 122620 loss=3.3947 lr=5.95e-05 tok=16.107B dt=965 wait=149 +17:14:27 step 122640 loss=3.3245 lr=5.95e-05 tok=16.110B dt=1029 wait=150 +17:15:31 step 122660 loss=3.3631 lr=5.95e-05 tok=16.113B dt=1093 wait=150 +17:16:34 step 122680 loss=3.3702 lr=5.95e-05 tok=16.115B dt=1156 wait=150 +17:17:38 step 122700 loss=3.3755 lr=5.94e-05 tok=16.118B dt=1220 wait=151 +17:17:50 pushed ckpt_w117.pt,log_w117.txt +17:19:21 waited 91s for global r=427 +17:19:30 global r=427 load 4.291 -> 4.284 +17:20:35 step 122720 loss=3.2985 lr=5.94e-05 tok=16.121B dt=1397 wait=151 +17:21:38 step 122740 loss=3.3298 lr=5.94e-05 tok=16.123B dt=1460 wait=151 +17:22:42 step 122760 loss=3.3371 lr=5.94e-05 tok=16.126B dt=1524 wait=152 +17:23:45 step 122780 loss=3.3346 lr=5.94e-05 tok=16.128B dt=1587 wait=152 +17:24:49 step 122800 loss=3.3586 lr=5.93e-05 tok=16.131B dt=1651 wait=152 +17:25:53 step 122820 loss=3.4310 lr=5.93e-05 tok=16.134B dt=1715 wait=153 +17:26:56 step 122840 loss=3.3183 lr=5.93e-05 tok=16.136B dt=1778 wait=153 +17:27:43 pushed ckpt_w117.pt,log_w117.txt +17:28:58 waited 76s for global r=428 +17:29:05 global r=428 load 4.301 -> 4.284 +17:29:38 step 122860 loss=3.4730 lr=5.93e-05 tok=16.139B dt=1940 wait=153 +17:30:42 step 122880 loss=3.3498 lr=5.93e-05 tok=16.142B dt=2003 wait=154 +17:31:45 step 122900 loss=3.3274 lr=5.92e-05 tok=16.144B dt=2067 wait=154 +17:32:49 step 122920 loss=3.4291 lr=5.92e-05 tok=16.147B dt=2131 wait=154 +17:33:53 step 122940 loss=3.2840 lr=5.92e-05 tok=16.149B dt=2194 wait=155 +17:34:56 step 122960 loss=3.4263 lr=5.92e-05 tok=16.152B dt=2258 wait=155 +17:36:01 step 122980 loss=3.2271 lr=5.92e-05 tok=16.155B dt=2323 wait=155 +17:37:04 step 123000 loss=3.3689 lr=5.91e-05 tok=16.157B dt=2386 wait=156 +17:37:16 pushed ckpt_w117.pt,log_w117.txt +17:38:32 waited 76s for global r=429 +17:38:39 global r=429 load 4.291 -> 4.277 +17:39:44 step 123020 loss=3.3735 lr=5.91e-05 tok=16.160B dt=2546 wait=156 +17:40:47 step 123040 loss=3.3693 lr=5.91e-05 tok=16.162B dt=2609 wait=156 +17:41:51 step 123060 loss=3.3706 lr=5.91e-05 tok=16.165B dt=2673 wait=157 +17:42:55 step 123080 loss=3.2668 lr=5.91e-05 tok=16.168B dt=2737 wait=157 +17:43:59 step 123100 loss=3.1769 lr=5.90e-05 tok=16.170B dt=2801 wait=157 +17:45:03 step 123120 loss=3.4134 lr=5.90e-05 tok=16.173B dt=2865 wait=158 +17:46:07 step 123140 loss=3.2790 lr=5.90e-05 tok=16.176B dt=2929 wait=158 +17:46:51 pushed ckpt_w117.pt,log_w117.txt +17:48:06 waited 76s for global r=430 +17:48:13 global r=430 load 4.285 -> 4.288 +17:48:46 step 123160 loss=3.4334 lr=5.90e-05 tok=16.178B dt=3088 wait=158 +17:49:49 step 123180 loss=3.2895 lr=5.90e-05 tok=16.181B dt=3151 wait=159 +17:50:53 step 123200 loss=3.2910 lr=5.89e-05 tok=16.183B dt=3215 wait=159 +17:51:57 step 123220 loss=3.4584 lr=5.89e-05 tok=16.186B dt=3279 wait=159 +17:53:01 step 123240 loss=3.3228 lr=5.89e-05 tok=16.189B dt=3343 wait=160 +17:54:04 step 123260 loss=3.4068 lr=5.89e-05 tok=16.191B dt=3406 wait=160 +17:55:08 step 123280 loss=3.3310 lr=5.89e-05 tok=16.194B dt=3470 wait=160 +17:56:12 step 123300 loss=3.3995 lr=5.88e-05 tok=16.197B dt=3533 wait=161 +17:56:25 pushed ckpt_w117.pt,log_w117.txt +17:57:41 waited 76s for global r=431 +17:57:47 global r=431 load 4.328 -> 4.295 +17:58:52 step 123320 loss=3.3086 lr=5.88e-05 tok=16.199B dt=3694 wait=161 +17:59:55 step 123340 loss=3.3960 lr=5.88e-05 tok=16.202B dt=3757 wait=161 +18:00:58 step 123360 loss=3.3600 lr=5.88e-05 tok=16.204B dt=3820 wait=162 +18:02:03 step 123380 loss=3.3261 lr=5.88e-05 tok=16.207B dt=3884 wait=162 +18:03:06 step 123400 loss=3.3822 lr=5.87e-05 tok=16.210B dt=3948 wait=162 +18:04:10 step 123420 loss=3.4477 lr=5.87e-05 tok=16.212B dt=4012 wait=163 +18:05:14 step 123440 loss=3.2884 lr=5.87e-05 tok=16.215B dt=4076 wait=163 +18:05:58 pushed ckpt_w117.pt,log_w117.txt +18:07:13 waited 76s for global r=432 +18:07:19 global r=432 load 4.300 -> 4.294 +18:07:52 step 123460 loss=3.3997 lr=5.87e-05 tok=16.218B dt=4234 wait=163 +18:08:56 step 123480 loss=3.2364 lr=5.87e-05 tok=16.220B dt=4298 wait=164 +18:09:59 step 123500 loss=3.3473 lr=5.86e-05 tok=16.223B dt=4361 wait=164 +18:11:03 step 123520 loss=3.4008 lr=5.86e-05 tok=16.225B dt=4425 wait=164 +18:12:12 step 123540 loss=3.3275 lr=5.86e-05 tok=16.228B dt=4493 wait=165 +18:13:16 step 123560 loss=3.4341 lr=5.86e-05 tok=16.231B dt=4558 wait=165 +18:14:19 step 123580 loss=3.3918 lr=5.86e-05 tok=16.233B dt=4621 wait=165 +18:15:23 step 123600 loss=3.2841 lr=5.85e-05 tok=16.236B dt=4685 wait=165 +18:15:35 pushed ckpt_w117.pt,log_w117.txt +18:16:35 waited 61s for global r=433 +18:16:41 global r=433 load 4.287 -> 4.281 +18:17:46 step 123620 loss=3.3097 lr=5.85e-05 tok=16.239B dt=4828 wait=166 +18:18:50 step 123640 loss=3.3974 lr=5.85e-05 tok=16.241B dt=4892 wait=166 +18:19:54 step 123660 loss=3.3192 lr=5.85e-05 tok=16.244B dt=4955 wait=166 +18:20:57 step 123680 loss=3.3400 lr=5.85e-05 tok=16.246B dt=5019 wait=167 +18:22:01 step 123700 loss=3.3489 lr=5.84e-05 tok=16.249B dt=5083 wait=167 +18:23:05 step 123720 loss=3.4396 lr=5.84e-05 tok=16.252B dt=5147 wait=167 +18:24:09 step 123740 loss=3.2961 lr=5.84e-05 tok=16.254B dt=5211 wait=168 +18:24:53 pushed ckpt_w117.pt,log_w117.txt +18:26:08 waited 76s for global r=434 +18:26:19 global r=434 load 4.301 -> 4.290 +18:26:52 step 123760 loss=3.4595 lr=5.84e-05 tok=16.257B dt=5374 wait=168 +18:27:55 step 123780 loss=3.3015 lr=5.84e-05 tok=16.259B dt=5437 wait=168 +18:28:59 step 123800 loss=3.2660 lr=5.83e-05 tok=16.262B dt=5501 wait=169 +18:30:02 step 123820 loss=3.4082 lr=5.83e-05 tok=16.265B dt=5564 wait=169 +18:31:06 step 123840 loss=3.3248 lr=5.83e-05 tok=16.267B dt=5628 wait=169 +18:32:10 step 123860 loss=3.3003 lr=5.83e-05 tok=16.270B dt=5692 wait=170 +18:33:14 step 123880 loss=3.2812 lr=5.83e-05 tok=16.273B dt=5756 wait=170 +18:34:17 step 123900 loss=3.4134 lr=5.82e-05 tok=16.275B dt=5819 wait=170 +18:34:30 pushed ckpt_w117.pt,log_w117.txt +18:38:38 step 123920 loss=3.4056 lr=5.82e-05 tok=16.278B dt=6079 wait=171 +18:39:41 step 123940 loss=3.3049 lr=5.82e-05 tok=16.280B dt=6143 wait=171 +18:40:44 step 123960 loss=3.2594 lr=5.82e-05 tok=16.283B dt=6206 wait=171 +18:41:48 step 123980 loss=3.2799 lr=5.82e-05 tok=16.286B dt=6269 wait=172 +18:42:51 step 124000 loss=3.1794 lr=5.81e-05 tok=16.288B dt=6333 wait=172 +18:43:55 step 124020 loss=3.4163 lr=5.81e-05 tok=16.291B dt=6397 wait=172 +18:44:58 step 124040 loss=3.3300 lr=5.81e-05 tok=16.294B dt=6460 wait=173 +18:45:44 pushed ckpt_w117.pt,log_w117.txt +18:46:44 waited 61s for global r=435 +18:46:55 global r=435 load 4.301 -> 4.286 +18:47:28 step 124060 loss=3.4350 lr=5.81e-05 tok=16.296B dt=6609 wait=173 +18:48:31 step 124080 loss=3.3330 lr=5.81e-05 tok=16.299B dt=6673 wait=173 +18:49:34 step 124100 loss=3.3132 lr=5.80e-05 tok=16.301B dt=6736 wait=174 +18:50:38 step 124120 loss=3.4534 lr=5.80e-05 tok=16.304B dt=6800 wait=174 +18:51:42 step 124140 loss=3.3329 lr=5.80e-05 tok=16.307B dt=6864 wait=174 +18:52:46 step 124160 loss=3.3935 lr=5.80e-05 tok=16.309B dt=6928 wait=174 +18:53:49 step 124180 loss=3.3956 lr=5.80e-05 tok=16.312B dt=6991 wait=175 +18:54:53 step 124200 loss=3.3711 lr=5.79e-05 tok=16.315B dt=7055 wait=175 +18:55:05 pushed ckpt_w117.pt,log_w117.txt +18:56:21 waited 76s for global r=436 +18:56:34 global r=436 load 4.285 -> 4.277 +18:57:38 step 124220 loss=3.3052 lr=5.79e-05 tok=16.317B dt=7220 wait=175 +18:58:42 step 124240 loss=3.3948 lr=5.79e-05 tok=16.320B dt=7284 wait=176 +18:59:46 step 124260 loss=3.3295 lr=5.79e-05 tok=16.322B dt=7348 wait=176 +19:00:50 step 124280 loss=3.3642 lr=5.79e-05 tok=16.325B dt=7412 wait=176 +19:01:53 step 124300 loss=3.3283 lr=5.78e-05 tok=16.328B dt=7475 wait=177 +19:02:57 step 124320 loss=3.3980 lr=5.78e-05 tok=16.330B dt=7539 wait=177 +19:04:01 step 124340 loss=3.2866 lr=5.78e-05 tok=16.333B dt=7603 wait=177 +19:04:48 pushed ckpt_w117.pt,log_w117.txt +19:05:49 waited 61s for global r=437 +19:05:55 global r=437 load 4.297 -> 4.282 +19:06:28 step 124360 loss=3.4157 lr=5.78e-05 tok=16.336B dt=7750 wait=178 +19:07:31 step 124380 loss=3.3421 lr=5.78e-05 tok=16.338B dt=7813 wait=178 +19:08:35 step 124400 loss=3.2943 lr=5.78e-05 tok=16.341B dt=7877 wait=178 +19:09:39 step 124420 loss=3.4202 lr=5.77e-05 tok=16.343B dt=7940 wait=179 +19:10:42 step 124440 loss=3.2626 lr=5.77e-05 tok=16.346B dt=8004 wait=179 +19:11:46 step 124460 loss=3.3628 lr=5.77e-05 tok=16.349B dt=8068 wait=179 +19:12:49 step 124480 loss=3.3020 lr=5.77e-05 tok=16.351B dt=8131 wait=180 +19:13:53 step 124500 loss=3.3645 lr=5.77e-05 tok=16.354B dt=8194 wait=180 +19:14:05 pushed ckpt_w117.pt,log_w117.txt +19:15:20 waited 76s for global r=438 +19:15:30 global r=438 load 4.295 -> 4.283 +19:16:34 step 124520 loss=3.3306 lr=5.76e-05 tok=16.356B dt=8356 wait=180 +19:17:38 step 124540 loss=3.3466 lr=5.76e-05 tok=16.359B dt=8420 wait=181 +19:18:42 step 124560 loss=3.3051 lr=5.76e-05 tok=16.362B dt=8484 wait=181 +19:19:46 step 124580 loss=3.3001 lr=5.76e-05 tok=16.364B dt=8548 wait=181 +19:20:49 step 124600 loss=3.3823 lr=5.76e-05 tok=16.367B dt=8611 wait=182 +19:21:53 step 124620 loss=3.3967 lr=5.75e-05 tok=16.370B dt=8675 wait=182 +19:22:57 step 124640 loss=3.3064 lr=5.75e-05 tok=16.372B dt=8739 wait=182 +19:23:40 pushed ckpt_w117.pt,log_w117.txt +19:24:56 waited 76s for global r=439 +19:25:06 global r=439 load 4.316 -> 4.292 +19:25:39 step 124660 loss=3.4220 lr=5.75e-05 tok=16.375B dt=8901 wait=182 +19:26:43 step 124680 loss=3.3037 lr=5.75e-05 tok=16.377B dt=8965 wait=183 +19:27:47 step 124700 loss=3.2663 lr=5.75e-05 tok=16.380B dt=9028 wait=183 +19:28:50 step 124720 loss=3.4248 lr=5.74e-05 tok=16.383B dt=9092 wait=183 +19:29:53 step 124740 loss=3.2729 lr=5.74e-05 tok=16.385B dt=9155 wait=184 +19:30:57 step 124760 loss=3.4329 lr=5.74e-05 tok=16.388B dt=9219 wait=184 +19:32:00 step 124780 loss=3.2751 lr=5.74e-05 tok=16.391B dt=9282 wait=184 +19:33:04 step 124800 loss=3.3340 lr=5.74e-05 tok=16.393B dt=9346 wait=185 +19:33:16 pushed ckpt_w117.pt,log_w117.txt +19:34:17 waited 61s for global r=440 +19:34:23 global r=440 load 4.325 -> 4.290 +19:35:28 step 124820 loss=3.2203 lr=5.73e-05 tok=16.396B dt=9490 wait=185 +19:36:32 step 124840 loss=3.3667 lr=5.73e-05 tok=16.398B dt=9554 wait=185 +19:37:36 step 124860 loss=3.3125 lr=5.73e-05 tok=16.401B dt=9618 wait=185 +19:38:40 step 124880 loss=3.2958 lr=5.73e-05 tok=16.404B dt=9682 wait=186 +19:39:44 step 124900 loss=3.3457 lr=5.73e-05 tok=16.406B dt=9746 wait=186 +19:40:47 step 124920 loss=3.4173 lr=5.72e-05 tok=16.409B dt=9809 wait=186 +19:41:51 step 124940 loss=3.2985 lr=5.72e-05 tok=16.412B dt=9873 wait=187 +19:42:37 pushed ckpt_w117.pt,log_w117.txt +19:43:53 waited 76s for global r=441 +19:43:59 global r=441 load 4.285 -> 4.284 +19:44:33 step 124960 loss=3.3933 lr=5.72e-05 tok=16.414B dt=10035 wait=187 +19:45:37 step 124980 loss=3.2579 lr=5.72e-05 tok=16.417B dt=10099 wait=187 +19:46:40 step 125000 loss=3.3540 lr=5.72e-05 tok=16.419B dt=10162 wait=188 +19:47:44 step 125020 loss=3.4248 lr=5.71e-05 tok=16.422B dt=10226 wait=188 +19:48:48 step 125040 loss=3.2983 lr=5.71e-05 tok=16.425B dt=10290 wait=188 +19:49:52 step 125060 loss=3.4514 lr=5.71e-05 tok=16.427B dt=10354 wait=189 +19:50:56 step 125080 loss=3.3626 lr=5.71e-05 tok=16.430B dt=10418 wait=189 +19:52:00 step 125100 loss=3.4056 lr=5.71e-05 tok=16.432B dt=10481 wait=189 +19:52:12 pushed ckpt_w117.pt,log_w117.txt +19:53:28 waited 76s for global r=442 +19:53:33 global r=442 load 4.272 -> 4.290 +19:54:38 step 125120 loss=3.3147 lr=5.70e-05 tok=16.435B dt=10640 wait=190 +19:55:42 step 125140 loss=3.4008 lr=5.70e-05 tok=16.438B dt=10704 wait=190 +19:56:46 step 125160 loss=3.4024 lr=5.70e-05 tok=16.440B dt=10768 wait=190 +19:57:49 step 125180 loss=3.3670 lr=5.70e-05 tok=16.443B dt=10831 wait=191 +19:58:53 step 125200 loss=3.2275 lr=5.70e-05 tok=16.446B dt=10895 wait=191 +19:59:57 step 125220 loss=3.4112 lr=5.69e-05 tok=16.448B dt=10959 wait=191 +20:01:01 step 125240 loss=3.2981 lr=5.69e-05 tok=16.451B dt=11023 wait=192 +20:01:46 pushed ckpt_w117.pt,log_w117.txt +20:02:46 waited 61s for global r=443 +20:02:53 global r=443 load 4.305 -> 4.292 +20:03:27 step 125260 loss=3.4397 lr=5.69e-05 tok=16.453B dt=11169 wait=192 +20:04:30 step 125280 loss=3.3116 lr=5.69e-05 tok=16.456B dt=11232 wait=192 +20:05:52 step 125300 loss=3.3072 lr=5.69e-05 tok=16.459B dt=11313 wait=193 +20:06:58 step 125320 loss=3.4430 lr=5.68e-05 tok=16.461B dt=11380 wait=193 +20:08:02 step 125340 loss=3.3620 lr=5.68e-05 tok=16.464B dt=11444 wait=193 +20:09:06 step 125360 loss=3.4518 lr=5.68e-05 tok=16.467B dt=11508 wait=194 +20:10:10 step 125380 loss=3.3864 lr=5.68e-05 tok=16.469B dt=11572 wait=194 +20:11:14 step 125400 loss=3.4090 lr=5.68e-05 tok=16.472B dt=11636 wait=194 +20:11:28 pushed ckpt_w117.pt,log_w117.txt +20:15:12 global r=444 load 4.293 -> 4.288 +20:16:16 step 125420 loss=3.2901 lr=5.67e-05 tok=16.476B dt=11938 wait=195 +20:17:40 step 125440 loss=3.3463 lr=5.67e-05 tok=16.478B dt=12021 wait=195 +20:18:44 step 125460 loss=3.3101 lr=5.67e-05 tok=16.481B dt=12086 wait=196 +20:19:48 step 125480 loss=3.3515 lr=5.67e-05 tok=16.484B dt=12150 wait=196 +20:20:51 step 125500 loss=3.3579 lr=5.67e-05 tok=16.486B dt=12213 wait=196 +20:21:55 step 125520 loss=3.4447 lr=5.67e-05 tok=16.489B dt=12277 wait=197 +20:22:59 step 125540 loss=3.3028 lr=5.66e-05 tok=16.491B dt=12340 wait=197 +20:23:42 pushed ckpt_w117.pt,log_w117.txt +20:24:28 waited 45s for global r=445 +20:24:34 global r=445 load 4.290 -> 4.281 +20:25:07 step 125560 loss=3.3674 lr=5.66e-05 tok=16.494B dt=12469 wait=197 +20:26:19 step 125580 loss=3.2866 lr=5.66e-05 tok=16.497B dt=12541 wait=198 +20:27:36 step 125600 loss=3.2933 lr=5.66e-05 tok=16.499B dt=12618 wait=198 +20:28:40 step 125620 loss=3.4265 lr=5.66e-05 tok=16.502B dt=12681 wait=198 +20:29:43 step 125640 loss=3.3168 lr=5.65e-05 tok=16.505B dt=12745 wait=199 +20:30:47 step 125660 loss=3.3884 lr=5.65e-05 tok=16.507B dt=12809 wait=199 +20:31:50 step 125680 loss=3.3085 lr=5.65e-05 tok=16.510B dt=12872 wait=199 +20:32:54 step 125700 loss=3.3478 lr=5.65e-05 tok=16.512B dt=12936 wait=200 +20:33:08 pushed ckpt_w117.pt,log_w117.txt +20:33:53 waited 45s for global r=446 +20:34:00 global r=446 load 4.284 -> 4.287 +20:35:05 step 125720 loss=3.3366 lr=5.65e-05 tok=16.515B dt=13067 wait=200 +20:36:16 step 125740 loss=3.3629 lr=5.64e-05 tok=16.518B dt=13138 wait=200 +20:37:32 step 125760 loss=3.3197 lr=5.64e-05 tok=16.520B dt=13213 wait=201 +20:38:36 step 125780 loss=3.3424 lr=5.64e-05 tok=16.523B dt=13277 wait=201 +20:39:39 step 125800 loss=3.3154 lr=5.64e-05 tok=16.526B dt=13341 wait=201 +20:40:43 step 125820 loss=3.3997 lr=5.64e-05 tok=16.528B dt=13405 wait=202 +20:41:47 step 125840 loss=3.3129 lr=5.63e-05 tok=16.531B dt=13469 wait=202 +20:42:31 pushed ckpt_w117.pt,log_w117.txt +20:43:17 waited 45s for global r=447 +20:43:23 global r=447 load 4.287 -> 4.273 +20:43:56 step 125860 loss=3.4054 lr=5.63e-05 tok=16.533B dt=13598 wait=202 +20:45:00 step 125880 loss=3.3478 lr=5.63e-05 tok=16.536B dt=13662 wait=203 +20:46:24 step 125900 loss=3.3383 lr=5.63e-05 tok=16.539B dt=13746 wait=203 +20:47:27 step 125920 loss=3.4199 lr=5.63e-05 tok=16.541B dt=13809 wait=204 +20:48:31 step 125940 loss=3.3287 lr=5.62e-05 tok=16.544B dt=13873 wait=204 +20:49:35 step 125960 loss=3.4213 lr=5.62e-05 tok=16.547B dt=13937 wait=204 +20:50:39 step 125980 loss=3.3880 lr=5.62e-05 tok=16.549B dt=14001 wait=205 +20:51:42 step 126000 loss=3.3596 lr=5.62e-05 tok=16.552B dt=14064 wait=205 +20:51:55 pushed ckpt_w117.pt,log_w117.txt +20:52:40 waited 45s for global r=448 +20:52:50 global r=448 load 4.288 -> 4.270 +20:53:55 step 126020 loss=3.2371 lr=5.62e-05 tok=16.554B dt=14197 wait=205 +20:55:06 step 126040 loss=3.3522 lr=5.61e-05 tok=16.557B dt=14268 wait=206 +20:56:22 step 126060 loss=3.3402 lr=5.61e-05 tok=16.560B dt=14343 wait=206 +20:57:25 step 126080 loss=3.2604 lr=5.61e-05 tok=16.562B dt=14407 wait=206 +20:58:29 step 126100 loss=3.3738 lr=5.61e-05 tok=16.565B dt=14471 wait=207 +20:59:34 step 126120 loss=3.4517 lr=5.61e-05 tok=16.568B dt=14535 wait=207 +21:00:37 step 126140 loss=3.3158 lr=5.60e-05 tok=16.570B dt=14599 wait=207 +21:01:22 pushed ckpt_w117.pt,log_w117.txt +21:02:07 waited 46s for global r=449 +21:02:17 global r=449 load 4.277 -> 4.272 +21:02:50 step 126160 loss=3.3457 lr=5.60e-05 tok=16.573B dt=14732 wait=208 +21:03:53 step 126180 loss=3.3145 lr=5.60e-05 tok=16.575B dt=14795 wait=208 +21:05:11 step 126200 loss=3.3490 lr=5.60e-05 tok=16.578B dt=14873 wait=208 +21:06:21 step 126220 loss=3.3580 lr=5.60e-05 tok=16.581B dt=14943 wait=209 +21:07:25 step 126240 loss=3.3272 lr=5.59e-05 tok=16.583B dt=15007 wait=209 +21:08:28 step 126260 loss=3.3783 lr=5.59e-05 tok=16.586B dt=15070 wait=209 +21:09:32 step 126280 loss=3.3685 lr=5.59e-05 tok=16.588B dt=15134 wait=210 +21:10:36 step 126300 loss=3.3935 lr=5.59e-05 tok=16.591B dt=15198 wait=210 +21:10:48 pushed ckpt_w117.pt,log_w117.txt +21:11:33 waited 45s for global r=450 +21:11:49 global r=450 load 4.272 -> 4.270 +21:13:13 step 126320 loss=3.3566 lr=5.59e-05 tok=16.594B dt=15355 wait=210 +21:14:17 step 126340 loss=3.3941 lr=5.58e-05 tok=16.596B dt=15419 wait=211 +21:15:21 step 126360 loss=3.3329 lr=5.58e-05 tok=16.599B dt=15483 wait=211 +21:16:25 step 126380 loss=3.2871 lr=5.58e-05 tok=16.602B dt=15546 wait=211 +21:17:28 step 126400 loss=3.2985 lr=5.58e-05 tok=16.604B dt=15610 wait=212 +21:18:32 step 126420 loss=3.4500 lr=5.58e-05 tok=16.607B dt=15674 wait=212 +21:19:35 step 126440 loss=3.3008 lr=5.58e-05 tok=16.609B dt=15737 wait=212 +21:20:19 pushed ckpt_w117.pt,log_w117.txt +21:21:04 waited 45s for global r=451 +21:21:10 global r=451 load 4.276 -> 4.271 +21:21:49 step 126460 loss=3.4404 lr=5.57e-05 tok=16.612B dt=15871 wait=213 +21:23:06 step 126480 loss=3.3223 lr=5.57e-05 tok=16.615B dt=15948 wait=213 +21:24:10 step 126500 loss=3.3373 lr=5.57e-05 tok=16.617B dt=16012 wait=213 +21:25:14 step 126520 loss=3.4139 lr=5.57e-05 tok=16.620B dt=16075 wait=214 +21:26:17 step 126540 loss=3.3031 lr=5.57e-05 tok=16.623B dt=16139 wait=214 +21:27:21 step 126560 loss=3.4460 lr=5.56e-05 tok=16.625B dt=16203 wait=214 +21:28:25 step 126580 loss=3.3585 lr=5.56e-05 tok=16.628B dt=16266 wait=214 +21:29:28 step 126600 loss=3.3688 lr=5.56e-05 tok=16.630B dt=16330 wait=215 +21:29:43 pushed ckpt_w117.pt,log_w117.txt +21:30:28 waited 45s for global r=452 +21:30:34 global r=452 load 4.265 -> 4.274 +21:31:47 step 126620 loss=3.3417 lr=5.56e-05 tok=16.633B dt=16469 wait=215 +21:33:03 step 126640 loss=3.3331 lr=5.56e-05 tok=16.636B dt=16545 wait=215 +21:34:06 step 126660 loss=3.2840 lr=5.55e-05 tok=16.638B dt=16608 wait=216 +21:35:10 step 126680 loss=3.2884 lr=5.55e-05 tok=16.641B dt=16672 wait=216 +21:36:14 step 126700 loss=3.3360 lr=5.55e-05 tok=16.644B dt=16735 wait=216 +21:37:17 step 126720 loss=3.3795 lr=5.55e-05 tok=16.646B dt=16799 wait=217 +21:38:21 step 126740 loss=3.2792 lr=5.55e-05 tok=16.649B dt=16863 wait=217 +21:40:21 pushed ckpt_w117.pt,log_w117.txt +21:41:37 waited 76s for global r=453 +21:41:44 global r=453 load 4.288 -> 4.274 +21:42:17 step 126760 loss=3.4217 lr=5.54e-05 tok=16.651B dt=17099 wait=217 +21:43:36 step 126780 loss=3.3073 lr=5.54e-05 tok=16.654B dt=17177 wait=218 +21:44:44 step 126800 loss=3.2800 lr=5.54e-05 tok=16.657B dt=17246 wait=218 +21:45:48 step 126820 loss=3.4103 lr=5.54e-05 tok=16.659B dt=17310 wait=219 +21:46:51 step 126840 loss=3.3057 lr=5.54e-05 tok=16.662B dt=17373 wait=219 +21:47:55 step 126860 loss=3.4337 lr=5.53e-05 tok=16.664B dt=17437 wait=219 +21:48:59 step 126880 loss=3.3428 lr=5.53e-05 tok=16.667B dt=17500 wait=219 +21:50:02 step 126900 loss=3.3309 lr=5.53e-05 tok=16.670B dt=17564 wait=220 +21:50:14 pushed ckpt_w117.pt,log_w117.txt +21:50:59 waited 45s for global r=454 +21:51:09 global r=454 load 4.264 -> 4.280 +21:52:18 step 126920 loss=3.3041 lr=5.53e-05 tok=16.672B dt=17700 wait=220 +21:53:37 step 126940 loss=3.3806 lr=5.53e-05 tok=16.675B dt=17779 wait=220 +21:54:41 step 126960 loss=3.3657 lr=5.52e-05 tok=16.678B dt=17843 wait=221 +21:55:48 step 126980 loss=2.3265 lr=5.52e-05 tok=16.680B dt=17910 wait=221 +21:56:51 step 127000 loss=3.2936 lr=5.52e-05 tok=16.683B dt=17973 wait=221 +21:57:55 step 127020 loss=3.4021 lr=5.52e-05 tok=16.685B dt=18037 wait=222 +21:58:58 step 127040 loss=3.2288 lr=5.52e-05 tok=16.688B dt=18100 wait=222 +21:59:44 pushed ckpt_w117.pt,log_w117.txt +22:00:29 waited 45s for global r=455 +22:00:36 global r=455 load 4.286 -> 4.284 +22:01:09 step 127060 loss=3.3830 lr=5.51e-05 tok=16.691B dt=18231 wait=222 +22:02:21 step 127080 loss=3.2344 lr=5.51e-05 tok=16.693B dt=18303 wait=223 +22:03:36 step 127100 loss=3.3246 lr=5.51e-05 tok=16.696B dt=18378 wait=223 +22:04:40 step 127120 loss=3.3909 lr=5.51e-05 tok=16.699B dt=18442 wait=223 +22:05:44 step 127140 loss=3.3006 lr=5.51e-05 tok=16.701B dt=18505 wait=224 +22:06:48 step 127160 loss=3.3703 lr=5.51e-05 tok=16.704B dt=18569 wait=224 +22:07:51 step 127180 loss=3.3540 lr=5.50e-05 tok=16.706B dt=18633 wait=224 +22:08:55 step 127200 loss=3.3758 lr=5.50e-05 tok=16.709B dt=18696 wait=225 +22:09:10 pushed ckpt_w117.pt,log_w117.txt +22:09:55 waited 45s for global r=456 +22:10:01 global r=456 load 4.300 -> 4.287 +22:11:06 step 127220 loss=3.3315 lr=5.50e-05 tok=16.712B dt=18828 wait=225 +22:12:26 step 127240 loss=3.4188 lr=5.50e-05 tok=16.714B dt=18908 wait=225 +22:13:33 step 127260 loss=3.3389 lr=5.50e-05 tok=16.717B dt=18975 wait=225 +22:14:37 step 127280 loss=3.3240 lr=5.49e-05 tok=16.720B dt=19039 wait=226 +22:15:40 step 127300 loss=3.3047 lr=5.49e-05 tok=16.722B dt=19102 wait=226 +22:16:44 step 127320 loss=3.3702 lr=5.49e-05 tok=16.725B dt=19166 wait=226 +22:17:47 step 127340 loss=3.2677 lr=5.49e-05 tok=16.727B dt=19229 wait=227 +22:18:33 pushed ckpt_w117.pt,log_w117.txt +22:19:03 waited 30s for global r=457 +22:19:08 global r=457 load 4.309 -> 4.291 +22:19:42 step 127360 loss=3.4305 lr=5.49e-05 tok=16.730B dt=19344 wait=227 +22:20:45 step 127380 loss=3.3606 lr=5.48e-05 tok=16.733B dt=19407 wait=227 +22:22:08 step 127400 loss=3.4041 lr=5.48e-05 tok=16.735B dt=19490 wait=228 +22:23:14 step 127420 loss=3.4449 lr=5.48e-05 tok=16.738B dt=19556 wait=228 +22:24:18 step 127440 loss=3.3723 lr=5.48e-05 tok=16.741B dt=19619 wait=228 +22:25:21 step 127460 loss=3.4587 lr=5.48e-05 tok=16.743B dt=19683 wait=229 +22:26:25 step 127480 loss=3.4082 lr=5.47e-05 tok=16.746B dt=19747 wait=229 +22:27:28 step 127500 loss=3.4045 lr=5.47e-05 tok=16.748B dt=19810 wait=229 +22:27:41 pushed ckpt_w117.pt,log_w117.txt +22:28:27 waited 46s for global r=458 +22:28:39 global r=458 load 4.326 -> 4.289 +22:29:44 step 127520 loss=3.3096 lr=5.47e-05 tok=16.751B dt=19946 wait=230 +22:31:01 step 127540 loss=3.4167 lr=5.47e-05 tok=16.754B dt=20023 wait=230 +22:32:17 step 127560 loss=3.2923 lr=5.47e-05 tok=16.756B dt=20099 wait=230 +22:33:20 step 127580 loss=3.2970 lr=5.46e-05 tok=16.759B dt=20162 wait=231 +22:34:24 step 127600 loss=3.3201 lr=5.46e-05 tok=16.761B dt=20226 wait=231 +22:35:27 step 127620 loss=3.3889 lr=5.46e-05 tok=16.764B dt=20289 wait=231 +22:36:31 step 127640 loss=3.3405 lr=5.46e-05 tok=16.767B dt=20352 wait=231 +22:37:14 pushed ckpt_w117.pt,log_w117.txt +22:38:00 waited 45s for global r=459 +22:38:06 global r=459 load 4.290 -> 4.289 +22:38:39 step 127660 loss=3.4167 lr=5.46e-05 tok=16.769B dt=20481 wait=232 +22:39:42 step 127680 loss=3.2758 lr=5.45e-05 tok=16.772B dt=20544 wait=232 +22:41:01 step 127700 loss=3.2914 lr=5.45e-05 tok=16.775B dt=20622 wait=232 +22:42:09 step 127720 loss=3.3751 lr=5.45e-05 tok=16.777B dt=20691 wait=233 +22:43:13 step 127740 loss=3.2889 lr=5.45e-05 tok=16.780B dt=20755 wait=233 +22:44:17 step 127760 loss=3.4269 lr=5.45e-05 tok=16.782B dt=20819 wait=234 +22:45:21 step 127780 loss=3.3247 lr=5.45e-05 tok=16.785B dt=20882 wait=234 +22:46:24 step 127800 loss=3.3757 lr=5.44e-05 tok=16.788B dt=20946 wait=234 +22:46:38 pushed ckpt_w117.pt,log_w117.txt +22:47:24 waited 45s for global r=460 +22:47:29 global r=460 load 4.323 -> 4.285 +22:48:55 step 127820 loss=3.3110 lr=5.44e-05 tok=16.790B dt=21096 wait=235 +22:49:58 step 127840 loss=3.3693 lr=5.44e-05 tok=16.793B dt=21160 wait=235 +22:51:02 step 127860 loss=3.3032 lr=5.44e-05 tok=16.796B dt=21224 wait=235 +22:52:06 step 127880 loss=3.5682 lr=5.44e-05 tok=16.798B dt=21288 wait=236 +22:53:09 step 127900 loss=3.3635 lr=5.43e-05 tok=16.801B dt=21351 wait=236 +22:54:13 step 127920 loss=3.4552 lr=5.43e-05 tok=16.803B dt=21415 wait=236 +22:55:17 step 127940 loss=3.1995 lr=5.43e-05 tok=16.806B dt=21479 wait=236 diff --git a/log_w118.txt b/log_w118.txt new file mode 100644 index 0000000000000000000000000000000000000000..c86c26ebc22c16b6fc1f4b9459da3a2b18eeb391 --- /dev/null +++ b/log_w118.txt @@ -0,0 +1,409 @@ +launch +20:05:24 name=w118 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 65/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:05:30 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:05:30 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +20:05:40 params 1.08B +20:05:41 resumed step=125250 seen=16.49B round=443 +20:05:41 stream salt=125250 +20:07:24 blocks compiled +20:07:24 start: accum=4, 131072 tokens per optimiser step +20:09:51 step 125251 loss=3.0956 lr=5.69e-05 tok=16.489B dt=147 wait=100 +20:09:54 step 125252 loss=3.1021 lr=5.69e-05 tok=16.489B dt=150 wait=100 +20:10:23 step 125260 loss=3.4778 lr=5.69e-05 tok=16.490B dt=179 wait=101 +20:11:32 step 125280 loss=3.3514 lr=5.69e-05 tok=16.493B dt=248 wait=101 +20:12:37 step 125300 loss=3.4422 lr=5.69e-05 tok=16.495B dt=313 wait=102 +20:13:41 step 125320 loss=3.3836 lr=5.68e-05 tok=16.498B dt=377 wait=102 +20:14:46 step 125340 loss=3.4517 lr=5.68e-05 tok=16.501B dt=442 wait=102 +20:14:58 global r=444 load 4.264 -> 4.288 +20:16:02 step 125420 loss=3.3976 lr=5.67e-05 tok=16.503B dt=518 wait=103 +20:17:07 step 125440 loss=3.4734 lr=5.67e-05 tok=16.506B dt=583 wait=103 +20:18:11 step 125460 loss=3.4441 lr=5.67e-05 tok=16.509B dt=647 wait=103 +20:19:16 step 125480 loss=3.3419 lr=5.67e-05 tok=16.511B dt=712 wait=104 +20:20:20 step 125500 loss=3.4183 lr=5.67e-05 tok=16.514B dt=776 wait=104 +20:21:25 step 125520 loss=3.4596 lr=5.67e-05 tok=16.516B dt=841 wait=104 +20:22:29 step 125540 loss=3.3144 lr=5.66e-05 tok=16.519B dt=905 wait=105 +20:23:15 pushed ckpt_w118.pt,log_w118.txt +20:24:31 waited 76s for global r=445 +20:24:37 global r=445 load 4.286 -> 4.281 +20:25:11 step 125560 loss=3.4372 lr=5.66e-05 tok=16.522B dt=1067 wait=105 +20:26:15 step 125580 loss=3.3341 lr=5.66e-05 tok=16.524B dt=1131 wait=105 +20:27:19 step 125600 loss=3.3603 lr=5.66e-05 tok=16.527B dt=1195 wait=106 +20:28:24 step 125620 loss=3.4306 lr=5.66e-05 tok=16.529B dt=1260 wait=106 +20:29:28 step 125640 loss=3.3527 lr=5.65e-05 tok=16.532B dt=1324 wait=106 +20:30:32 step 125660 loss=3.4110 lr=5.65e-05 tok=16.535B dt=1388 wait=107 +20:31:36 step 125680 loss=3.3829 lr=5.65e-05 tok=16.537B dt=1452 wait=107 +20:32:40 step 125700 loss=3.4087 lr=5.65e-05 tok=16.540B dt=1517 wait=107 +20:32:54 pushed ckpt_w118.pt,log_w118.txt +20:33:54 waited 60s for global r=446 +20:34:03 global r=446 load 4.266 -> 4.287 +20:35:09 step 125720 loss=3.1466 lr=5.65e-05 tok=16.543B dt=1665 wait=107 +20:36:13 step 125740 loss=3.3910 lr=5.64e-05 tok=16.545B dt=1729 wait=108 +20:37:18 step 125760 loss=3.3646 lr=5.64e-05 tok=16.548B dt=1794 wait=108 +20:38:22 step 125780 loss=3.3359 lr=5.64e-05 tok=16.550B dt=1858 wait=108 +20:39:26 step 125800 loss=3.3595 lr=5.64e-05 tok=16.553B dt=1922 wait=109 +20:40:31 step 125820 loss=3.4065 lr=5.64e-05 tok=16.556B dt=1987 wait=109 +20:41:35 step 125840 loss=3.3257 lr=5.63e-05 tok=16.558B dt=2051 wait=109 +20:42:20 pushed ckpt_w118.pt,log_w118.txt +20:43:21 waited 60s for global r=447 +20:43:27 global r=447 load 4.267 -> 4.273 +20:44:01 step 125860 loss=3.4707 lr=5.63e-05 tok=16.561B dt=2197 wait=110 +20:45:05 step 125880 loss=3.3148 lr=5.63e-05 tok=16.564B dt=2261 wait=110 +20:46:10 step 125900 loss=3.3227 lr=5.63e-05 tok=16.566B dt=2326 wait=110 +20:47:15 step 125920 loss=3.4523 lr=5.63e-05 tok=16.569B dt=2391 wait=111 +20:48:19 step 125940 loss=3.3596 lr=5.62e-05 tok=16.571B dt=2455 wait=111 +20:49:23 step 125960 loss=3.4518 lr=5.62e-05 tok=16.574B dt=2519 wait=111 +20:50:27 step 125980 loss=3.3739 lr=5.62e-05 tok=16.577B dt=2583 wait=112 +20:51:32 step 126000 loss=3.5162 lr=5.62e-05 tok=16.579B dt=2648 wait=112 +20:51:47 pushed ckpt_w118.pt,log_w118.txt +20:52:47 waited 60s for global r=448 +20:52:52 global r=448 load 4.267 -> 4.270 +20:53:58 step 126020 loss=3.3952 lr=5.62e-05 tok=16.582B dt=2794 wait=112 +20:55:02 step 126040 loss=3.4631 lr=5.61e-05 tok=16.585B dt=2858 wait=113 +20:56:07 step 126060 loss=3.3434 lr=5.61e-05 tok=16.587B dt=2923 wait=113 +20:57:11 step 126080 loss=3.3617 lr=5.61e-05 tok=16.590B dt=2987 wait=113 +20:58:15 step 126100 loss=3.3767 lr=5.61e-05 tok=16.592B dt=3052 wait=113 +20:59:20 step 126120 loss=3.4892 lr=5.61e-05 tok=16.595B dt=3116 wait=114 +21:00:24 step 126140 loss=3.3647 lr=5.60e-05 tok=16.598B dt=3180 wait=114 +21:01:10 pushed ckpt_w118.pt,log_w118.txt +21:02:11 waited 61s for global r=449 +21:02:15 global r=449 load 4.290 -> 4.272 +21:02:49 step 126160 loss=3.5335 lr=5.60e-05 tok=16.600B dt=3325 wait=114 +21:03:53 step 126180 loss=3.3620 lr=5.60e-05 tok=16.603B dt=3389 wait=115 +21:04:57 step 126200 loss=3.3362 lr=5.60e-05 tok=16.606B dt=3453 wait=115 +21:06:02 step 126220 loss=3.4517 lr=5.60e-05 tok=16.608B dt=3518 wait=115 +21:07:06 step 126240 loss=3.3455 lr=5.59e-05 tok=16.611B dt=3582 wait=116 +21:08:11 step 126260 loss=3.4970 lr=5.59e-05 tok=16.613B dt=3647 wait=116 +21:09:15 step 126280 loss=3.3842 lr=5.59e-05 tok=16.616B dt=3711 wait=116 +21:10:20 step 126300 loss=3.4517 lr=5.59e-05 tok=16.619B dt=3776 wait=117 +21:10:35 pushed ckpt_w118.pt,log_w118.txt +21:11:36 waited 60s for global r=450 +21:11:45 global r=450 load 4.282 -> 4.270 +21:12:51 step 126320 loss=3.4028 lr=5.59e-05 tok=16.621B dt=3927 wait=117 +21:13:55 step 126340 loss=3.4537 lr=5.58e-05 tok=16.624B dt=3991 wait=117 +21:15:00 step 126360 loss=3.4554 lr=5.58e-05 tok=16.626B dt=4056 wait=118 +21:16:04 step 126380 loss=3.3991 lr=5.58e-05 tok=16.629B dt=4120 wait=118 +21:17:09 step 126400 loss=3.3662 lr=5.58e-05 tok=16.632B dt=4185 wait=118 +21:18:13 step 126420 loss=3.4917 lr=5.58e-05 tok=16.634B dt=4249 wait=119 +21:19:17 step 126440 loss=3.3546 lr=5.58e-05 tok=16.637B dt=4313 wait=119 +21:20:02 pushed ckpt_w118.pt,log_w118.txt +21:21:02 waited 60s for global r=451 +21:21:07 global r=451 load 4.283 -> 4.271 +21:21:40 step 126460 loss=3.4589 lr=5.57e-05 tok=16.640B dt=4456 wait=119 +21:22:44 step 126480 loss=3.3101 lr=5.57e-05 tok=16.642B dt=4520 wait=120 +21:23:48 step 126500 loss=3.3454 lr=5.57e-05 tok=16.645B dt=4584 wait=120 +21:24:51 step 126520 loss=3.4085 lr=5.57e-05 tok=16.647B dt=4647 wait=120 +21:25:55 step 126540 loss=3.3146 lr=5.57e-05 tok=16.650B dt=4711 wait=121 +21:26:59 step 126560 loss=3.4221 lr=5.56e-05 tok=16.653B dt=4775 wait=121 +21:28:03 step 126580 loss=3.2345 lr=5.56e-05 tok=16.655B dt=4839 wait=121 +21:29:07 step 126600 loss=3.3904 lr=5.56e-05 tok=16.658B dt=4903 wait=121 +21:29:20 pushed ckpt_w118.pt,log_w118.txt +21:30:20 waited 61s for global r=452 +21:30:25 global r=452 load 4.295 -> 4.274 +21:31:30 step 126620 loss=3.3112 lr=5.56e-05 tok=16.661B dt=5046 wait=122 +21:32:34 step 126640 loss=3.3511 lr=5.56e-05 tok=16.663B dt=5110 wait=122 +21:33:37 step 126660 loss=3.3893 lr=5.55e-05 tok=16.666B dt=5174 wait=122 +21:34:41 step 126680 loss=3.3365 lr=5.55e-05 tok=16.668B dt=5237 wait=123 +21:35:45 step 126700 loss=3.2985 lr=5.55e-05 tok=16.671B dt=5301 wait=123 +21:36:49 step 126720 loss=3.3953 lr=5.55e-05 tok=16.674B dt=5365 wait=123 +21:37:52 step 126740 loss=3.2526 lr=5.55e-05 tok=16.676B dt=5428 wait=124 +21:40:21 push fail 1/3 (Request ID: Root=1-6ab83b5f-2d6503a9174467870c0b8ad7;17627530-902e-4e47-8d07-c38f585efa01) + +429 Too +21:40:36 pushed ckpt_w118.pt,log_w118.txt (attempt 2) +21:41:36 waited 61s for global r=453 +21:41:42 global r=453 load 4.283 -> 4.274 +21:42:15 step 126760 loss=3.4558 lr=5.54e-05 tok=16.679B dt=5692 wait=124 +21:43:19 step 126780 loss=3.2695 lr=5.54e-05 tok=16.682B dt=5755 wait=124 +21:44:22 step 126800 loss=3.3657 lr=5.54e-05 tok=16.684B dt=5819 wait=125 +21:45:27 step 126820 loss=3.4244 lr=5.54e-05 tok=16.687B dt=5883 wait=125 +21:46:30 step 126840 loss=3.3243 lr=5.54e-05 tok=16.689B dt=5946 wait=125 +21:47:34 step 126860 loss=3.4119 lr=5.53e-05 tok=16.692B dt=6010 wait=126 +21:48:38 step 126880 loss=3.3239 lr=5.53e-05 tok=16.695B dt=6074 wait=126 +21:49:42 step 126900 loss=3.3850 lr=5.53e-05 tok=16.697B dt=6138 wait=126 +21:49:56 pushed ckpt_w118.pt,log_w118.txt +21:50:57 waited 61s for global r=454 +21:51:01 global r=454 load 4.297 -> 4.280 +21:52:06 step 126920 loss=3.3572 lr=5.53e-05 tok=16.700B dt=6282 wait=126 +21:53:10 step 126940 loss=3.4123 lr=5.53e-05 tok=16.703B dt=6346 wait=127 +21:54:14 step 126960 loss=3.3808 lr=5.52e-05 tok=16.705B dt=6410 wait=127 +21:55:17 step 126980 loss=3.2818 lr=5.52e-05 tok=16.708B dt=6473 wait=127 +21:56:22 step 127000 loss=3.2766 lr=5.52e-05 tok=16.710B dt=6538 wait=128 +21:57:25 step 127020 loss=3.3694 lr=5.52e-05 tok=16.713B dt=6601 wait=128 +21:58:29 step 127040 loss=3.2604 lr=5.52e-05 tok=16.716B dt=6665 wait=128 +21:59:13 pushed ckpt_w118.pt,log_w118.txt +22:00:29 waited 76s for global r=455 +22:00:37 global r=455 load 4.282 -> 4.284 +22:01:10 step 127060 loss=3.4313 lr=5.51e-05 tok=16.718B dt=6826 wait=129 +22:02:14 step 127080 loss=3.3065 lr=5.51e-05 tok=16.721B dt=6890 wait=129 +22:03:17 step 127100 loss=3.2592 lr=5.51e-05 tok=16.723B dt=6953 wait=129 +22:04:21 step 127120 loss=3.4097 lr=5.51e-05 tok=16.726B dt=7017 wait=130 +22:05:25 step 127140 loss=3.3137 lr=5.51e-05 tok=16.729B dt=7081 wait=130 +22:06:29 step 127160 loss=3.4414 lr=5.51e-05 tok=16.731B dt=7145 wait=130 +22:07:33 step 127180 loss=3.3212 lr=5.50e-05 tok=16.734B dt=7209 wait=130 +22:08:36 step 127200 loss=3.4115 lr=5.50e-05 tok=16.737B dt=7272 wait=131 +22:08:48 pushed ckpt_w118.pt,log_w118.txt +22:09:49 waited 61s for global r=456 +22:09:56 global r=456 load 4.265 -> 4.287 +22:11:00 step 127220 loss=3.3328 lr=5.50e-05 tok=16.739B dt=7416 wait=131 +22:12:04 step 127240 loss=3.3936 lr=5.50e-05 tok=16.742B dt=7480 wait=131 +22:13:08 step 127260 loss=3.4265 lr=5.50e-05 tok=16.744B dt=7544 wait=132 +22:14:12 step 127280 loss=3.3596 lr=5.49e-05 tok=16.747B dt=7608 wait=132 +22:15:15 step 127300 loss=3.3457 lr=5.49e-05 tok=16.750B dt=7671 wait=132 +22:16:19 step 127320 loss=3.4320 lr=5.49e-05 tok=16.752B dt=7735 wait=133 +22:17:23 step 127340 loss=3.2329 lr=5.49e-05 tok=16.755B dt=7799 wait=133 +22:18:07 pushed ckpt_w118.pt,log_w118.txt +22:19:07 waited 60s for global r=457 +22:19:13 global r=457 load 4.315 -> 4.291 +22:19:46 step 127360 loss=3.4195 lr=5.49e-05 tok=16.758B dt=7942 wait=133 +22:20:50 step 127380 loss=3.3410 lr=5.48e-05 tok=16.760B dt=8006 wait=134 +22:21:53 step 127400 loss=3.3128 lr=5.48e-05 tok=16.763B dt=8069 wait=134 +22:22:57 step 127420 loss=3.3948 lr=5.48e-05 tok=16.765B dt=8133 wait=134 +22:24:01 step 127440 loss=3.3175 lr=5.48e-05 tok=16.768B dt=8197 wait=134 +22:25:05 step 127460 loss=3.4125 lr=5.48e-05 tok=16.771B dt=8261 wait=135 +22:26:09 step 127480 loss=3.3736 lr=5.47e-05 tok=16.773B dt=8325 wait=135 +22:27:13 step 127500 loss=3.3511 lr=5.47e-05 tok=16.776B dt=8389 wait=135 +22:27:24 pushed ckpt_w118.pt,log_w118.txt +22:28:40 waited 76s for global r=458 +22:28:46 global r=458 load 4.294 -> 4.289 +22:29:52 step 127520 loss=3.1275 lr=5.47e-05 tok=16.779B dt=8548 wait=136 +22:30:55 step 127540 loss=3.2963 lr=5.47e-05 tok=16.781B dt=8611 wait=136 +22:31:59 step 127560 loss=3.3749 lr=5.47e-05 tok=16.784B dt=8675 wait=136 +22:33:02 step 127580 loss=3.3011 lr=5.46e-05 tok=16.786B dt=8738 wait=137 +22:34:06 step 127600 loss=3.3530 lr=5.46e-05 tok=16.789B dt=8802 wait=137 +22:35:10 step 127620 loss=3.3624 lr=5.46e-05 tok=16.792B dt=8866 wait=137 +22:36:14 step 127640 loss=3.2968 lr=5.46e-05 tok=16.794B dt=8930 wait=137 +22:36:58 pushed ckpt_w118.pt,log_w118.txt +22:37:59 waited 60s for global r=459 +22:38:04 global r=459 load 4.314 -> 4.289 +22:38:38 step 127660 loss=3.3595 lr=5.46e-05 tok=16.797B dt=9074 wait=138 +22:39:41 step 127680 loss=3.2790 lr=5.45e-05 tok=16.799B dt=9137 wait=138 +22:40:45 step 127700 loss=3.2354 lr=5.45e-05 tok=16.802B dt=9201 wait=138 +22:41:49 step 127720 loss=3.3322 lr=5.45e-05 tok=16.805B dt=9265 wait=139 +22:42:53 step 127740 loss=3.2173 lr=5.45e-05 tok=16.807B dt=9329 wait=139 +22:43:57 step 127760 loss=3.3821 lr=5.45e-05 tok=16.810B dt=9393 wait=139 +22:45:00 step 127780 loss=3.3792 lr=5.45e-05 tok=16.813B dt=9456 wait=140 +22:46:04 step 127800 loss=3.3782 lr=5.44e-05 tok=16.815B dt=9520 wait=140 +22:46:16 pushed ckpt_w118.pt,log_w118.txt +22:47:17 waited 61s for global r=460 +22:47:23 global r=460 load 4.269 -> 4.285 +22:48:28 step 127820 loss=3.3763 lr=5.44e-05 tok=16.818B dt=9664 wait=140 +22:49:32 step 127840 loss=3.3857 lr=5.44e-05 tok=16.820B dt=9728 wait=141 +22:50:36 step 127860 loss=3.3609 lr=5.44e-05 tok=16.823B dt=9792 wait=141 +22:51:39 step 127880 loss=3.3224 lr=5.44e-05 tok=16.826B dt=9855 wait=141 +22:52:43 step 127900 loss=3.3126 lr=5.43e-05 tok=16.828B dt=9919 wait=142 +22:53:47 step 127920 loss=3.3856 lr=5.43e-05 tok=16.831B dt=9983 wait=142 +22:54:50 step 127940 loss=3.2730 lr=5.43e-05 tok=16.834B dt=10046 wait=142 +22:55:35 pushed ckpt_w118.pt,log_w118.txt +22:56:51 waited 76s for global r=461 +22:56:56 global r=461 load 4.284 -> 4.280 +22:57:28 step 127960 loss=3.4048 lr=5.43e-05 tok=16.836B dt=10204 wait=142 +22:58:32 step 127980 loss=3.3238 lr=5.43e-05 tok=16.839B dt=10268 wait=143 +22:59:35 step 128000 loss=3.3277 lr=5.42e-05 tok=16.841B dt=10332 wait=143 +23:00:39 step 128020 loss=3.3282 lr=5.42e-05 tok=16.844B dt=10395 wait=143 +23:01:43 step 128040 loss=3.3037 lr=5.42e-05 tok=16.847B dt=10459 wait=144 +23:02:47 step 128060 loss=3.4271 lr=5.42e-05 tok=16.849B dt=10523 wait=144 +23:03:50 step 128080 loss=3.3325 lr=5.42e-05 tok=16.852B dt=10587 wait=144 +23:04:55 step 128100 loss=3.3477 lr=5.41e-05 tok=16.855B dt=10651 wait=145 +23:05:07 pushed ckpt_w118.pt,log_w118.txt +23:09:18 step 128120 loss=3.4187 lr=5.41e-05 tok=16.857B dt=10914 wait=145 +23:10:21 step 128140 loss=3.3754 lr=5.41e-05 tok=16.860B dt=10977 wait=145 +23:11:26 step 128160 loss=3.2814 lr=5.41e-05 tok=16.862B dt=11042 wait=146 +23:12:30 step 128180 loss=3.2657 lr=5.41e-05 tok=16.865B dt=11106 wait=146 +23:13:33 step 128200 loss=3.2172 lr=5.40e-05 tok=16.868B dt=11169 wait=146 +23:14:37 step 128220 loss=3.4045 lr=5.40e-05 tok=16.870B dt=11233 wait=146 +23:15:40 step 128240 loss=3.3314 lr=5.40e-05 tok=16.873B dt=11297 wait=147 +23:16:28 pushed ckpt_w118.pt,log_w118.txt +23:20:03 step 128260 loss=3.4328 lr=5.40e-05 tok=16.876B dt=11559 wait=147 +23:21:06 step 128280 loss=3.4502 lr=5.40e-05 tok=16.878B dt=11623 wait=147 +23:22:10 step 128300 loss=3.3873 lr=5.40e-05 tok=16.881B dt=11686 wait=148 +23:23:14 step 128320 loss=3.3801 lr=5.39e-05 tok=16.883B dt=11750 wait=148 +23:23:20 global r=462 load 4.281 -> 4.283 +23:23:52 step 128260 loss=3.3731 lr=5.40e-05 tok=16.885B dt=11788 wait=148 +23:24:56 step 128280 loss=3.2753 lr=5.40e-05 tok=16.887B dt=11852 wait=149 +23:26:00 step 128300 loss=3.2644 lr=5.40e-05 tok=16.890B dt=11916 wait=149 +23:27:03 step 128320 loss=3.4089 lr=5.39e-05 tok=16.893B dt=11979 wait=149 +23:28:07 step 128340 loss=3.3197 lr=5.39e-05 tok=16.895B dt=12043 wait=150 +23:29:11 step 128360 loss=3.4152 lr=5.39e-05 tok=16.898B dt=12107 wait=150 +23:30:15 step 128380 loss=3.3328 lr=5.39e-05 tok=16.900B dt=12171 wait=150 +23:31:18 step 128400 loss=3.3742 lr=5.39e-05 tok=16.903B dt=12234 wait=150 +23:31:33 pushed ckpt_w118.pt,log_w118.txt +23:35:40 step 128420 loss=3.3791 lr=5.38e-05 tok=16.906B dt=12496 wait=151 +23:36:44 step 128440 loss=3.3476 lr=5.38e-05 tok=16.908B dt=12560 wait=151 +23:37:48 step 128460 loss=3.2277 lr=5.38e-05 tok=16.911B dt=12624 wait=151 +23:38:51 step 128480 loss=3.2301 lr=5.38e-05 tok=16.914B dt=12687 wait=152 +23:39:33 global r=463 load 4.308 -> 4.269 +23:40:37 step 128420 loss=3.3437 lr=5.38e-05 tok=16.917B dt=12793 wait=152 +23:41:42 step 128440 loss=3.2838 lr=5.38e-05 tok=16.920B dt=12858 wait=153 +23:42:46 step 128460 loss=3.3363 lr=5.38e-05 tok=16.923B dt=12922 wait=153 +23:43:49 step 128480 loss=3.2948 lr=5.38e-05 tok=16.925B dt=12985 wait=153 +23:44:53 step 128500 loss=3.3367 lr=5.38e-05 tok=16.928B dt=13049 wait=153 +23:45:57 step 128520 loss=3.4159 lr=5.37e-05 tok=16.931B dt=13113 wait=154 +23:47:00 step 128540 loss=3.3331 lr=5.37e-05 tok=16.933B dt=13176 wait=154 +23:48:00 pushed ckpt_w118.pt,log_w118.txt +23:48:46 waited 45s for global r=464 +23:48:57 global r=464 load 4.299 -> 4.271 +23:49:30 step 128560 loss=3.3909 lr=5.37e-05 tok=16.936B dt=13326 wait=154 +23:50:34 step 128580 loss=3.2773 lr=5.37e-05 tok=16.938B dt=13390 wait=155 +23:51:37 step 128600 loss=3.3006 lr=5.37e-05 tok=16.941B dt=13453 wait=155 +23:52:41 step 128620 loss=3.4695 lr=5.36e-05 tok=16.944B dt=13517 wait=155 +23:53:44 step 128640 loss=3.3272 lr=5.36e-05 tok=16.946B dt=13580 wait=156 +23:54:48 step 128660 loss=3.3700 lr=5.36e-05 tok=16.949B dt=13644 wait=156 +23:55:52 step 128680 loss=3.3413 lr=5.36e-05 tok=16.952B dt=13708 wait=156 +23:56:55 step 128700 loss=3.3596 lr=5.36e-05 tok=16.954B dt=13772 wait=157 +23:57:07 pushed ckpt_w118.pt,log_w118.txt +23:58:08 waited 61s for global r=465 +23:58:15 global r=465 load 4.287 -> 4.282 +23:59:21 step 128720 loss=3.3308 lr=5.35e-05 tok=16.957B dt=13917 wait=157 +00:00:25 step 128740 loss=3.3661 lr=5.35e-05 tok=16.959B dt=13981 wait=157 +00:01:28 step 128760 loss=3.3737 lr=5.35e-05 tok=16.962B dt=14044 wait=158 +00:02:33 step 128780 loss=3.2962 lr=5.35e-05 tok=16.965B dt=14109 wait=158 +00:03:36 step 128800 loss=3.3132 lr=5.35e-05 tok=16.967B dt=14173 wait=158 +00:04:40 step 128820 loss=3.4479 lr=5.35e-05 tok=16.970B dt=14236 wait=159 +00:05:44 step 128840 loss=3.3003 lr=5.34e-05 tok=16.973B dt=14300 wait=159 +00:06:29 pushed ckpt_w118.pt,log_w118.txt +00:07:31 waited 61s for global r=466 +00:07:38 global r=466 load 4.284 -> 4.287 +00:08:12 step 128860 loss=3.3995 lr=5.34e-05 tok=16.975B dt=14448 wait=159 +00:09:15 step 128880 loss=3.3075 lr=5.34e-05 tok=16.978B dt=14511 wait=160 +00:10:19 step 128900 loss=3.2729 lr=5.34e-05 tok=16.980B dt=14575 wait=160 +00:11:23 step 128920 loss=3.3791 lr=5.34e-05 tok=16.983B dt=14639 wait=160 +00:12:27 step 128940 loss=3.3002 lr=5.33e-05 tok=16.986B dt=14703 wait=161 +00:13:31 step 128960 loss=3.3803 lr=5.33e-05 tok=16.988B dt=14767 wait=161 +00:14:35 step 128980 loss=3.3572 lr=5.33e-05 tok=16.991B dt=14831 wait=161 +00:15:39 step 129000 loss=3.3337 lr=5.33e-05 tok=16.993B dt=14895 wait=162 +00:15:55 pushed ckpt_w118.pt,log_w118.txt +00:16:45 waited 50s for global r=467 +00:16:52 global r=467 load 4.281 -> 4.278 +00:17:59 step 129020 loss=3.2894 lr=5.33e-05 tok=16.996B dt=15035 wait=162 +00:19:02 step 129040 loss=3.3484 lr=5.32e-05 tok=16.999B dt=15098 wait=162 +00:20:06 step 129060 loss=3.3884 lr=5.32e-05 tok=17.001B dt=15162 wait=163 +00:21:10 step 129080 loss=3.2800 lr=5.32e-05 tok=17.004B dt=15226 wait=163 +00:22:14 step 129100 loss=3.2957 lr=5.32e-05 tok=17.007B dt=15290 wait=163 +00:23:18 step 129120 loss=3.4316 lr=5.32e-05 tok=17.009B dt=15354 wait=163 +00:24:21 step 129140 loss=3.2565 lr=5.31e-05 tok=17.012B dt=15417 wait=164 +00:25:10 pushed ckpt_w118.pt,log_w118.txt +00:26:10 waited 61s for global r=468 +00:26:21 global r=468 load 4.283 -> 4.290 +00:26:55 step 129160 loss=3.4057 lr=5.31e-05 tok=17.014B dt=15571 wait=164 +00:27:59 step 129180 loss=3.2254 lr=5.31e-05 tok=17.017B dt=15635 wait=164 +00:29:03 step 129200 loss=3.3346 lr=5.31e-05 tok=17.020B dt=15699 wait=165 +00:30:07 step 129220 loss=3.4372 lr=5.31e-05 tok=17.022B dt=15763 wait=165 +00:31:10 step 129240 loss=3.3237 lr=5.31e-05 tok=17.025B dt=15826 wait=165 +00:32:14 step 129260 loss=3.4088 lr=5.30e-05 tok=17.028B dt=15890 wait=166 +00:33:18 step 129280 loss=3.3409 lr=5.30e-05 tok=17.030B dt=15954 wait=166 +00:34:22 step 129300 loss=3.3892 lr=5.30e-05 tok=17.033B dt=16018 wait=166 +00:34:51 pushed ckpt_w118.pt,log_w118.txt +00:35:38 waited 47s for global r=469 +00:35:46 global r=469 load 4.298 -> 4.286 +00:36:54 step 129320 loss=3.3531 lr=5.30e-05 tok=17.035B dt=16170 wait=167 +00:37:57 step 129340 loss=3.3922 lr=5.30e-05 tok=17.038B dt=16233 wait=167 +00:39:01 step 129360 loss=3.3876 lr=5.29e-05 tok=17.041B dt=16297 wait=167 +00:40:05 step 129380 loss=3.3233 lr=5.29e-05 tok=17.043B dt=16361 wait=168 +00:41:09 step 129400 loss=3.3083 lr=5.29e-05 tok=17.046B dt=16425 wait=168 +00:42:13 step 129420 loss=3.4397 lr=5.29e-05 tok=17.049B dt=16489 wait=168 +00:43:17 step 129440 loss=3.3351 lr=5.29e-05 tok=17.051B dt=16553 wait=169 +00:44:05 pushed ckpt_w118.pt,log_w118.txt +00:45:06 waited 62s for global r=470 +00:45:16 global r=470 load 4.306 -> 4.288 +00:45:50 step 129460 loss=3.3879 lr=5.28e-05 tok=17.054B dt=16706 wait=169 +00:46:54 step 129480 loss=3.3064 lr=5.28e-05 tok=17.056B dt=16770 wait=169 +00:47:58 step 129500 loss=3.2851 lr=5.28e-05 tok=17.059B dt=16834 wait=170 +00:49:02 step 129520 loss=3.4205 lr=5.28e-05 tok=17.062B dt=16898 wait=170 +00:50:06 step 129540 loss=3.3408 lr=5.28e-05 tok=17.064B dt=16962 wait=170 +00:51:10 step 129560 loss=3.4038 lr=5.27e-05 tok=17.067B dt=17026 wait=171 +00:52:13 step 129580 loss=3.3284 lr=5.27e-05 tok=17.070B dt=17089 wait=171 +00:53:17 step 129600 loss=3.3928 lr=5.27e-05 tok=17.072B dt=17153 wait=171 +00:53:33 pushed ckpt_w118.pt,log_w118.txt +00:54:33 waited 61s for global r=471 +00:54:41 global r=471 load 4.289 -> 4.287 +00:55:47 step 129620 loss=3.3172 lr=5.27e-05 tok=17.075B dt=17303 wait=172 +00:56:50 step 129640 loss=3.4118 lr=5.27e-05 tok=17.077B dt=17366 wait=172 +00:57:55 step 129660 loss=3.3155 lr=5.27e-05 tok=17.080B dt=17431 wait=172 +00:58:58 step 129680 loss=3.2772 lr=5.26e-05 tok=17.083B dt=17494 wait=172 +01:00:02 step 129700 loss=3.2933 lr=5.26e-05 tok=17.085B dt=17558 wait=173 +01:01:05 step 129720 loss=3.3881 lr=5.26e-05 tok=17.088B dt=17621 wait=173 +01:02:09 step 129740 loss=3.3123 lr=5.26e-05 tok=17.090B dt=17685 wait=173 +01:02:55 pushed ckpt_w118.pt,log_w118.txt +01:03:56 waited 61s for global r=472 +01:04:07 global r=472 load 4.276 -> 4.282 +01:04:41 step 129760 loss=3.3757 lr=5.26e-05 tok=17.093B dt=17837 wait=174 +01:05:45 step 129780 loss=3.1887 lr=5.25e-05 tok=17.096B dt=17901 wait=174 +01:06:48 step 129800 loss=3.3228 lr=5.25e-05 tok=17.098B dt=17964 wait=174 +01:07:52 step 129820 loss=3.3989 lr=5.25e-05 tok=17.101B dt=18028 wait=175 +01:08:55 step 129840 loss=3.2955 lr=5.25e-05 tok=17.104B dt=18091 wait=175 +01:10:00 step 129860 loss=3.4190 lr=5.25e-05 tok=17.106B dt=18156 wait=175 +01:11:04 step 129880 loss=3.3115 lr=5.24e-05 tok=17.109B dt=18220 wait=176 +01:12:07 step 129900 loss=3.3877 lr=5.24e-05 tok=17.111B dt=18283 wait=176 +01:12:26 pushed ckpt_w118.pt,log_w118.txt +01:13:27 waited 62s for global r=473 +01:13:35 global r=473 load 4.294 -> 4.291 +01:14:40 step 129920 loss=3.4108 lr=5.24e-05 tok=17.114B dt=18437 wait=176 +01:15:44 step 129940 loss=3.4418 lr=5.24e-05 tok=17.117B dt=18500 wait=177 +01:16:47 step 129960 loss=3.3885 lr=5.24e-05 tok=17.119B dt=18563 wait=177 +01:17:51 step 129980 loss=3.3202 lr=5.23e-05 tok=17.122B dt=18627 wait=177 +01:18:55 step 130000 loss=3.3821 lr=5.23e-05 tok=17.125B dt=18691 wait=178 +01:19:58 step 130020 loss=3.4350 lr=5.23e-05 tok=17.127B dt=18754 wait=178 +01:21:02 step 130040 loss=3.2494 lr=5.23e-05 tok=17.130B dt=18818 wait=178 +01:21:54 pushed ckpt_w118.pt,log_w118.txt +01:22:54 waited 61s for global r=474 +01:23:02 global r=474 load 4.312 -> 4.288 +01:23:35 step 130060 loss=3.4327 lr=5.23e-05 tok=17.132B dt=18971 wait=178 +01:24:39 step 130080 loss=3.3561 lr=5.23e-05 tok=17.135B dt=19035 wait=179 +01:25:42 step 130100 loss=3.3373 lr=5.22e-05 tok=17.138B dt=19098 wait=179 +01:26:46 step 130120 loss=3.4110 lr=5.22e-05 tok=17.140B dt=19162 wait=179 +01:27:50 step 130140 loss=3.3135 lr=5.22e-05 tok=17.143B dt=19226 wait=180 +01:28:54 step 130160 loss=3.3945 lr=5.22e-05 tok=17.146B dt=19290 wait=180 +01:29:57 step 130180 loss=3.3194 lr=5.22e-05 tok=17.148B dt=19353 wait=180 +01:31:01 step 130200 loss=3.4131 lr=5.21e-05 tok=17.151B dt=19417 wait=181 +01:31:17 pushed ckpt_w118.pt,log_w118.txt +01:32:19 waited 62s for global r=475 +01:32:28 global r=475 load 4.260 -> 4.277 +01:33:33 step 130220 loss=3.3285 lr=5.21e-05 tok=17.153B dt=19570 wait=181 +01:34:37 step 130240 loss=3.3008 lr=5.21e-05 tok=17.156B dt=19633 wait=181 +01:35:41 step 130260 loss=3.3897 lr=5.21e-05 tok=17.159B dt=19697 wait=182 +01:36:44 step 130280 loss=3.3097 lr=5.21e-05 tok=17.161B dt=19760 wait=182 +01:37:48 step 130300 loss=3.3430 lr=5.20e-05 tok=17.164B dt=19824 wait=182 +01:38:52 step 130320 loss=3.4110 lr=5.20e-05 tok=17.166B dt=19888 wait=183 +01:39:56 step 130340 loss=3.2672 lr=5.20e-05 tok=17.169B dt=19952 wait=183 +01:40:46 pushed ckpt_w118.pt,log_w118.txt +01:41:49 waited 63s for global r=476 +01:42:02 global r=476 load 4.295 -> 4.277 +01:42:35 step 130360 loss=3.4086 lr=5.20e-05 tok=17.172B dt=20111 wait=183 +01:43:39 step 130380 loss=3.2824 lr=5.20e-05 tok=17.174B dt=20175 wait=184 +01:44:43 step 130400 loss=3.3235 lr=5.20e-05 tok=17.177B dt=20239 wait=184 +01:45:47 step 130420 loss=3.3628 lr=5.19e-05 tok=17.180B dt=20303 wait=184 +01:46:51 step 130440 loss=3.2671 lr=5.19e-05 tok=17.182B dt=20367 wait=185 +01:47:55 step 130460 loss=3.4007 lr=5.19e-05 tok=17.185B dt=20431 wait=185 +01:48:58 step 130480 loss=3.3252 lr=5.19e-05 tok=17.187B dt=20494 wait=185 +01:50:02 step 130500 loss=3.3827 lr=5.19e-05 tok=17.190B dt=20558 wait=186 +01:50:20 pushed ckpt_w118.pt,log_w118.txt +01:51:20 waited 61s for global r=477 +01:51:30 global r=477 load 4.290 -> 4.286 +01:52:36 step 130520 loss=3.3296 lr=5.18e-05 tok=17.193B dt=20712 wait=186 +01:53:40 step 130540 loss=3.3662 lr=5.18e-05 tok=17.195B dt=20776 wait=186 +01:54:44 step 130560 loss=3.3359 lr=5.18e-05 tok=17.198B dt=20840 wait=187 +01:55:47 step 130580 loss=3.3380 lr=5.18e-05 tok=17.201B dt=20903 wait=187 +01:56:51 step 130600 loss=3.3304 lr=5.18e-05 tok=17.203B dt=20967 wait=187 +01:57:55 step 130620 loss=3.4318 lr=5.17e-05 tok=17.206B dt=21031 wait=188 +01:58:59 step 130640 loss=3.3334 lr=5.17e-05 tok=17.208B dt=21095 wait=188 +01:59:48 pushed ckpt_w118.pt,log_w118.txt +02:00:34 waited 45s for global r=478 +02:00:47 global r=478 load 4.269 -> 4.275 +02:01:20 step 130660 loss=3.4376 lr=5.17e-05 tok=17.211B dt=21236 wait=188 +02:02:24 step 130680 loss=3.2576 lr=5.17e-05 tok=17.214B dt=21300 wait=189 +02:03:27 step 130700 loss=3.3211 lr=5.17e-05 tok=17.216B dt=21363 wait=189 +02:04:31 step 130720 loss=3.4639 lr=5.16e-05 tok=17.219B dt=21427 wait=189 +02:05:35 step 130740 loss=3.2822 lr=5.16e-05 tok=17.222B dt=21491 wait=190 +02:06:39 step 130760 loss=3.3370 lr=5.16e-05 tok=17.224B dt=21555 wait=190 +02:07:43 step 130780 loss=3.0651 lr=5.16e-05 tok=17.227B dt=21619 wait=190 +02:08:48 step 130800 loss=3.1745 lr=5.16e-05 tok=17.229B dt=21684 wait=190 +02:09:02 pushed ckpt_w118.pt,log_w118.txt +02:10:03 waited 61s for global r=479 +02:10:12 global r=479 load 4.278 -> 4.286 +02:11:18 step 130820 loss=3.0639 lr=5.16e-05 tok=17.232B dt=21834 wait=191 +02:12:22 step 130840 loss=3.1745 lr=5.15e-05 tok=17.235B dt=21898 wait=191 +02:13:26 step 130860 loss=3.2671 lr=5.15e-05 tok=17.237B dt=21962 wait=191 +02:14:29 step 130880 loss=3.3002 lr=5.15e-05 tok=17.240B dt=22026 wait=192 +02:15:33 step 130900 loss=3.3460 lr=5.15e-05 tok=17.243B dt=22089 wait=192 +02:16:37 step 130920 loss=3.3628 lr=5.15e-05 tok=17.245B dt=22153 wait=192 +02:17:41 step 130940 loss=3.1820 lr=5.14e-05 tok=17.248B dt=22217 wait=193 diff --git a/log_w119.txt b/log_w119.txt new file mode 100644 index 0000000000000000000000000000000000000000..f16bcf80331255122bc32b1816463a424ff87a2b --- /dev/null +++ b/log_w119.txt @@ -0,0 +1,204 @@ +launch +20:05:24 name=w119 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 66/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:05:31 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:05:31 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +20:05:42 params 1.08B +20:05:43 resumed step=125250 seen=16.49B round=443 +20:05:43 stream salt=125250 +20:07:24 blocks compiled +20:07:24 start: accum=4, 131072 tokens per optimiser step +20:09:58 step 125251 loss=3.0530 lr=5.69e-05 tok=16.489B dt=154 wait=107 +20:10:02 step 125252 loss=3.0438 lr=5.69e-05 tok=16.489B dt=157 wait=107 +20:10:31 step 125260 loss=3.3842 lr=5.69e-05 tok=16.490B dt=187 wait=108 +20:11:39 step 125280 loss=3.3607 lr=5.69e-05 tok=16.493B dt=254 wait=108 +20:12:42 step 125300 loss=3.3075 lr=5.69e-05 tok=16.495B dt=318 wait=109 +20:13:45 step 125320 loss=3.3013 lr=5.68e-05 tok=16.498B dt=381 wait=109 +20:14:49 step 125340 loss=3.3664 lr=5.68e-05 tok=16.501B dt=444 wait=109 +20:15:00 global r=444 load 4.316 -> 4.288 +20:16:03 step 125420 loss=3.3806 lr=5.67e-05 tok=16.503B dt=519 wait=109 +20:17:07 step 125440 loss=3.3896 lr=5.67e-05 tok=16.506B dt=582 wait=110 +20:18:10 step 125460 loss=3.3666 lr=5.67e-05 tok=16.509B dt=646 wait=110 +20:19:14 step 125480 loss=3.3367 lr=5.67e-05 tok=16.511B dt=709 wait=110 +20:20:17 step 125500 loss=3.3558 lr=5.67e-05 tok=16.514B dt=772 wait=111 +20:21:20 step 125520 loss=3.3848 lr=5.67e-05 tok=16.516B dt=836 wait=111 +20:22:25 step 125540 loss=3.2168 lr=5.66e-05 tok=16.519B dt=900 wait=111 +20:23:12 pushed ckpt_w119.pt,log_w119.txt +20:24:27 waited 76s for global r=445 +20:24:37 global r=445 load 4.317 -> 4.281 +20:25:11 step 125560 loss=3.4148 lr=5.66e-05 tok=16.522B dt=1066 wait=112 +20:26:14 step 125580 loss=3.2611 lr=5.66e-05 tok=16.524B dt=1130 wait=112 +20:27:18 step 125600 loss=3.2746 lr=5.66e-05 tok=16.527B dt=1193 wait=112 +20:28:21 step 125620 loss=3.3498 lr=5.66e-05 tok=16.529B dt=1256 wait=113 +20:29:24 step 125640 loss=3.2462 lr=5.65e-05 tok=16.532B dt=1320 wait=113 +20:30:27 step 125660 loss=3.4004 lr=5.65e-05 tok=16.535B dt=1383 wait=113 +20:31:31 step 125680 loss=3.3173 lr=5.65e-05 tok=16.537B dt=1447 wait=114 +20:32:34 step 125700 loss=3.3374 lr=5.65e-05 tok=16.540B dt=1510 wait=114 +20:32:46 pushed ckpt_w119.pt,log_w119.txt +20:33:47 waited 61s for global r=446 +20:33:55 global r=446 load 4.315 -> 4.287 +20:34:59 step 125720 loss=3.2908 lr=5.65e-05 tok=16.543B dt=1655 wait=114 +20:36:02 step 125740 loss=3.3253 lr=5.64e-05 tok=16.545B dt=1718 wait=115 +20:37:05 step 125760 loss=3.3414 lr=5.64e-05 tok=16.548B dt=1781 wait=115 +20:38:09 step 125780 loss=3.2255 lr=5.64e-05 tok=16.550B dt=1844 wait=115 +20:39:12 step 125800 loss=3.3383 lr=5.64e-05 tok=16.553B dt=1908 wait=116 +20:40:15 step 125820 loss=3.4237 lr=5.64e-05 tok=16.556B dt=1971 wait=116 +20:41:19 step 125840 loss=3.2734 lr=5.63e-05 tok=16.558B dt=2034 wait=116 +20:42:03 pushed ckpt_w119.pt,log_w119.txt +20:43:19 waited 76s for global r=447 +20:43:26 global r=447 load 4.323 -> 4.273 +20:43:59 step 125860 loss=3.3842 lr=5.63e-05 tok=16.561B dt=2195 wait=116 +20:45:02 step 125880 loss=3.2713 lr=5.63e-05 tok=16.564B dt=2258 wait=117 +20:46:06 step 125900 loss=3.2953 lr=5.63e-05 tok=16.566B dt=2321 wait=117 +20:47:10 step 125920 loss=3.3003 lr=5.63e-05 tok=16.569B dt=2385 wait=118 +20:48:13 step 125940 loss=3.2940 lr=5.62e-05 tok=16.571B dt=2448 wait=118 +20:49:16 step 125960 loss=3.2520 lr=5.62e-05 tok=16.574B dt=2512 wait=118 +20:50:20 step 125980 loss=3.3164 lr=5.62e-05 tok=16.577B dt=2575 wait=118 +20:51:23 step 126000 loss=3.4002 lr=5.62e-05 tok=16.579B dt=2639 wait=119 +20:51:35 pushed ckpt_w119.pt,log_w119.txt +20:52:36 waited 61s for global r=448 +20:52:43 global r=448 load 4.270 -> 4.270 +20:53:48 step 126020 loss=3.3188 lr=5.62e-05 tok=16.582B dt=2784 wait=119 +20:54:52 step 126040 loss=3.4323 lr=5.61e-05 tok=16.585B dt=2847 wait=119 +20:55:56 step 126060 loss=3.2526 lr=5.61e-05 tok=16.587B dt=2911 wait=120 +20:56:59 step 126080 loss=3.3656 lr=5.61e-05 tok=16.590B dt=2975 wait=120 +20:58:03 step 126100 loss=3.3535 lr=5.61e-05 tok=16.592B dt=3038 wait=120 +20:59:06 step 126120 loss=3.3889 lr=5.61e-05 tok=16.595B dt=3102 wait=121 +21:00:09 step 126140 loss=3.2124 lr=5.60e-05 tok=16.598B dt=3165 wait=121 +21:00:53 pushed ckpt_w119.pt,log_w119.txt +21:02:09 waited 76s for global r=449 +21:02:14 global r=449 load 4.293 -> 4.272 +21:02:48 step 126160 loss=3.4762 lr=5.60e-05 tok=16.600B dt=3323 wait=121 +21:03:51 step 126180 loss=3.3401 lr=5.60e-05 tok=16.603B dt=3386 wait=122 +21:04:54 step 126200 loss=3.3014 lr=5.60e-05 tok=16.606B dt=3450 wait=122 +21:05:57 step 126220 loss=3.4171 lr=5.60e-05 tok=16.608B dt=3513 wait=122 +21:07:01 step 126240 loss=3.2868 lr=5.59e-05 tok=16.611B dt=3577 wait=123 +21:08:05 step 126260 loss=3.4105 lr=5.59e-05 tok=16.613B dt=3640 wait=123 +21:09:08 step 126280 loss=3.3611 lr=5.59e-05 tok=16.616B dt=3704 wait=123 +21:10:12 step 126300 loss=3.3805 lr=5.59e-05 tok=16.619B dt=3767 wait=124 +21:10:25 pushed ckpt_w119.pt,log_w119.txt +21:11:41 waited 76s for global r=450 +21:11:48 global r=450 load 4.281 -> 4.270 +21:12:53 step 126320 loss=3.3194 lr=5.59e-05 tok=16.621B dt=3928 wait=124 +21:13:56 step 126340 loss=3.3463 lr=5.58e-05 tok=16.624B dt=3992 wait=125 +21:14:59 step 126360 loss=3.3272 lr=5.58e-05 tok=16.626B dt=4055 wait=125 +21:16:03 step 126380 loss=3.2751 lr=5.58e-05 tok=16.629B dt=4118 wait=125 +21:17:06 step 126400 loss=3.3168 lr=5.58e-05 tok=16.632B dt=4182 wait=126 +21:18:09 step 126420 loss=3.4276 lr=5.58e-05 tok=16.634B dt=4245 wait=126 +21:19:13 step 126440 loss=3.3093 lr=5.58e-05 tok=16.637B dt=4308 wait=126 +21:20:01 pushed ckpt_w119.pt,log_w119.txt +21:21:02 waited 60s for global r=451 +21:21:08 global r=451 load 4.291 -> 4.271 +21:21:41 step 126460 loss=3.4097 lr=5.57e-05 tok=16.640B dt=4456 wait=126 +21:22:44 step 126480 loss=3.2970 lr=5.57e-05 tok=16.642B dt=4520 wait=127 +21:23:47 step 126500 loss=3.2926 lr=5.57e-05 tok=16.645B dt=4583 wait=127 +21:24:51 step 126520 loss=3.3796 lr=5.57e-05 tok=16.647B dt=4646 wait=128 +21:25:54 step 126540 loss=3.2141 lr=5.57e-05 tok=16.650B dt=4710 wait=128 +21:26:58 step 126560 loss=3.3614 lr=5.56e-05 tok=16.653B dt=4773 wait=128 +21:28:01 step 126580 loss=3.2985 lr=5.56e-05 tok=16.655B dt=4837 wait=128 +21:29:05 step 126600 loss=3.3790 lr=5.56e-05 tok=16.658B dt=4900 wait=129 +21:29:19 pushed ckpt_w119.pt,log_w119.txt +21:30:19 waited 61s for global r=452 +21:30:28 global r=452 load 4.291 -> 4.274 +21:31:33 step 126620 loss=3.3110 lr=5.56e-05 tok=16.661B dt=5048 wait=129 +21:32:36 step 126640 loss=3.3590 lr=5.56e-05 tok=16.663B dt=5111 wait=130 +21:33:39 step 126660 loss=3.2931 lr=5.55e-05 tok=16.666B dt=5175 wait=130 +21:34:42 step 126680 loss=3.3124 lr=5.55e-05 tok=16.668B dt=5238 wait=130 +21:35:46 step 126700 loss=3.2907 lr=5.55e-05 tok=16.671B dt=5302 wait=131 +21:36:49 step 126720 loss=3.4202 lr=5.55e-05 tok=16.674B dt=5365 wait=131 +21:37:53 step 126740 loss=3.2485 lr=5.55e-05 tok=16.676B dt=5428 wait=131 +21:40:20 push fail 1/3 Server error '500 Internal Server Error' for url 'https://huggingface.co/api/models/Asilarkness/Diff +21:40:35 pushed ckpt_w119.pt,log_w119.txt (attempt 2) +21:41:36 waited 61s for global r=453 +21:41:41 global r=453 load 4.298 -> 4.274 +21:42:14 step 126760 loss=3.4420 lr=5.54e-05 tok=16.679B dt=5690 wait=132 +21:43:18 step 126780 loss=3.3090 lr=5.54e-05 tok=16.682B dt=5753 wait=132 +21:44:21 step 126800 loss=3.3358 lr=5.54e-05 tok=16.684B dt=5816 wait=132 +21:45:24 step 126820 loss=3.4134 lr=5.54e-05 tok=16.687B dt=5880 wait=132 +21:46:28 step 126840 loss=3.3085 lr=5.54e-05 tok=16.689B dt=5944 wait=133 +21:47:32 step 126860 loss=3.4099 lr=5.53e-05 tok=16.692B dt=6007 wait=133 +21:48:35 step 126880 loss=3.3451 lr=5.53e-05 tok=16.695B dt=6070 wait=133 +21:49:38 step 126900 loss=3.3403 lr=5.53e-05 tok=16.697B dt=6134 wait=134 +21:49:53 pushed ckpt_w119.pt,log_w119.txt +21:51:09 waited 76s for global r=454 +21:51:14 global r=454 load 4.297 -> 4.280 +21:52:19 step 126920 loss=3.3694 lr=5.53e-05 tok=16.700B dt=6295 wait=134 +21:53:23 step 126940 loss=3.4173 lr=5.53e-05 tok=16.703B dt=6358 wait=134 +21:54:27 step 126960 loss=3.2866 lr=5.52e-05 tok=16.705B dt=6422 wait=135 +21:55:31 step 126980 loss=3.3442 lr=5.52e-05 tok=16.708B dt=6486 wait=135 +21:56:34 step 127000 loss=3.2978 lr=5.52e-05 tok=16.710B dt=6550 wait=135 +21:57:37 step 127020 loss=3.4065 lr=5.52e-05 tok=16.713B dt=6613 wait=136 +21:58:40 step 127040 loss=3.2691 lr=5.52e-05 tok=16.716B dt=6676 wait=136 +21:59:25 pushed ckpt_w119.pt,log_w119.txt +22:00:26 waited 61s for global r=455 +22:00:32 global r=455 load 4.303 -> 4.284 +22:01:05 step 127060 loss=3.4199 lr=5.51e-05 tok=16.718B dt=6821 wait=136 +22:02:09 step 127080 loss=3.2773 lr=5.51e-05 tok=16.721B dt=6884 wait=136 +22:03:12 step 127100 loss=3.3419 lr=5.51e-05 tok=16.723B dt=6947 wait=137 +22:04:16 step 127120 loss=3.3314 lr=5.51e-05 tok=16.726B dt=7011 wait=137 +22:05:19 step 127140 loss=3.3091 lr=5.51e-05 tok=16.729B dt=7074 wait=137 +22:06:22 step 127160 loss=3.4173 lr=5.51e-05 tok=16.731B dt=7138 wait=138 +22:07:26 step 127180 loss=3.3736 lr=5.50e-05 tok=16.734B dt=7201 wait=138 +22:08:29 step 127200 loss=3.3550 lr=5.50e-05 tok=16.737B dt=7265 wait=138 +22:08:41 pushed ckpt_w119.pt,log_w119.txt +22:09:57 waited 76s for global r=456 +22:10:02 global r=456 load 4.319 -> 4.287 +22:11:08 step 127220 loss=3.3043 lr=5.50e-05 tok=16.739B dt=7423 wait=139 +22:12:11 step 127240 loss=3.3703 lr=5.50e-05 tok=16.742B dt=7487 wait=139 +22:13:15 step 127260 loss=3.3274 lr=5.50e-05 tok=16.744B dt=7551 wait=139 +22:14:18 step 127280 loss=3.2812 lr=5.49e-05 tok=16.747B dt=7614 wait=140 +22:15:22 step 127300 loss=3.2789 lr=5.49e-05 tok=16.750B dt=7677 wait=140 +22:16:25 step 127320 loss=3.3607 lr=5.49e-05 tok=16.752B dt=7741 wait=140 +22:17:29 step 127340 loss=3.2611 lr=5.49e-05 tok=16.755B dt=7804 wait=140 +22:18:16 pushed ckpt_w119.pt,log_w119.txt +22:19:16 waited 60s for global r=457 +22:19:24 global r=457 load 4.306 -> 4.291 +22:19:57 step 127360 loss=3.4093 lr=5.49e-05 tok=16.758B dt=7952 wait=141 +22:21:00 step 127380 loss=3.2681 lr=5.48e-05 tok=16.760B dt=8015 wait=141 +22:22:03 step 127400 loss=3.3135 lr=5.48e-05 tok=16.763B dt=8079 wait=141 +22:23:06 step 127420 loss=3.4102 lr=5.48e-05 tok=16.765B dt=8142 wait=142 +22:24:10 step 127440 loss=3.3184 lr=5.48e-05 tok=16.768B dt=8205 wait=142 +22:25:13 step 127460 loss=3.3587 lr=5.48e-05 tok=16.771B dt=8268 wait=142 +22:26:17 step 127480 loss=3.2627 lr=5.47e-05 tok=16.773B dt=8332 wait=143 +22:27:20 step 127500 loss=3.3890 lr=5.47e-05 tok=16.776B dt=8396 wait=143 +22:27:34 pushed ckpt_w119.pt,log_w119.txt +22:28:35 waited 61s for global r=458 +22:28:41 global r=458 load 4.305 -> 4.289 +22:29:46 step 127520 loss=3.2875 lr=5.47e-05 tok=16.779B dt=8541 wait=143 +22:30:49 step 127540 loss=3.3205 lr=5.47e-05 tok=16.781B dt=8605 wait=144 +22:31:53 step 127560 loss=3.2830 lr=5.47e-05 tok=16.784B dt=8669 wait=144 +22:32:57 step 127580 loss=3.2579 lr=5.46e-05 tok=16.786B dt=8733 wait=144 +22:34:01 step 127600 loss=3.2723 lr=5.46e-05 tok=16.789B dt=8796 wait=144 +22:35:04 step 127620 loss=3.4017 lr=5.46e-05 tok=16.792B dt=8859 wait=145 +22:36:07 step 127640 loss=3.2449 lr=5.46e-05 tok=16.794B dt=8923 wait=145 +22:36:54 pushed ckpt_w119.pt,log_w119.txt +22:37:55 waited 61s for global r=459 +22:38:03 global r=459 load 4.287 -> 4.289 +22:38:36 step 127660 loss=3.4260 lr=5.46e-05 tok=16.797B dt=9071 wait=145 +22:39:39 step 127680 loss=3.2896 lr=5.45e-05 tok=16.799B dt=9135 wait=146 +22:40:42 step 127700 loss=3.2894 lr=5.45e-05 tok=16.802B dt=9198 wait=146 +22:41:46 step 127720 loss=3.3934 lr=5.45e-05 tok=16.805B dt=9261 wait=146 +22:42:49 step 127740 loss=3.3081 lr=5.45e-05 tok=16.807B dt=9325 wait=147 +22:43:53 step 127760 loss=3.4153 lr=5.45e-05 tok=16.810B dt=9389 wait=147 +22:44:56 step 127780 loss=3.3408 lr=5.45e-05 tok=16.813B dt=9452 wait=147 +22:46:00 step 127800 loss=3.3947 lr=5.44e-05 tok=16.815B dt=9515 wait=148 +22:46:12 pushed ckpt_w119.pt,log_w119.txt +22:47:28 waited 76s for global r=460 +22:47:35 global r=460 load 4.287 -> 4.285 +22:48:39 step 127820 loss=3.3125 lr=5.44e-05 tok=16.818B dt=9675 wait=148 +22:49:42 step 127840 loss=3.2785 lr=5.44e-05 tok=16.820B dt=9738 wait=148 +22:50:46 step 127860 loss=3.3671 lr=5.44e-05 tok=16.823B dt=9802 wait=149 +22:51:50 step 127880 loss=3.2893 lr=5.44e-05 tok=16.826B dt=9865 wait=149 +22:52:53 step 127900 loss=3.3454 lr=5.43e-05 tok=16.828B dt=9929 wait=149 +22:53:57 step 127920 loss=3.4207 lr=5.43e-05 tok=16.831B dt=9992 wait=150 +22:55:00 step 127940 loss=3.2443 lr=5.43e-05 tok=16.834B dt=10056 wait=150 +22:55:46 pushed ckpt_w119.pt,log_w119.txt +22:56:46 waited 60s for global r=461 +22:56:52 global r=461 load 4.288 -> 4.280 +22:57:25 step 127960 loss=3.3783 lr=5.43e-05 tok=16.836B dt=10201 wait=150 +22:58:29 step 127980 loss=3.2801 lr=5.43e-05 tok=16.839B dt=10265 wait=151 +22:59:32 step 128000 loss=3.2817 lr=5.42e-05 tok=16.841B dt=10328 wait=151 +23:00:36 step 128020 loss=3.4219 lr=5.42e-05 tok=16.844B dt=10391 wait=151 +23:01:39 step 128040 loss=3.2637 lr=5.42e-05 tok=16.847B dt=10455 wait=152 +23:02:43 step 128060 loss=3.3583 lr=5.42e-05 tok=16.849B dt=10518 wait=152 +23:03:46 step 128080 loss=3.2924 lr=5.42e-05 tok=16.852B dt=10581 wait=152 +23:04:49 step 128100 loss=3.3226 lr=5.41e-05 tok=16.855B dt=10645 wait=153 diff --git a/log_w120.txt b/log_w120.txt new file mode 100644 index 0000000000000000000000000000000000000000..ec5e8972a136fdc71d65a6418213ed2838c7abca --- /dev/null +++ b/log_w120.txt @@ -0,0 +1,29 @@ +launch +23:24:25 name=w120 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 67/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +23:24:31 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +23:24:31 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +23:24:40 params 1.08B +23:24:42 resumed step=128250 seen=16.87B round=462 +23:24:42 stream salt=128250 +23:26:41 blocks compiled +23:26:41 start: accum=4, 131072 tokens per optimiser step +23:30:12 step 128251 loss=3.3904 lr=5.40e-05 tok=16.874B dt=211 wait=175 +23:30:15 step 128252 loss=3.2667 lr=5.40e-05 tok=16.874B dt=215 wait=175 +23:30:42 step 128260 loss=3.4453 lr=5.40e-05 tok=16.876B dt=242 wait=175 +23:31:50 step 128280 loss=3.3841 lr=5.40e-05 tok=16.878B dt=309 wait=176 +23:32:55 step 128300 loss=3.4527 lr=5.40e-05 tok=16.881B dt=374 wait=176 +23:33:59 step 128320 loss=3.3476 lr=5.39e-05 tok=16.883B dt=439 wait=176 +23:35:04 step 128340 loss=3.4099 lr=5.39e-05 tok=16.886B dt=503 wait=177 +23:36:08 step 128360 loss=3.5586 lr=5.39e-05 tok=16.889B dt=567 wait=177 +23:37:12 step 128380 loss=3.3882 lr=5.39e-05 tok=16.891B dt=632 wait=177 +23:38:16 step 128400 loss=3.4503 lr=5.39e-05 tok=16.894B dt=695 wait=178 +23:38:32 pushed ckpt_w120.pt,log_w120.txt +23:39:02 waited 30s for global r=463 +23:39:21 global r=463 load 4.295 -> 4.269 +23:40:27 step 128420 loss=3.3310 lr=5.38e-05 tok=16.896B dt=827 wait=178 +23:41:31 step 128440 loss=3.4238 lr=5.38e-05 tok=16.899B dt=891 wait=178 +23:42:35 step 128460 loss=3.3843 lr=5.38e-05 tok=16.902B dt=955 wait=179 +23:43:39 step 128480 loss=3.3554 lr=5.38e-05 tok=16.904B dt=1019 wait=179 +23:44:43 step 128500 loss=3.3938 lr=5.38e-05 tok=16.907B dt=1083 wait=179 +23:45:47 step 128520 loss=3.4552 lr=5.37e-05 tok=16.910B dt=1147 wait=180 +23:46:51 step 128540 loss=3.4152 lr=5.37e-05 tok=16.912B dt=1211 wait=180 diff --git a/log_w121.txt b/log_w121.txt new file mode 100644 index 0000000000000000000000000000000000000000..0c28bdf037e5999f3ffc47f5d6718b288d2f266b --- /dev/null +++ b/log_w121.txt @@ -0,0 +1,216 @@ +launch +23:24:27 name=w121 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 68/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +23:24:33 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +23:24:33 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +23:24:42 params 1.08B +23:24:44 resumed step=128250 seen=16.87B round=462 +23:24:44 stream salt=128250 +23:26:44 blocks compiled +23:26:44 start: accum=4, 131072 tokens per optimiser step +23:30:36 step 128251 loss=3.2572 lr=5.40e-05 tok=16.874B dt=232 wait=199 +23:30:39 step 128252 loss=3.0413 lr=5.40e-05 tok=16.874B dt=236 wait=199 +23:31:07 step 128260 loss=3.4195 lr=5.40e-05 tok=16.876B dt=263 wait=199 +23:32:14 step 128280 loss=3.3018 lr=5.40e-05 tok=16.878B dt=330 wait=200 +23:33:18 step 128300 loss=3.3922 lr=5.40e-05 tok=16.881B dt=395 wait=200 +23:34:22 step 128320 loss=3.3423 lr=5.39e-05 tok=16.883B dt=459 wait=200 +23:35:26 step 128340 loss=3.4099 lr=5.39e-05 tok=16.886B dt=523 wait=201 +23:36:30 step 128360 loss=3.5601 lr=5.39e-05 tok=16.889B dt=587 wait=201 +23:37:34 step 128380 loss=3.3001 lr=5.39e-05 tok=16.891B dt=651 wait=201 +23:38:38 step 128400 loss=3.4183 lr=5.39e-05 tok=16.894B dt=715 wait=202 +23:38:53 pushed ckpt_w121.pt,log_w121.txt +23:39:08 waited 15s for global r=463 +23:39:24 global r=463 load 4.270 -> 4.269 +23:40:30 step 128420 loss=3.3594 lr=5.38e-05 tok=16.896B dt=827 wait=202 +23:41:52 step 128440 loss=3.3662 lr=5.38e-05 tok=16.899B dt=908 wait=202 +23:42:58 step 128460 loss=3.3624 lr=5.38e-05 tok=16.902B dt=975 wait=203 +23:44:02 step 128480 loss=3.2613 lr=5.38e-05 tok=16.904B dt=1039 wait=203 +23:45:06 step 128500 loss=3.2396 lr=5.38e-05 tok=16.907B dt=1103 wait=203 +23:46:10 step 128520 loss=3.4236 lr=5.37e-05 tok=16.910B dt=1166 wait=204 +23:47:14 step 128540 loss=3.2946 lr=5.37e-05 tok=16.912B dt=1230 wait=204 +23:47:58 pushed ckpt_w121.pt,log_w121.txt +23:48:43 waited 45s for global r=464 +23:48:52 global r=464 load 4.286 -> 4.271 +23:49:26 step 128560 loss=3.3982 lr=5.37e-05 tok=16.915B dt=1362 wait=204 +23:50:36 step 128580 loss=3.2745 lr=5.37e-05 tok=16.917B dt=1432 wait=205 +23:51:53 step 128600 loss=3.2527 lr=5.37e-05 tok=16.920B dt=1510 wait=205 +23:52:57 step 128620 loss=3.3796 lr=5.36e-05 tok=16.923B dt=1574 wait=205 +23:54:02 step 128640 loss=3.2312 lr=5.36e-05 tok=16.925B dt=1638 wait=206 +23:55:05 step 128660 loss=3.3587 lr=5.36e-05 tok=16.928B dt=1702 wait=206 +23:56:10 step 128680 loss=3.3016 lr=5.36e-05 tok=16.931B dt=1766 wait=206 +23:57:14 step 128700 loss=3.4193 lr=5.36e-05 tok=16.933B dt=1830 wait=207 +23:57:31 pushed ckpt_w121.pt,log_w121.txt +23:58:01 waited 30s for global r=465 +23:58:07 global r=465 load 4.304 -> 4.282 +23:59:11 step 128720 loss=3.3158 lr=5.35e-05 tok=16.936B dt=1948 wait=207 +00:00:21 step 128740 loss=3.3219 lr=5.35e-05 tok=16.938B dt=2018 wait=207 +00:01:38 step 128760 loss=3.3513 lr=5.35e-05 tok=16.941B dt=2095 wait=208 +00:02:42 step 128780 loss=3.3562 lr=5.35e-05 tok=16.944B dt=2159 wait=208 +00:03:46 step 128800 loss=3.2662 lr=5.35e-05 tok=16.946B dt=2223 wait=208 +00:04:50 step 128820 loss=3.4089 lr=5.35e-05 tok=16.949B dt=2287 wait=209 +00:05:54 step 128840 loss=3.3186 lr=5.34e-05 tok=16.952B dt=2351 wait=209 +00:06:40 pushed ckpt_w121.pt,log_w121.txt +00:07:25 waited 45s for global r=466 +00:07:33 global r=466 load 4.281 -> 4.287 +00:08:06 step 128860 loss=3.4083 lr=5.34e-05 tok=16.954B dt=2483 wait=210 +00:09:10 step 128880 loss=3.3046 lr=5.34e-05 tok=16.957B dt=2546 wait=210 +00:10:26 step 128900 loss=3.2979 lr=5.34e-05 tok=16.959B dt=2622 wait=210 +00:11:38 step 128920 loss=3.4552 lr=5.34e-05 tok=16.962B dt=2694 wait=211 +00:12:42 step 128940 loss=3.3483 lr=5.33e-05 tok=16.965B dt=2758 wait=211 +00:13:46 step 128960 loss=3.3738 lr=5.33e-05 tok=16.967B dt=2822 wait=211 +00:14:49 step 128980 loss=3.3766 lr=5.33e-05 tok=16.970B dt=2886 wait=212 +00:15:53 step 129000 loss=3.4218 lr=5.33e-05 tok=16.973B dt=2950 wait=212 +00:16:05 pushed ckpt_w121.pt,log_w121.txt +00:16:51 waited 45s for global r=467 +00:16:59 global r=467 load 4.293 -> 4.278 +00:18:24 step 129020 loss=3.3458 lr=5.33e-05 tok=16.975B dt=3100 wait=212 +00:19:29 step 129040 loss=3.3529 lr=5.32e-05 tok=16.978B dt=3166 wait=213 +00:20:33 step 129060 loss=3.3645 lr=5.32e-05 tok=16.980B dt=3229 wait=213 +00:21:37 step 129080 loss=3.2725 lr=5.32e-05 tok=16.983B dt=3293 wait=213 +00:22:41 step 129100 loss=3.3114 lr=5.32e-05 tok=16.986B dt=3357 wait=214 +00:23:45 step 129120 loss=3.4304 lr=5.32e-05 tok=16.988B dt=3421 wait=214 +00:24:48 step 129140 loss=3.2095 lr=5.31e-05 tok=16.991B dt=3485 wait=214 +00:25:33 pushed ckpt_w121.pt,log_w121.txt +00:26:18 waited 45s for global r=468 +00:26:24 global r=468 load 4.299 -> 4.290 +00:27:03 step 129160 loss=3.3673 lr=5.31e-05 tok=16.993B dt=3619 wait=215 +00:28:20 step 129180 loss=3.2693 lr=5.31e-05 tok=16.996B dt=3697 wait=215 +00:29:24 step 129200 loss=3.3306 lr=5.31e-05 tok=16.999B dt=3761 wait=215 +00:30:28 step 129220 loss=3.4594 lr=5.31e-05 tok=17.001B dt=3825 wait=216 +00:31:32 step 129240 loss=3.2891 lr=5.31e-05 tok=17.004B dt=3889 wait=216 +00:32:36 step 129260 loss=3.3774 lr=5.30e-05 tok=17.007B dt=3953 wait=216 +00:33:40 step 129280 loss=3.2927 lr=5.30e-05 tok=17.009B dt=4016 wait=217 +00:34:44 step 129300 loss=3.3711 lr=5.30e-05 tok=17.012B dt=4081 wait=217 +00:34:58 pushed ckpt_w121.pt,log_w121.txt +00:35:43 waited 45s for global r=469 +00:35:50 global r=469 load 4.271 -> 4.286 +00:37:03 step 129320 loss=3.3460 lr=5.30e-05 tok=17.014B dt=4219 wait=217 +00:38:18 step 129340 loss=3.3735 lr=5.30e-05 tok=17.017B dt=4294 wait=218 +00:39:22 step 129360 loss=3.3717 lr=5.29e-05 tok=17.020B dt=4358 wait=218 +00:40:26 step 129380 loss=3.3438 lr=5.29e-05 tok=17.022B dt=4422 wait=218 +00:41:30 step 129400 loss=3.3030 lr=5.29e-05 tok=17.025B dt=4486 wait=219 +00:42:34 step 129420 loss=3.3087 lr=5.29e-05 tok=17.028B dt=4550 wait=219 +00:43:37 step 129440 loss=3.2631 lr=5.29e-05 tok=17.030B dt=4614 wait=219 +00:44:23 pushed ckpt_w121.pt,log_w121.txt +00:45:08 waited 45s for global r=470 +00:45:15 global r=470 load 4.284 -> 4.288 +00:45:48 step 129460 loss=3.4126 lr=5.28e-05 tok=17.033B dt=4744 wait=220 +00:47:11 step 129480 loss=3.2590 lr=5.28e-05 tok=17.035B dt=4827 wait=220 +00:48:15 step 129500 loss=3.2738 lr=5.28e-05 tok=17.038B dt=4891 wait=221 +00:49:18 step 129520 loss=3.4026 lr=5.28e-05 tok=17.041B dt=4955 wait=221 +00:50:22 step 129540 loss=3.3081 lr=5.28e-05 tok=17.043B dt=5018 wait=221 +00:51:26 step 129560 loss=3.3769 lr=5.27e-05 tok=17.046B dt=5083 wait=222 +00:52:30 step 129580 loss=3.3253 lr=5.27e-05 tok=17.049B dt=5146 wait=222 +00:53:34 step 129600 loss=3.3856 lr=5.27e-05 tok=17.051B dt=5210 wait=222 +00:53:50 pushed ckpt_w121.pt,log_w121.txt +00:54:35 waited 45s for global r=471 +00:54:41 global r=471 load 4.299 -> 4.287 +00:55:53 step 129620 loss=3.3256 lr=5.27e-05 tok=17.054B dt=5349 wait=223 +00:57:09 step 129640 loss=3.3010 lr=5.27e-05 tok=17.056B dt=5426 wait=223 +00:58:13 step 129660 loss=3.3636 lr=5.27e-05 tok=17.059B dt=5489 wait=223 +00:59:17 step 129680 loss=3.2873 lr=5.26e-05 tok=17.062B dt=5553 wait=224 +01:00:21 step 129700 loss=3.3424 lr=5.26e-05 tok=17.064B dt=5617 wait=224 +01:01:25 step 129720 loss=3.4031 lr=5.26e-05 tok=17.067B dt=5681 wait=224 +01:02:29 step 129740 loss=3.3005 lr=5.26e-05 tok=17.070B dt=5745 wait=225 +01:03:14 pushed ckpt_w121.pt,log_w121.txt +01:03:59 waited 45s for global r=472 +01:04:09 global r=472 load 4.289 -> 4.282 +01:04:42 step 129760 loss=3.4315 lr=5.26e-05 tok=17.072B dt=5878 wait=225 +01:05:59 step 129780 loss=3.3055 lr=5.25e-05 tok=17.075B dt=5956 wait=225 +01:07:09 step 129800 loss=3.2845 lr=5.25e-05 tok=17.077B dt=6026 wait=226 +01:08:13 step 129820 loss=3.3986 lr=5.25e-05 tok=17.080B dt=6089 wait=226 +01:09:17 step 129840 loss=3.2758 lr=5.25e-05 tok=17.083B dt=6153 wait=226 +01:10:20 step 129860 loss=3.3375 lr=5.25e-05 tok=17.085B dt=6217 wait=227 +01:11:25 step 129880 loss=3.2983 lr=5.24e-05 tok=17.088B dt=6281 wait=227 +01:12:29 step 129900 loss=3.3373 lr=5.24e-05 tok=17.090B dt=6345 wait=228 +01:12:41 pushed ckpt_w121.pt,log_w121.txt +01:13:26 waited 45s for global r=473 +01:13:32 global r=473 load 4.306 -> 4.291 +01:14:37 step 129920 loss=3.3159 lr=5.24e-05 tok=17.093B dt=6474 wait=228 +01:16:01 step 129940 loss=3.3722 lr=5.24e-05 tok=17.096B dt=6558 wait=228 +01:17:05 step 129960 loss=3.3765 lr=5.24e-05 tok=17.098B dt=6622 wait=229 +01:18:10 step 129980 loss=3.3234 lr=5.23e-05 tok=17.101B dt=6686 wait=229 +01:19:13 step 130000 loss=3.3197 lr=5.23e-05 tok=17.104B dt=6750 wait=229 +01:20:17 step 130020 loss=3.4049 lr=5.23e-05 tok=17.106B dt=6814 wait=230 +01:21:21 step 130040 loss=3.2760 lr=5.23e-05 tok=17.109B dt=6877 wait=230 +01:22:07 pushed ckpt_w121.pt,log_w121.txt +01:22:52 waited 45s for global r=474 +01:23:00 global r=474 load 4.290 -> 4.288 +01:23:33 step 130060 loss=3.3986 lr=5.23e-05 tok=17.111B dt=7010 wait=230 +01:24:45 step 130080 loss=3.3271 lr=5.23e-05 tok=17.114B dt=7081 wait=231 +01:26:00 step 130100 loss=3.3290 lr=5.22e-05 tok=17.117B dt=7157 wait=231 +01:27:04 step 130120 loss=3.4120 lr=5.22e-05 tok=17.119B dt=7221 wait=232 +01:28:08 step 130140 loss=3.2529 lr=5.22e-05 tok=17.122B dt=7285 wait=232 +01:29:12 step 130160 loss=3.3719 lr=5.22e-05 tok=17.125B dt=7349 wait=232 +01:30:17 step 130180 loss=3.3318 lr=5.22e-05 tok=17.127B dt=7413 wait=233 +01:31:21 step 130200 loss=3.3038 lr=5.21e-05 tok=17.130B dt=7477 wait=233 +01:31:34 pushed ckpt_w121.pt,log_w121.txt +01:32:20 waited 45s for global r=475 +01:32:28 global r=475 load 4.294 -> 4.277 +01:33:33 step 130220 loss=3.1799 lr=5.21e-05 tok=17.132B dt=7610 wait=233 +01:34:55 step 130240 loss=3.2507 lr=5.21e-05 tok=17.135B dt=7692 wait=234 +01:36:01 step 130260 loss=3.3429 lr=5.21e-05 tok=17.138B dt=7758 wait=234 +01:37:05 step 130280 loss=3.2946 lr=5.21e-05 tok=17.140B dt=7822 wait=234 +01:38:10 step 130300 loss=3.2654 lr=5.20e-05 tok=17.143B dt=7886 wait=235 +01:39:14 step 130320 loss=3.3877 lr=5.20e-05 tok=17.146B dt=7950 wait=235 +01:40:18 step 130340 loss=3.3364 lr=5.20e-05 tok=17.148B dt=8014 wait=235 +01:41:01 pushed ckpt_w121.pt,log_w121.txt +01:41:47 waited 45s for global r=476 +01:41:55 global r=476 load 4.280 -> 4.277 +01:42:28 step 130360 loss=3.4253 lr=5.20e-05 tok=17.151B dt=8144 wait=236 +01:43:39 step 130380 loss=3.2090 lr=5.20e-05 tok=17.153B dt=8215 wait=236 +01:44:55 step 130400 loss=3.2444 lr=5.20e-05 tok=17.156B dt=8292 wait=237 +01:45:59 step 130420 loss=3.4228 lr=5.19e-05 tok=17.159B dt=8356 wait=237 +01:47:03 step 130440 loss=3.3195 lr=5.19e-05 tok=17.161B dt=8420 wait=237 +01:48:07 step 130460 loss=3.4051 lr=5.19e-05 tok=17.164B dt=8484 wait=238 +01:49:11 step 130480 loss=3.3121 lr=5.19e-05 tok=17.166B dt=8547 wait=238 +01:50:15 step 130500 loss=3.3111 lr=5.19e-05 tok=17.169B dt=8612 wait=238 +01:50:27 pushed ckpt_w121.pt,log_w121.txt +01:51:12 waited 45s for global r=477 +01:51:20 global r=477 load 4.281 -> 4.286 +01:52:25 step 130520 loss=3.3057 lr=5.18e-05 tok=17.172B dt=8742 wait=239 +01:53:37 step 130540 loss=3.3793 lr=5.18e-05 tok=17.174B dt=8814 wait=239 +01:54:53 step 130560 loss=3.3956 lr=5.18e-05 tok=17.177B dt=8890 wait=239 +01:55:57 step 130580 loss=3.3152 lr=5.18e-05 tok=17.180B dt=8954 wait=240 +01:57:01 step 130600 loss=3.3144 lr=5.18e-05 tok=17.182B dt=9018 wait=240 +01:58:06 step 130620 loss=3.4777 lr=5.17e-05 tok=17.185B dt=9082 wait=241 +01:59:09 step 130640 loss=3.2720 lr=5.17e-05 tok=17.187B dt=9146 wait=241 +01:59:56 pushed ckpt_w121.pt,log_w121.txt +02:00:41 waited 45s for global r=478 +02:00:50 global r=478 load 4.281 -> 4.275 +02:01:23 step 130660 loss=3.4264 lr=5.17e-05 tok=17.190B dt=9280 wait=241 +02:02:27 step 130680 loss=3.1571 lr=5.17e-05 tok=17.193B dt=9344 wait=242 +02:03:51 step 130700 loss=3.3626 lr=5.17e-05 tok=17.195B dt=9428 wait=242 +02:04:55 step 130720 loss=3.4207 lr=5.16e-05 tok=17.198B dt=9492 wait=242 +02:05:59 step 130740 loss=3.2743 lr=5.16e-05 tok=17.201B dt=9556 wait=243 +02:07:03 step 130760 loss=3.4140 lr=5.16e-05 tok=17.203B dt=9620 wait=243 +02:08:07 step 130780 loss=3.3122 lr=5.16e-05 tok=17.206B dt=9684 wait=243 +02:09:12 step 130800 loss=3.3206 lr=5.16e-05 tok=17.208B dt=9748 wait=244 +02:09:27 pushed ckpt_w121.pt,log_w121.txt +02:10:13 waited 45s for global r=479 +02:10:21 global r=479 load 4.288 -> 4.286 +02:11:26 step 130820 loss=3.2902 lr=5.16e-05 tok=17.211B dt=9882 wait=244 +02:12:38 step 130840 loss=3.3402 lr=5.15e-05 tok=17.214B dt=9954 wait=245 +02:13:53 step 130860 loss=3.3752 lr=5.15e-05 tok=17.216B dt=10030 wait=245 +02:14:57 step 130880 loss=3.3540 lr=5.15e-05 tok=17.219B dt=10094 wait=245 +02:16:02 step 130900 loss=3.3881 lr=5.15e-05 tok=17.222B dt=10159 wait=246 +02:17:06 step 130920 loss=3.4596 lr=5.15e-05 tok=17.224B dt=10223 wait=246 +02:18:11 step 130940 loss=3.3250 lr=5.14e-05 tok=17.227B dt=10287 wait=246 +02:18:58 pushed ckpt_w121.pt,log_w121.txt +02:19:43 waited 45s for global r=480 +02:19:50 global r=480 load 4.291 -> 4.283 +02:20:29 step 130960 loss=3.4154 lr=5.14e-05 tok=17.229B dt=10425 wait=247 +02:21:40 step 130980 loss=3.2863 lr=5.14e-05 tok=17.232B dt=10496 wait=247 +02:22:44 step 131000 loss=3.2931 lr=5.14e-05 tok=17.235B dt=10560 wait=248 +02:23:48 step 131020 loss=3.4546 lr=5.14e-05 tok=17.237B dt=10624 wait=248 +02:24:52 step 131040 loss=3.2522 lr=5.13e-05 tok=17.240B dt=10688 wait=248 +02:25:56 step 131060 loss=3.3986 lr=5.13e-05 tok=17.243B dt=10752 wait=249 +02:27:00 step 131080 loss=3.2940 lr=5.13e-05 tok=17.245B dt=10816 wait=249 +02:28:04 step 131100 loss=3.3094 lr=5.13e-05 tok=17.248B dt=10880 wait=249 +02:28:16 pushed ckpt_w121.pt,log_w121.txt +02:32:23 step 131120 loss=3.3484 lr=5.13e-05 tok=17.250B dt=11139 wait=250 +02:33:26 step 131140 loss=3.3009 lr=5.13e-05 tok=17.253B dt=11203 wait=250 +02:34:30 step 131160 loss=3.2269 lr=5.12e-05 tok=17.256B dt=11267 wait=251 +02:35:34 step 131180 loss=3.2143 lr=5.12e-05 tok=17.258B dt=11331 wait=251 +02:36:39 step 131200 loss=3.1865 lr=5.12e-05 tok=17.261B dt=11395 wait=251 +02:37:42 step 131220 loss=3.4165 lr=5.12e-05 tok=17.263B dt=11459 wait=252 +02:38:46 step 131240 loss=3.2983 lr=5.12e-05 tok=17.266B dt=11523 wait=252 diff --git a/log_w122.txt b/log_w122.txt new file mode 100644 index 0000000000000000000000000000000000000000..7fac9848ec26e5f4be258aed17d9fca1cb20a6ed --- /dev/null +++ b/log_w122.txt @@ -0,0 +1,216 @@ +launch +23:24:29 name=w122 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 69/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +23:24:35 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +23:24:35 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +23:24:44 params 1.08B +23:24:46 resumed step=128250 seen=16.87B round=462 +23:24:46 stream salt=128250 +23:26:42 blocks compiled +23:26:42 start: accum=4, 131072 tokens per optimiser step +23:29:59 step 128251 loss=3.1042 lr=5.40e-05 tok=16.874B dt=196 wait=156 +23:30:02 step 128252 loss=3.2025 lr=5.40e-05 tok=16.874B dt=200 wait=156 +23:30:29 step 128260 loss=3.4550 lr=5.40e-05 tok=16.876B dt=227 wait=156 +23:31:37 step 128280 loss=3.3384 lr=5.40e-05 tok=16.878B dt=294 wait=157 +23:32:44 step 128300 loss=3.3747 lr=5.40e-05 tok=16.881B dt=362 wait=157 +23:33:47 step 128320 loss=3.3436 lr=5.39e-05 tok=16.883B dt=425 wait=158 +23:34:51 step 128340 loss=3.3664 lr=5.39e-05 tok=16.886B dt=489 wait=158 +23:35:54 step 128360 loss=3.4807 lr=5.39e-05 tok=16.889B dt=552 wait=158 +23:36:58 step 128380 loss=3.3566 lr=5.39e-05 tok=16.891B dt=616 wait=158 +23:38:01 step 128400 loss=3.4134 lr=5.39e-05 tok=16.894B dt=679 wait=159 +23:38:15 pushed ckpt_w122.pt,log_w122.txt +23:39:00 waited 45s for global r=463 +23:39:17 global r=463 load 4.282 -> 4.269 +23:40:23 step 128420 loss=3.3219 lr=5.38e-05 tok=16.896B dt=821 wait=159 +23:41:27 step 128440 loss=3.3612 lr=5.38e-05 tok=16.899B dt=885 wait=159 +23:42:30 step 128460 loss=3.3900 lr=5.38e-05 tok=16.902B dt=948 wait=160 +23:43:33 step 128480 loss=3.2930 lr=5.38e-05 tok=16.904B dt=1011 wait=160 +23:44:37 step 128500 loss=3.3463 lr=5.38e-05 tok=16.907B dt=1075 wait=160 +23:45:40 step 128520 loss=3.4390 lr=5.37e-05 tok=16.910B dt=1138 wait=161 +23:46:44 step 128540 loss=3.2567 lr=5.37e-05 tok=16.912B dt=1202 wait=161 +23:47:29 pushed ckpt_w122.pt,log_w122.txt +23:48:45 waited 76s for global r=464 +23:48:55 global r=464 load 4.279 -> 4.271 +23:49:29 step 128560 loss=3.3633 lr=5.37e-05 tok=16.915B dt=1367 wait=161 +23:50:32 step 128580 loss=3.2787 lr=5.37e-05 tok=16.917B dt=1430 wait=162 +23:51:36 step 128600 loss=3.2734 lr=5.37e-05 tok=16.920B dt=1494 wait=162 +23:52:39 step 128620 loss=3.4036 lr=5.36e-05 tok=16.923B dt=1557 wait=162 +23:53:43 step 128640 loss=3.2049 lr=5.36e-05 tok=16.925B dt=1620 wait=163 +23:54:46 step 128660 loss=3.3834 lr=5.36e-05 tok=16.928B dt=1684 wait=163 +23:55:50 step 128680 loss=3.3118 lr=5.36e-05 tok=16.931B dt=1748 wait=163 +23:56:54 step 128700 loss=3.3543 lr=5.36e-05 tok=16.933B dt=1812 wait=164 +23:57:05 pushed ckpt_w122.pt,log_w122.txt +23:58:06 waited 61s for global r=465 +23:58:13 global r=465 load 4.276 -> 4.282 +23:59:18 step 128720 loss=3.2871 lr=5.35e-05 tok=16.936B dt=1956 wait=164 +00:00:21 step 128740 loss=3.3216 lr=5.35e-05 tok=16.938B dt=2019 wait=164 +00:01:25 step 128760 loss=3.3321 lr=5.35e-05 tok=16.941B dt=2082 wait=165 +00:02:28 step 128780 loss=3.2718 lr=5.35e-05 tok=16.944B dt=2146 wait=165 +00:03:32 step 128800 loss=3.2320 lr=5.35e-05 tok=16.946B dt=2210 wait=165 +00:04:36 step 128820 loss=3.4031 lr=5.35e-05 tok=16.949B dt=2274 wait=166 +00:05:39 step 128840 loss=3.2198 lr=5.34e-05 tok=16.952B dt=2337 wait=166 +00:06:23 pushed ckpt_w122.pt,log_w122.txt +00:07:23 waited 61s for global r=466 +00:07:31 global r=466 load 4.319 -> 4.287 +00:08:03 step 128860 loss=3.3663 lr=5.34e-05 tok=16.954B dt=2481 wait=166 +00:09:07 step 128880 loss=3.2707 lr=5.34e-05 tok=16.957B dt=2544 wait=166 +00:10:10 step 128900 loss=3.2989 lr=5.34e-05 tok=16.959B dt=2608 wait=167 +00:11:13 step 128920 loss=3.4125 lr=5.34e-05 tok=16.962B dt=2671 wait=167 +00:12:17 step 128940 loss=3.2378 lr=5.33e-05 tok=16.965B dt=2734 wait=167 +00:13:20 step 128960 loss=3.3598 lr=5.33e-05 tok=16.967B dt=2798 wait=168 +00:14:24 step 128980 loss=3.3015 lr=5.33e-05 tok=16.970B dt=2861 wait=168 +00:15:27 step 129000 loss=3.3695 lr=5.33e-05 tok=16.973B dt=2925 wait=168 +00:15:39 pushed ckpt_w122.pt,log_w122.txt +00:16:40 waited 61s for global r=467 +00:16:46 global r=467 load 4.281 -> 4.278 +00:17:50 step 129020 loss=3.2554 lr=5.33e-05 tok=16.975B dt=3068 wait=169 +00:18:54 step 129040 loss=3.3155 lr=5.32e-05 tok=16.978B dt=3132 wait=169 +00:19:57 step 129060 loss=3.2773 lr=5.32e-05 tok=16.980B dt=3195 wait=169 +00:21:01 step 129080 loss=3.2576 lr=5.32e-05 tok=16.983B dt=3259 wait=170 +00:22:04 step 129100 loss=3.2703 lr=5.32e-05 tok=16.986B dt=3322 wait=170 +00:23:07 step 129120 loss=3.3691 lr=5.32e-05 tok=16.988B dt=3385 wait=170 +00:24:11 step 129140 loss=3.2787 lr=5.31e-05 tok=16.991B dt=3449 wait=171 +00:24:54 pushed ckpt_w122.pt,log_w122.txt +00:26:10 waited 76s for global r=468 +00:26:18 global r=468 load 4.313 -> 4.290 +00:26:51 step 129160 loss=3.4208 lr=5.31e-05 tok=16.993B dt=3609 wait=171 +00:27:54 step 129180 loss=3.2983 lr=5.31e-05 tok=16.996B dt=3672 wait=171 +00:28:58 step 129200 loss=3.3131 lr=5.31e-05 tok=16.999B dt=3736 wait=172 +00:30:01 step 129220 loss=3.3490 lr=5.31e-05 tok=17.001B dt=3799 wait=172 +00:31:05 step 129240 loss=3.2470 lr=5.31e-05 tok=17.004B dt=3863 wait=172 +00:32:08 step 129260 loss=3.3666 lr=5.30e-05 tok=17.007B dt=3926 wait=173 +00:33:12 step 129280 loss=3.3148 lr=5.30e-05 tok=17.009B dt=3990 wait=173 +00:34:15 step 129300 loss=3.3358 lr=5.30e-05 tok=17.012B dt=4053 wait=173 +00:34:28 pushed ckpt_w122.pt,log_w122.txt +00:35:43 waited 76s for global r=469 +00:35:49 global r=469 load 4.307 -> 4.286 +00:36:53 step 129320 loss=3.2589 lr=5.30e-05 tok=17.014B dt=4211 wait=174 +00:37:57 step 129340 loss=3.3254 lr=5.30e-05 tok=17.017B dt=4274 wait=174 +00:39:00 step 129360 loss=3.2910 lr=5.29e-05 tok=17.020B dt=4338 wait=174 +00:40:03 step 129380 loss=3.2357 lr=5.29e-05 tok=17.022B dt=4401 wait=175 +00:41:06 step 129400 loss=3.2914 lr=5.29e-05 tok=17.025B dt=4464 wait=175 +00:42:09 step 129420 loss=3.3888 lr=5.29e-05 tok=17.028B dt=4527 wait=175 +00:43:17 step 129440 loss=3.2944 lr=5.29e-05 tok=17.030B dt=4595 wait=176 +00:44:01 pushed ckpt_w122.pt,log_w122.txt +00:45:02 waited 61s for global r=470 +00:45:11 global r=470 load 4.290 -> 4.288 +00:45:44 step 129460 loss=3.3681 lr=5.28e-05 tok=17.033B dt=4742 wait=176 +00:46:47 step 129480 loss=3.3066 lr=5.28e-05 tok=17.035B dt=4805 wait=176 +00:47:50 step 129500 loss=3.2532 lr=5.28e-05 tok=17.038B dt=4868 wait=177 +00:48:53 step 129520 loss=3.3918 lr=5.28e-05 tok=17.041B dt=4931 wait=177 +00:49:57 step 129540 loss=3.3179 lr=5.28e-05 tok=17.043B dt=4995 wait=177 +00:51:00 step 129560 loss=3.4174 lr=5.27e-05 tok=17.046B dt=5058 wait=178 +00:52:04 step 129580 loss=3.2943 lr=5.27e-05 tok=17.049B dt=5122 wait=178 +00:53:08 step 129600 loss=3.3237 lr=5.27e-05 tok=17.051B dt=5185 wait=178 +00:53:20 pushed ckpt_w122.pt,log_w122.txt +00:54:35 waited 76s for global r=471 +00:54:44 global r=471 load 4.290 -> 4.287 +00:55:48 step 129620 loss=3.2945 lr=5.27e-05 tok=17.054B dt=5346 wait=179 +00:56:51 step 129640 loss=3.3544 lr=5.27e-05 tok=17.056B dt=5409 wait=179 +00:57:55 step 129660 loss=3.3594 lr=5.27e-05 tok=17.059B dt=5473 wait=179 +00:58:58 step 129680 loss=3.2735 lr=5.26e-05 tok=17.062B dt=5536 wait=179 +01:00:01 step 129700 loss=3.2483 lr=5.26e-05 tok=17.064B dt=5599 wait=180 +01:01:05 step 129720 loss=3.3949 lr=5.26e-05 tok=17.067B dt=5663 wait=180 +01:02:09 step 129740 loss=3.1849 lr=5.26e-05 tok=17.070B dt=5726 wait=180 +01:02:56 pushed ckpt_w122.pt,log_w122.txt +01:03:57 waited 61s for global r=472 +01:04:04 global r=472 load 4.305 -> 4.282 +01:04:37 step 129760 loss=3.3082 lr=5.26e-05 tok=17.072B dt=5875 wait=181 +01:05:41 step 129780 loss=3.1652 lr=5.25e-05 tok=17.075B dt=5939 wait=181 +01:06:44 step 129800 loss=3.1999 lr=5.25e-05 tok=17.077B dt=6002 wait=181 +01:07:48 step 129820 loss=3.3181 lr=5.25e-05 tok=17.080B dt=6066 wait=182 +01:08:51 step 129840 loss=3.2609 lr=5.25e-05 tok=17.083B dt=6129 wait=182 +01:09:55 step 129860 loss=3.3863 lr=5.25e-05 tok=17.085B dt=6193 wait=182 +01:10:58 step 129880 loss=3.2850 lr=5.24e-05 tok=17.088B dt=6256 wait=183 +01:12:02 step 129900 loss=3.3250 lr=5.24e-05 tok=17.090B dt=6320 wait=183 +01:12:18 pushed ckpt_w122.pt,log_w122.txt +01:13:18 waited 61s for global r=473 +01:13:28 global r=473 load 4.294 -> 4.291 +01:14:32 step 129920 loss=3.2894 lr=5.24e-05 tok=17.093B dt=6470 wait=183 +01:15:36 step 129940 loss=3.3556 lr=5.24e-05 tok=17.096B dt=6533 wait=184 +01:16:39 step 129960 loss=3.3362 lr=5.24e-05 tok=17.098B dt=6597 wait=184 +01:17:43 step 129980 loss=3.2773 lr=5.23e-05 tok=17.101B dt=6661 wait=184 +01:18:46 step 130000 loss=3.3169 lr=5.23e-05 tok=17.104B dt=6724 wait=185 +01:19:50 step 130020 loss=3.3967 lr=5.23e-05 tok=17.106B dt=6788 wait=185 +01:20:53 step 130040 loss=3.2781 lr=5.23e-05 tok=17.109B dt=6851 wait=185 +01:21:41 pushed ckpt_w122.pt,log_w122.txt +01:22:57 waited 76s for global r=474 +01:23:02 global r=474 load 4.288 -> 4.288 +01:23:35 step 130060 loss=3.4066 lr=5.23e-05 tok=17.111B dt=7013 wait=186 +01:24:38 step 130080 loss=3.2773 lr=5.23e-05 tok=17.114B dt=7076 wait=186 +01:25:41 step 130100 loss=3.2989 lr=5.22e-05 tok=17.117B dt=7139 wait=186 +01:26:45 step 130120 loss=3.3997 lr=5.22e-05 tok=17.119B dt=7203 wait=186 +01:27:48 step 130140 loss=3.2497 lr=5.22e-05 tok=17.122B dt=7266 wait=187 +01:28:52 step 130160 loss=3.3491 lr=5.22e-05 tok=17.125B dt=7330 wait=187 +01:29:55 step 130180 loss=3.3381 lr=5.22e-05 tok=17.127B dt=7393 wait=187 +01:30:59 step 130200 loss=3.3439 lr=5.21e-05 tok=17.130B dt=7456 wait=188 +01:31:17 pushed ckpt_w122.pt,log_w122.txt +01:32:18 waited 61s for global r=475 +01:32:26 global r=475 load 4.302 -> 4.277 +01:33:30 step 130220 loss=3.2886 lr=5.21e-05 tok=17.132B dt=7608 wait=188 +01:34:33 step 130240 loss=3.2942 lr=5.21e-05 tok=17.135B dt=7671 wait=188 +01:35:36 step 130260 loss=3.2981 lr=5.21e-05 tok=17.138B dt=7734 wait=189 +01:36:40 step 130280 loss=3.2552 lr=5.21e-05 tok=17.140B dt=7797 wait=189 +01:37:43 step 130300 loss=3.3300 lr=5.20e-05 tok=17.143B dt=7861 wait=189 +01:38:46 step 130320 loss=3.3686 lr=5.20e-05 tok=17.146B dt=7924 wait=190 +01:39:50 step 130340 loss=3.2875 lr=5.20e-05 tok=17.148B dt=7988 wait=190 +01:40:34 pushed ckpt_w122.pt,log_w122.txt +01:41:50 waited 76s for global r=476 +01:41:59 global r=476 load 4.277 -> 4.277 +01:42:32 step 130360 loss=3.3685 lr=5.20e-05 tok=17.151B dt=8150 wait=190 +01:43:36 step 130380 loss=3.2879 lr=5.20e-05 tok=17.153B dt=8213 wait=191 +01:44:39 step 130400 loss=3.2785 lr=5.20e-05 tok=17.156B dt=8277 wait=191 +01:45:42 step 130420 loss=3.3848 lr=5.19e-05 tok=17.159B dt=8340 wait=191 +01:46:46 step 130440 loss=3.2421 lr=5.19e-05 tok=17.161B dt=8404 wait=191 +01:47:49 step 130460 loss=3.3152 lr=5.19e-05 tok=17.164B dt=8467 wait=192 +01:48:53 step 130480 loss=3.3037 lr=5.19e-05 tok=17.166B dt=8530 wait=192 +01:49:58 step 130500 loss=2.8481 lr=5.19e-05 tok=17.169B dt=8595 wait=192 +01:50:10 pushed ckpt_w122.pt,log_w122.txt +01:51:11 waited 61s for global r=477 +01:51:17 global r=477 load 4.305 -> 4.286 +01:52:22 step 130520 loss=3.2386 lr=5.18e-05 tok=17.172B dt=8740 wait=193 +01:53:25 step 130540 loss=3.3591 lr=5.18e-05 tok=17.174B dt=8803 wait=193 +01:54:28 step 130560 loss=3.3244 lr=5.18e-05 tok=17.177B dt=8866 wait=193 +01:55:32 step 130580 loss=3.2873 lr=5.18e-05 tok=17.180B dt=8930 wait=194 +01:56:35 step 130600 loss=3.2500 lr=5.18e-05 tok=17.182B dt=8993 wait=194 +01:57:38 step 130620 loss=3.3949 lr=5.17e-05 tok=17.185B dt=9056 wait=194 +01:58:41 step 130640 loss=3.2624 lr=5.17e-05 tok=17.187B dt=9119 wait=195 +01:59:28 pushed ckpt_w122.pt,log_w122.txt +02:00:43 waited 76s for global r=478 +02:00:53 global r=478 load 4.302 -> 4.275 +02:01:26 step 130660 loss=3.4082 lr=5.17e-05 tok=17.190B dt=9284 wait=195 +02:02:29 step 130680 loss=3.2853 lr=5.17e-05 tok=17.193B dt=9347 wait=195 +02:03:33 step 130700 loss=3.3348 lr=5.17e-05 tok=17.195B dt=9411 wait=196 +02:04:37 step 130720 loss=3.3951 lr=5.16e-05 tok=17.198B dt=9475 wait=196 +02:05:40 step 130740 loss=3.2260 lr=5.16e-05 tok=17.201B dt=9538 wait=196 +02:06:44 step 130760 loss=3.3180 lr=5.16e-05 tok=17.203B dt=9601 wait=197 +02:07:47 step 130780 loss=3.3335 lr=5.16e-05 tok=17.206B dt=9665 wait=197 +02:08:50 step 130800 loss=3.3246 lr=5.16e-05 tok=17.208B dt=9728 wait=197 +02:09:02 pushed ckpt_w122.pt,log_w122.txt +02:10:02 waited 60s for global r=479 +02:10:10 global r=479 load 4.313 -> 4.286 +02:11:15 step 130820 loss=3.3310 lr=5.16e-05 tok=17.211B dt=9872 wait=198 +02:12:18 step 130840 loss=3.3572 lr=5.15e-05 tok=17.214B dt=9936 wait=198 +02:13:22 step 130860 loss=3.3969 lr=5.15e-05 tok=17.216B dt=10000 wait=198 +02:14:26 step 130880 loss=3.2850 lr=5.15e-05 tok=17.219B dt=10063 wait=199 +02:15:29 step 130900 loss=3.3407 lr=5.15e-05 tok=17.222B dt=10127 wait=199 +02:16:32 step 130920 loss=3.3774 lr=5.15e-05 tok=17.224B dt=10190 wait=199 +02:17:36 step 130940 loss=3.2491 lr=5.14e-05 tok=17.227B dt=10254 wait=199 +02:18:21 pushed ckpt_w122.pt,log_w122.txt +02:19:37 waited 76s for global r=480 +02:19:42 global r=480 load 4.304 -> 4.283 +02:20:15 step 130960 loss=3.4321 lr=5.14e-05 tok=17.229B dt=10413 wait=200 +02:21:19 step 130980 loss=3.2434 lr=5.14e-05 tok=17.232B dt=10476 wait=200 +02:22:22 step 131000 loss=3.2576 lr=5.14e-05 tok=17.235B dt=10540 wait=200 +02:23:26 step 131020 loss=3.4149 lr=5.14e-05 tok=17.237B dt=10603 wait=201 +02:24:29 step 131040 loss=3.2913 lr=5.13e-05 tok=17.240B dt=10666 wait=201 +02:25:32 step 131060 loss=3.3955 lr=5.13e-05 tok=17.243B dt=10730 wait=201 +02:26:35 step 131080 loss=3.2762 lr=5.13e-05 tok=17.245B dt=10793 wait=202 +02:27:38 step 131100 loss=3.3136 lr=5.13e-05 tok=17.248B dt=10856 wait=202 +02:27:51 pushed ckpt_w122.pt,log_w122.txt +02:31:57 step 131120 loss=3.3611 lr=5.13e-05 tok=17.250B dt=11115 wait=202 +02:33:01 step 131140 loss=3.2602 lr=5.13e-05 tok=17.253B dt=11178 wait=202 +02:34:04 step 131160 loss=3.2217 lr=5.12e-05 tok=17.256B dt=11242 wait=203 +02:35:07 step 131180 loss=3.2548 lr=5.12e-05 tok=17.258B dt=11305 wait=203 +02:36:11 step 131200 loss=3.1130 lr=5.12e-05 tok=17.261B dt=11368 wait=203 +02:37:14 step 131220 loss=3.3813 lr=5.12e-05 tok=17.263B dt=11432 wait=204 +02:38:17 step 131240 loss=3.2521 lr=5.12e-05 tok=17.266B dt=11495 wait=204 diff --git a/log_w123.txt b/log_w123.txt new file mode 100644 index 0000000000000000000000000000000000000000..f6e29e8daa2e6c848b65b13d6eb45908596ed450 --- /dev/null +++ b/log_w123.txt @@ -0,0 +1,197 @@ +launch +08:49:16 name=w123 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 70/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +08:49:23 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +08:49:23 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +08:49:32 params 1.08B +08:49:34 resumed step=131250 seen=17.27B round=481 +08:49:34 stream salt=131250 +08:51:17 blocks compiled +08:51:17 start: accum=4, 131072 tokens per optimiser step +08:53:35 step 131251 loss=3.1709 lr=5.11e-05 tok=17.268B dt=138 wait=96 +08:53:38 step 131252 loss=3.2297 lr=5.11e-05 tok=17.268B dt=141 wait=96 +08:54:07 step 131260 loss=3.5322 lr=5.11e-05 tok=17.269B dt=170 wait=96 +08:55:15 step 131280 loss=3.3223 lr=5.11e-05 tok=17.271B dt=238 wait=97 +08:56:19 step 131300 loss=3.4338 lr=5.11e-05 tok=17.274B dt=302 wait=97 +08:57:22 step 131320 loss=3.4013 lr=5.11e-05 tok=17.277B dt=365 wait=97 +08:58:26 step 131340 loss=3.4242 lr=5.11e-05 tok=17.279B dt=429 wait=98 +08:59:31 step 131360 loss=3.5402 lr=5.10e-05 tok=17.282B dt=494 wait=98 +09:00:35 step 131380 loss=3.3792 lr=5.10e-05 tok=17.284B dt=558 wait=99 +09:01:38 step 131400 loss=3.4065 lr=5.10e-05 tok=17.287B dt=621 wait=99 +09:01:54 pushed ckpt_w123.pt,log_w123.txt +09:02:54 waited 61s for global r=482 +09:03:06 global r=482 load 4.297 -> 4.279 +09:04:15 step 131420 loss=3.3154 lr=5.10e-05 tok=17.290B dt=778 wait=99 +09:05:19 step 131440 loss=3.3484 lr=5.10e-05 tok=17.292B dt=842 wait=100 +09:06:23 step 131460 loss=3.3517 lr=5.10e-05 tok=17.295B dt=906 wait=100 +09:07:27 step 131480 loss=3.3518 lr=5.09e-05 tok=17.298B dt=970 wait=100 +09:08:30 step 131500 loss=3.3607 lr=5.09e-05 tok=17.300B dt=1033 wait=101 +09:09:34 step 131520 loss=3.4123 lr=5.09e-05 tok=17.303B dt=1097 wait=101 +09:10:38 step 131540 loss=3.2946 lr=5.09e-05 tok=17.305B dt=1161 wait=101 +09:11:21 pushed ckpt_w123.pt,log_w123.txt +09:12:22 waited 61s for global r=483 +09:12:27 global r=483 load 4.295 -> 4.282 +09:13:01 step 131560 loss=3.4500 lr=5.09e-05 tok=17.308B dt=1304 wait=102 +09:14:05 step 131580 loss=3.3492 lr=5.08e-05 tok=17.311B dt=1368 wait=102 +09:15:09 step 131600 loss=3.2756 lr=5.08e-05 tok=17.313B dt=1432 wait=102 +09:16:13 step 131620 loss=3.4173 lr=5.08e-05 tok=17.316B dt=1496 wait=103 +09:17:16 step 131640 loss=3.2812 lr=5.08e-05 tok=17.319B dt=1559 wait=103 +09:18:20 step 131660 loss=3.3890 lr=5.08e-05 tok=17.321B dt=1623 wait=103 +09:19:23 step 131680 loss=3.3215 lr=5.07e-05 tok=17.324B dt=1686 wait=104 +09:20:26 step 131700 loss=3.3420 lr=5.07e-05 tok=17.326B dt=1749 wait=104 +09:20:38 pushed ckpt_w123.pt,log_w123.txt +09:21:39 waited 61s for global r=484 +09:21:46 global r=484 load 4.322 -> 4.286 +09:22:51 step 131720 loss=3.2943 lr=5.07e-05 tok=17.329B dt=1894 wait=104 +09:23:55 step 131740 loss=3.3637 lr=5.07e-05 tok=17.332B dt=1958 wait=105 +09:24:58 step 131760 loss=3.3306 lr=5.07e-05 tok=17.334B dt=2021 wait=105 +09:26:01 step 131780 loss=3.3157 lr=5.07e-05 tok=17.337B dt=2084 wait=105 +09:27:05 step 131800 loss=3.3327 lr=5.06e-05 tok=17.340B dt=2148 wait=106 +09:28:09 step 131820 loss=3.4136 lr=5.06e-05 tok=17.342B dt=2212 wait=106 +09:29:13 step 131840 loss=3.2876 lr=5.06e-05 tok=17.345B dt=2276 wait=106 +09:29:59 pushed ckpt_w123.pt,log_w123.txt +09:31:00 waited 61s for global r=485 +09:31:08 global r=485 load 4.294 -> 4.296 +09:31:41 step 131860 loss=3.4308 lr=5.06e-05 tok=17.347B dt=2424 wait=107 +09:32:44 step 131880 loss=3.3502 lr=5.06e-05 tok=17.350B dt=2487 wait=107 +09:33:48 step 131900 loss=3.3310 lr=5.05e-05 tok=17.353B dt=2551 wait=107 +09:34:52 step 131920 loss=3.4467 lr=5.05e-05 tok=17.355B dt=2615 wait=108 +09:35:55 step 131940 loss=3.3063 lr=5.05e-05 tok=17.358B dt=2678 wait=108 +09:36:59 step 131960 loss=3.4389 lr=5.05e-05 tok=17.360B dt=2742 wait=108 +09:38:03 step 131980 loss=3.3194 lr=5.05e-05 tok=17.363B dt=2806 wait=109 +09:39:07 step 132000 loss=3.4197 lr=5.04e-05 tok=17.366B dt=2870 wait=109 +09:39:22 pushed ckpt_w123.pt,log_w123.txt +09:40:23 waited 60s for global r=486 +09:40:27 global r=486 load 4.269 -> 4.263 +09:41:32 step 132020 loss=3.3746 lr=5.04e-05 tok=17.368B dt=3015 wait=109 +09:42:37 step 132040 loss=3.3651 lr=5.04e-05 tok=17.371B dt=3080 wait=110 +09:43:41 step 132060 loss=3.4549 lr=5.04e-05 tok=17.374B dt=3144 wait=110 +09:44:45 step 132080 loss=3.3311 lr=5.04e-05 tok=17.376B dt=3208 wait=110 +09:45:49 step 132100 loss=3.3512 lr=5.04e-05 tok=17.379B dt=3272 wait=111 +09:46:52 step 132120 loss=3.4800 lr=5.03e-05 tok=17.381B dt=3335 wait=111 +09:47:56 step 132140 loss=3.2960 lr=5.03e-05 tok=17.384B dt=3399 wait=111 +09:48:41 pushed ckpt_w123.pt,log_w123.txt +09:49:42 waited 61s for global r=487 +09:49:50 global r=487 load 4.278 -> 4.277 +09:50:23 step 132160 loss=3.4262 lr=5.03e-05 tok=17.387B dt=3546 wait=112 +09:51:27 step 132180 loss=3.3109 lr=5.03e-05 tok=17.389B dt=3610 wait=112 +09:52:31 step 132200 loss=3.3514 lr=5.03e-05 tok=17.392B dt=3674 wait=112 +09:53:35 step 132220 loss=3.4679 lr=5.02e-05 tok=17.395B dt=3738 wait=113 +09:54:38 step 132240 loss=3.3135 lr=5.02e-05 tok=17.397B dt=3801 wait=113 +09:55:42 step 132260 loss=3.4710 lr=5.02e-05 tok=17.400B dt=3865 wait=113 +09:56:46 step 132280 loss=3.3546 lr=5.02e-05 tok=17.402B dt=3929 wait=113 +09:57:49 step 132300 loss=3.3858 lr=5.02e-05 tok=17.405B dt=3992 wait=114 +09:58:01 pushed ckpt_w123.pt,log_w123.txt +09:59:02 waited 61s for global r=488 +09:59:10 global r=488 load 4.278 -> 4.281 +10:00:16 step 132320 loss=3.3210 lr=5.01e-05 tok=17.408B dt=4139 wait=114 +10:01:19 step 132340 loss=3.3910 lr=5.01e-05 tok=17.410B dt=4202 wait=114 +10:02:23 step 132360 loss=3.3547 lr=5.01e-05 tok=17.413B dt=4266 wait=115 +10:03:26 step 132380 loss=3.3173 lr=5.01e-05 tok=17.416B dt=4329 wait=115 +10:04:29 step 132400 loss=3.3785 lr=5.01e-05 tok=17.418B dt=4392 wait=115 +10:05:32 step 132420 loss=3.3760 lr=5.01e-05 tok=17.421B dt=4455 wait=116 +10:06:35 step 132440 loss=3.3015 lr=5.00e-05 tok=17.423B dt=4518 wait=116 +10:07:22 pushed ckpt_w123.pt,log_w123.txt +10:08:53 waited 91s for global r=489 +10:09:04 global r=489 load 4.279 -> 4.281 +10:09:36 step 132460 loss=3.3528 lr=5.00e-05 tok=17.426B dt=4699 wait=116 +10:10:39 step 132480 loss=3.3128 lr=5.00e-05 tok=17.429B dt=4762 wait=117 +10:11:42 step 132500 loss=3.2936 lr=5.00e-05 tok=17.431B dt=4825 wait=117 +10:12:45 step 132520 loss=3.3393 lr=5.00e-05 tok=17.434B dt=4888 wait=117 +10:13:48 step 132540 loss=3.2746 lr=4.99e-05 tok=17.437B dt=4951 wait=118 +10:14:51 step 132560 loss=3.3328 lr=4.99e-05 tok=17.439B dt=5014 wait=118 +10:15:54 step 132580 loss=3.2819 lr=4.99e-05 tok=17.442B dt=5077 wait=118 +10:16:57 step 132600 loss=3.3005 lr=4.99e-05 tok=17.444B dt=5140 wait=118 +10:17:13 pushed ckpt_w123.pt,log_w123.txt +10:18:14 waited 60s for global r=490 +10:18:21 global r=490 load 4.286 -> 4.287 +10:19:24 step 132620 loss=3.2503 lr=4.99e-05 tok=17.447B dt=5287 wait=119 +10:20:27 step 132640 loss=3.3188 lr=4.99e-05 tok=17.450B dt=5350 wait=119 +10:21:30 step 132660 loss=3.2447 lr=4.98e-05 tok=17.452B dt=5413 wait=119 +10:22:34 step 132680 loss=3.2645 lr=4.98e-05 tok=17.455B dt=5477 wait=120 +10:23:37 step 132700 loss=3.3047 lr=4.98e-05 tok=17.457B dt=5540 wait=120 +10:24:40 step 132720 loss=3.3933 lr=4.98e-05 tok=17.460B dt=5603 wait=120 +10:25:43 step 132740 loss=3.2791 lr=4.98e-05 tok=17.463B dt=5666 wait=121 +10:26:39 pushed ckpt_w123.pt,log_w123.txt +10:27:40 waited 61s for global r=491 +10:27:47 global r=491 load 4.303 -> 4.299 +10:28:19 step 132760 loss=3.4008 lr=4.97e-05 tok=17.465B dt=5822 wait=121 +10:29:23 step 132780 loss=3.1015 lr=4.97e-05 tok=17.468B dt=5886 wait=121 +10:30:26 step 132800 loss=3.2781 lr=4.97e-05 tok=17.471B dt=5949 wait=122 +10:31:29 step 132820 loss=3.3797 lr=4.97e-05 tok=17.473B dt=6012 wait=122 +10:32:32 step 132840 loss=3.2388 lr=4.97e-05 tok=17.476B dt=6075 wait=122 +10:33:35 step 132860 loss=3.3475 lr=4.96e-05 tok=17.478B dt=6138 wait=123 +10:34:38 step 132880 loss=3.2729 lr=4.96e-05 tok=17.481B dt=6201 wait=123 +10:35:41 step 132900 loss=3.3199 lr=4.96e-05 tok=17.484B dt=6264 wait=123 +10:35:53 pushed ckpt_w123.pt,log_w123.txt +10:36:39 waited 46s for global r=492 +10:36:54 global r=492 load 4.280 -> 4.280 +10:37:59 step 132920 loss=3.2780 lr=4.96e-05 tok=17.486B dt=6402 wait=124 +10:39:02 step 132940 loss=3.3677 lr=4.96e-05 tok=17.489B dt=6465 wait=124 +10:40:05 step 132960 loss=3.2816 lr=4.96e-05 tok=17.492B dt=6528 wait=124 +10:41:09 step 132980 loss=3.2871 lr=4.95e-05 tok=17.494B dt=6592 wait=125 +10:42:12 step 133000 loss=3.2872 lr=4.95e-05 tok=17.497B dt=6655 wait=125 +10:43:16 step 133020 loss=3.3853 lr=4.95e-05 tok=17.499B dt=6719 wait=125 +10:44:19 step 133040 loss=3.2971 lr=4.95e-05 tok=17.502B dt=6782 wait=126 +10:45:05 pushed ckpt_w123.pt,log_w123.txt +10:46:06 waited 61s for global r=493 +10:46:12 global r=493 load 4.294 -> 4.287 +10:46:45 step 133060 loss=3.3809 lr=4.95e-05 tok=17.505B dt=6928 wait=126 +10:47:48 step 133080 loss=3.3168 lr=4.94e-05 tok=17.507B dt=6991 wait=126 +10:48:51 step 133100 loss=3.2798 lr=4.94e-05 tok=17.510B dt=7054 wait=126 +10:49:54 step 133120 loss=3.3958 lr=4.94e-05 tok=17.513B dt=7117 wait=127 +10:50:57 step 133140 loss=3.2520 lr=4.94e-05 tok=17.515B dt=7180 wait=127 +10:52:01 step 133160 loss=3.4163 lr=4.94e-05 tok=17.518B dt=7244 wait=127 +10:53:04 step 133180 loss=3.2803 lr=4.94e-05 tok=17.520B dt=7307 wait=128 +10:54:08 step 133200 loss=3.1229 lr=4.93e-05 tok=17.523B dt=7371 wait=128 +10:54:21 pushed ckpt_w123.pt,log_w123.txt +10:55:23 waited 62s for global r=494 +10:55:31 global r=494 load 4.286 -> 4.283 +10:56:36 step 133220 loss=3.2665 lr=4.93e-05 tok=17.526B dt=7519 wait=128 +10:57:39 step 133240 loss=3.3522 lr=4.93e-05 tok=17.528B dt=7582 wait=129 +10:58:42 step 133260 loss=3.3064 lr=4.93e-05 tok=17.531B dt=7645 wait=129 +10:59:45 step 133280 loss=3.3024 lr=4.93e-05 tok=17.534B dt=7708 wait=129 +11:00:49 step 133300 loss=3.2955 lr=4.92e-05 tok=17.536B dt=7772 wait=130 +11:01:52 step 133320 loss=3.3776 lr=4.92e-05 tok=17.539B dt=7835 wait=130 +11:02:56 step 133340 loss=3.0694 lr=4.92e-05 tok=17.541B dt=7899 wait=130 +11:03:42 pushed ckpt_w123.pt,log_w123.txt +11:04:43 waited 61s for global r=495 +11:04:50 global r=495 load 4.278 -> 4.289 +11:05:24 step 133360 loss=3.3033 lr=4.92e-05 tok=17.544B dt=8047 wait=131 +11:06:27 step 133380 loss=3.3028 lr=4.92e-05 tok=17.547B dt=8110 wait=131 +11:07:31 step 133400 loss=3.2774 lr=4.91e-05 tok=17.549B dt=8174 wait=131 +11:08:34 step 133420 loss=3.3655 lr=4.91e-05 tok=17.552B dt=8237 wait=132 +11:09:37 step 133440 loss=3.2423 lr=4.91e-05 tok=17.554B dt=8300 wait=132 +11:10:40 step 133460 loss=3.2789 lr=4.91e-05 tok=17.557B dt=8363 wait=132 +11:11:43 step 133480 loss=3.2943 lr=4.91e-05 tok=17.560B dt=8426 wait=133 +11:12:47 step 133500 loss=3.3359 lr=4.91e-05 tok=17.562B dt=8490 wait=133 +11:13:00 pushed ckpt_w123.pt,log_w123.txt +11:14:01 waited 61s for global r=496 +11:14:08 global r=496 load 4.307 -> 4.296 +11:15:13 step 133520 loss=3.2856 lr=4.90e-05 tok=17.565B dt=8636 wait=133 +11:16:16 step 133540 loss=3.3332 lr=4.90e-05 tok=17.568B dt=8699 wait=134 +11:17:19 step 133560 loss=3.3170 lr=4.90e-05 tok=17.570B dt=8762 wait=134 +11:18:22 step 133580 loss=3.3162 lr=4.90e-05 tok=17.573B dt=8825 wait=134 +11:19:25 step 133600 loss=3.2415 lr=4.90e-05 tok=17.575B dt=8888 wait=135 +11:20:28 step 133620 loss=3.3010 lr=4.89e-05 tok=17.578B dt=8951 wait=135 +11:21:31 step 133640 loss=3.2638 lr=4.89e-05 tok=17.581B dt=9014 wait=135 +11:22:19 pushed ckpt_w123.pt,log_w123.txt +11:23:19 waited 60s for global r=497 +11:23:29 global r=497 load 4.313 -> 4.298 +11:24:02 step 133660 loss=3.3373 lr=4.89e-05 tok=17.583B dt=9165 wait=136 +11:25:05 step 133680 loss=3.2642 lr=4.89e-05 tok=17.586B dt=9228 wait=136 +11:26:08 step 133700 loss=3.2721 lr=4.89e-05 tok=17.589B dt=9291 wait=136 +11:27:11 step 133720 loss=3.3381 lr=4.89e-05 tok=17.591B dt=9354 wait=137 +11:28:15 step 133740 loss=3.2834 lr=4.88e-05 tok=17.594B dt=9418 wait=137 +11:29:18 step 133760 loss=3.3742 lr=4.88e-05 tok=17.596B dt=9481 wait=137 +11:30:21 step 133780 loss=3.3087 lr=4.88e-05 tok=17.599B dt=9544 wait=138 +11:31:24 step 133800 loss=3.3221 lr=4.88e-05 tok=17.602B dt=9607 wait=138 +11:31:39 pushed ckpt_w123.pt,log_w123.txt +11:32:41 waited 61s for global r=498 +11:32:49 global r=498 load 4.306 -> 4.295 +11:33:54 step 133820 loss=3.2960 lr=4.88e-05 tok=17.604B dt=9757 wait=138 +11:34:57 step 133840 loss=3.3743 lr=4.87e-05 tok=17.607B dt=9820 wait=138 +11:36:01 step 133860 loss=3.3476 lr=4.87e-05 tok=17.610B dt=9884 wait=139 +11:37:04 step 133880 loss=3.2632 lr=4.87e-05 tok=17.612B dt=9947 wait=139 +11:38:07 step 133900 loss=3.3687 lr=4.87e-05 tok=17.615B dt=10010 wait=139 +11:39:10 step 133920 loss=3.3824 lr=4.87e-05 tok=17.617B dt=10073 wait=140 +11:40:13 step 133940 loss=3.2103 lr=4.87e-05 tok=17.620B dt=10136 wait=140 diff --git a/log_w124.txt b/log_w124.txt new file mode 100644 index 0000000000000000000000000000000000000000..a1ea19e011c18b48d30cddb4a8b0389818ea38b4 --- /dev/null +++ b/log_w124.txt @@ -0,0 +1,456 @@ +launch +08:49:21 name=w124 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 71/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +08:49:26 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +08:49:26 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +08:49:36 params 1.08B +08:49:37 resumed step=131250 seen=17.27B round=481 +08:49:37 stream salt=131250 +08:51:18 blocks compiled +08:51:18 start: accum=4, 131072 tokens per optimiser step +08:53:49 step 131251 loss=3.0793 lr=5.11e-05 tok=17.268B dt=151 wait=106 +08:53:53 step 131252 loss=3.0857 lr=5.11e-05 tok=17.268B dt=155 wait=106 +08:54:22 step 131260 loss=3.4447 lr=5.11e-05 tok=17.269B dt=184 wait=106 +08:55:30 step 131280 loss=3.3175 lr=5.11e-05 tok=17.271B dt=252 wait=107 +08:56:34 step 131300 loss=3.3475 lr=5.11e-05 tok=17.274B dt=316 wait=107 +08:57:38 step 131320 loss=3.3724 lr=5.11e-05 tok=17.277B dt=379 wait=107 +08:58:41 step 131340 loss=3.3094 lr=5.11e-05 tok=17.279B dt=443 wait=108 +08:59:45 step 131360 loss=3.5133 lr=5.10e-05 tok=17.282B dt=507 wait=108 +09:00:49 step 131380 loss=3.3395 lr=5.10e-05 tok=17.284B dt=571 wait=108 +09:01:52 step 131400 loss=3.4079 lr=5.10e-05 tok=17.287B dt=634 wait=109 +09:02:07 pushed ckpt_w124.pt,log_w124.txt +09:02:52 waited 45s for global r=482 +09:03:03 global r=482 load 4.277 -> 4.279 +09:04:24 step 131420 loss=3.3461 lr=5.10e-05 tok=17.290B dt=786 wait=109 +09:05:33 step 131440 loss=3.4057 lr=5.10e-05 tok=17.292B dt=855 wait=109 +09:06:36 step 131460 loss=3.3680 lr=5.10e-05 tok=17.295B dt=918 wait=110 +09:07:40 step 131480 loss=3.3189 lr=5.09e-05 tok=17.298B dt=982 wait=110 +09:08:44 step 131500 loss=3.3045 lr=5.09e-05 tok=17.300B dt=1046 wait=110 +09:09:47 step 131520 loss=3.4434 lr=5.09e-05 tok=17.303B dt=1109 wait=111 +09:10:51 step 131540 loss=3.2390 lr=5.09e-05 tok=17.305B dt=1172 wait=111 +09:11:35 pushed ckpt_w124.pt,log_w124.txt +09:12:20 waited 45s for global r=483 +09:12:26 global r=483 load 4.292 -> 4.282 +09:13:00 step 131560 loss=3.4223 lr=5.09e-05 tok=17.308B dt=1302 wait=111 +09:14:24 step 131580 loss=3.2547 lr=5.08e-05 tok=17.311B dt=1386 wait=112 +09:15:27 step 131600 loss=3.2849 lr=5.08e-05 tok=17.313B dt=1449 wait=112 +09:16:31 step 131620 loss=3.4236 lr=5.08e-05 tok=17.316B dt=1513 wait=112 +09:17:35 step 131640 loss=3.3078 lr=5.08e-05 tok=17.319B dt=1576 wait=112 +09:18:39 step 131660 loss=3.3372 lr=5.08e-05 tok=17.321B dt=1641 wait=113 +09:19:42 step 131680 loss=3.2534 lr=5.07e-05 tok=17.324B dt=1704 wait=113 +09:20:46 step 131700 loss=3.3035 lr=5.07e-05 tok=17.326B dt=1768 wait=113 +09:21:01 pushed ckpt_w124.pt,log_w124.txt +09:21:31 waited 30s for global r=484 +09:21:39 global r=484 load 4.297 -> 4.286 +09:22:51 step 131720 loss=3.2676 lr=5.07e-05 tok=17.329B dt=1893 wait=114 +09:24:08 step 131740 loss=3.3206 lr=5.07e-05 tok=17.332B dt=1970 wait=114 +09:25:11 step 131760 loss=3.3251 lr=5.07e-05 tok=17.334B dt=2033 wait=114 +09:26:15 step 131780 loss=3.2878 lr=5.07e-05 tok=17.337B dt=2097 wait=115 +09:27:19 step 131800 loss=3.3178 lr=5.06e-05 tok=17.340B dt=2161 wait=115 +09:28:22 step 131820 loss=3.3911 lr=5.06e-05 tok=17.342B dt=2224 wait=115 +09:29:25 step 131840 loss=3.2405 lr=5.06e-05 tok=17.345B dt=2287 wait=116 +09:30:11 pushed ckpt_w124.pt,log_w124.txt +09:30:56 waited 45s for global r=485 +09:31:02 global r=485 load 4.324 -> 4.296 +09:31:35 step 131860 loss=3.4128 lr=5.06e-05 tok=17.347B dt=2417 wait=116 +09:32:47 step 131880 loss=3.2925 lr=5.06e-05 tok=17.350B dt=2489 wait=116 +09:34:03 step 131900 loss=3.2213 lr=5.05e-05 tok=17.353B dt=2564 wait=117 +09:35:07 step 131920 loss=3.3893 lr=5.05e-05 tok=17.355B dt=2628 wait=117 +09:36:10 step 131940 loss=3.2588 lr=5.05e-05 tok=17.358B dt=2692 wait=117 +09:37:14 step 131960 loss=3.3818 lr=5.05e-05 tok=17.360B dt=2756 wait=117 +09:38:17 step 131980 loss=3.3040 lr=5.05e-05 tok=17.363B dt=2819 wait=118 +09:39:21 step 132000 loss=3.3216 lr=5.04e-05 tok=17.366B dt=2883 wait=118 +09:39:34 pushed ckpt_w124.pt,log_w124.txt +09:40:20 waited 45s for global r=486 +09:40:26 global r=486 load 4.274 -> 4.263 +09:41:31 step 132020 loss=3.2855 lr=5.04e-05 tok=17.368B dt=3013 wait=118 +09:42:54 step 132040 loss=3.3361 lr=5.04e-05 tok=17.371B dt=3096 wait=119 +09:43:59 step 132060 loss=3.3447 lr=5.04e-05 tok=17.374B dt=3161 wait=119 +09:45:03 step 132080 loss=3.2944 lr=5.04e-05 tok=17.376B dt=3224 wait=119 +09:46:06 step 132100 loss=3.2874 lr=5.04e-05 tok=17.379B dt=3288 wait=120 +09:47:10 step 132120 loss=3.3957 lr=5.03e-05 tok=17.381B dt=3351 wait=120 +09:48:13 step 132140 loss=3.2913 lr=5.03e-05 tok=17.384B dt=3415 wait=120 +09:49:00 pushed ckpt_w124.pt,log_w124.txt +09:49:46 waited 45s for global r=487 +09:49:54 global r=487 load 4.278 -> 4.277 +09:50:27 step 132160 loss=3.3924 lr=5.03e-05 tok=17.387B dt=3549 wait=121 +09:51:37 step 132180 loss=3.2970 lr=5.03e-05 tok=17.389B dt=3619 wait=121 +09:52:54 step 132200 loss=3.3433 lr=5.03e-05 tok=17.392B dt=3695 wait=121 +09:53:57 step 132220 loss=3.3979 lr=5.02e-05 tok=17.395B dt=3759 wait=122 +09:55:00 step 132240 loss=3.2935 lr=5.02e-05 tok=17.397B dt=3822 wait=122 +09:56:04 step 132260 loss=3.3420 lr=5.02e-05 tok=17.400B dt=3886 wait=122 +09:57:07 step 132280 loss=3.2836 lr=5.02e-05 tok=17.402B dt=3949 wait=122 +09:58:10 step 132300 loss=3.3144 lr=5.02e-05 tok=17.405B dt=4012 wait=123 +09:58:22 pushed ckpt_w124.pt,log_w124.txt +09:59:08 waited 45s for global r=488 +09:59:15 global r=488 load 4.297 -> 4.281 +10:00:20 step 132320 loss=3.3410 lr=5.01e-05 tok=17.408B dt=4142 wait=123 +10:01:35 step 132340 loss=3.3468 lr=5.01e-05 tok=17.410B dt=4217 wait=123 +10:02:47 step 132360 loss=3.2824 lr=5.01e-05 tok=17.413B dt=4289 wait=124 +10:03:50 step 132380 loss=3.2539 lr=5.01e-05 tok=17.416B dt=4352 wait=124 +10:04:54 step 132400 loss=3.2529 lr=5.01e-05 tok=17.418B dt=4416 wait=124 +10:05:57 step 132420 loss=3.3808 lr=5.01e-05 tok=17.421B dt=4479 wait=125 +10:07:00 step 132440 loss=3.2224 lr=5.00e-05 tok=17.423B dt=4542 wait=125 +10:07:47 pushed ckpt_w124.pt,log_w124.txt +10:08:48 waited 61s for global r=489 +10:08:55 global r=489 load 4.317 -> 4.281 +10:09:28 step 132460 loss=3.3460 lr=5.00e-05 tok=17.426B dt=4690 wait=125 +10:10:39 step 132480 loss=3.2731 lr=5.00e-05 tok=17.429B dt=4760 wait=126 +10:11:55 step 132500 loss=3.3057 lr=5.00e-05 tok=17.431B dt=4837 wait=126 +10:12:59 step 132520 loss=3.3816 lr=5.00e-05 tok=17.434B dt=4901 wait=126 +10:14:02 step 132540 loss=3.2554 lr=4.99e-05 tok=17.437B dt=4964 wait=127 +10:15:06 step 132560 loss=3.4054 lr=4.99e-05 tok=17.439B dt=5028 wait=127 +10:16:09 step 132580 loss=3.3224 lr=4.99e-05 tok=17.442B dt=5091 wait=127 +10:17:13 step 132600 loss=3.2511 lr=4.99e-05 tok=17.444B dt=5155 wait=128 +10:17:29 pushed ckpt_w124.pt,log_w124.txt +10:18:15 waited 45s for global r=490 +10:18:22 global r=490 load 4.303 -> 4.287 +10:19:26 step 132620 loss=3.2300 lr=4.99e-05 tok=17.447B dt=5288 wait=128 +10:20:38 step 132640 loss=3.3570 lr=4.99e-05 tok=17.450B dt=5360 wait=128 +10:21:53 step 132660 loss=3.3234 lr=4.98e-05 tok=17.452B dt=5435 wait=128 +10:22:57 step 132680 loss=3.2356 lr=4.98e-05 tok=17.455B dt=5499 wait=129 +10:24:01 step 132700 loss=3.2822 lr=4.98e-05 tok=17.457B dt=5562 wait=129 +10:25:04 step 132720 loss=3.3417 lr=4.98e-05 tok=17.460B dt=5626 wait=129 +10:26:08 step 132740 loss=3.2514 lr=4.98e-05 tok=17.463B dt=5689 wait=130 +10:26:56 pushed ckpt_w124.pt,log_w124.txt +10:27:26 waited 30s for global r=491 +10:27:32 global r=491 load 4.320 -> 4.299 +10:28:06 step 132760 loss=3.3671 lr=4.97e-05 tok=17.465B dt=5807 wait=130 +10:29:09 step 132780 loss=3.2928 lr=4.97e-05 tok=17.468B dt=5871 wait=130 +10:30:26 step 132800 loss=3.2849 lr=4.97e-05 tok=17.471B dt=5948 wait=131 +10:31:37 step 132820 loss=3.3301 lr=4.97e-05 tok=17.473B dt=6019 wait=131 +10:32:40 step 132840 loss=3.2178 lr=4.97e-05 tok=17.476B dt=6082 wait=131 +10:33:44 step 132860 loss=3.3470 lr=4.96e-05 tok=17.478B dt=6145 wait=132 +10:34:47 step 132880 loss=3.2605 lr=4.96e-05 tok=17.481B dt=6209 wait=132 +10:35:51 step 132900 loss=3.3423 lr=4.96e-05 tok=17.484B dt=6273 wait=132 +10:36:03 pushed ckpt_w124.pt,log_w124.txt +10:36:48 waited 45s for global r=492 +10:36:54 global r=492 load 4.295 -> 4.280 +10:38:15 step 132920 loss=3.2711 lr=4.96e-05 tok=17.486B dt=6416 wait=133 +10:39:22 step 132940 loss=3.3329 lr=4.96e-05 tok=17.489B dt=6484 wait=133 +10:40:26 step 132960 loss=3.3347 lr=4.96e-05 tok=17.492B dt=6548 wait=133 +10:41:30 step 132980 loss=3.3140 lr=4.95e-05 tok=17.494B dt=6612 wait=134 +10:42:34 step 133000 loss=3.3085 lr=4.95e-05 tok=17.497B dt=6676 wait=134 +10:43:37 step 133020 loss=3.3987 lr=4.95e-05 tok=17.499B dt=6739 wait=134 +10:44:41 step 133040 loss=3.2973 lr=4.95e-05 tok=17.502B dt=6802 wait=134 +10:45:28 pushed ckpt_w124.pt,log_w124.txt +10:45:59 waited 30s for global r=493 +10:46:07 global r=493 load 4.285 -> 4.287 +10:46:40 step 133060 loss=3.3978 lr=4.95e-05 tok=17.505B dt=6922 wait=135 +10:48:03 step 133080 loss=3.3252 lr=4.94e-05 tok=17.507B dt=7005 wait=135 +10:49:08 step 133100 loss=3.3033 lr=4.94e-05 tok=17.510B dt=7070 wait=135 +10:50:11 step 133120 loss=3.3871 lr=4.94e-05 tok=17.513B dt=7133 wait=136 +10:51:15 step 133140 loss=3.2500 lr=4.94e-05 tok=17.515B dt=7197 wait=136 +10:52:18 step 133160 loss=3.4195 lr=4.94e-05 tok=17.518B dt=7260 wait=136 +10:53:22 step 133180 loss=3.3019 lr=4.94e-05 tok=17.520B dt=7323 wait=137 +10:54:25 step 133200 loss=3.3443 lr=4.93e-05 tok=17.523B dt=7387 wait=137 +10:54:39 pushed ckpt_w124.pt,log_w124.txt +10:55:24 waited 45s for global r=494 +10:55:32 global r=494 load 4.281 -> 4.283 +10:56:43 step 133220 loss=3.2675 lr=4.93e-05 tok=17.526B dt=7525 wait=137 +10:58:00 step 133240 loss=3.3718 lr=4.93e-05 tok=17.528B dt=7602 wait=138 +10:59:04 step 133260 loss=3.3535 lr=4.93e-05 tok=17.531B dt=7666 wait=138 +11:00:08 step 133280 loss=3.2619 lr=4.93e-05 tok=17.534B dt=7730 wait=138 +11:01:12 step 133300 loss=3.2602 lr=4.92e-05 tok=17.536B dt=7794 wait=139 +11:02:16 step 133320 loss=3.4145 lr=4.92e-05 tok=17.539B dt=7857 wait=139 +11:03:19 step 133340 loss=3.1912 lr=4.92e-05 tok=17.541B dt=7921 wait=139 +11:04:05 pushed ckpt_w124.pt,log_w124.txt +11:04:51 waited 45s for global r=495 +11:04:57 global r=495 load 4.291 -> 4.289 +11:05:30 step 133360 loss=3.3955 lr=4.92e-05 tok=17.544B dt=8052 wait=140 +11:06:42 step 133380 loss=3.3005 lr=4.92e-05 tok=17.547B dt=8123 wait=140 +11:07:58 step 133400 loss=3.1438 lr=4.91e-05 tok=17.549B dt=8200 wait=140 +11:09:01 step 133420 loss=3.3894 lr=4.91e-05 tok=17.552B dt=8263 wait=141 +11:10:05 step 133440 loss=3.2609 lr=4.91e-05 tok=17.554B dt=8327 wait=141 +11:11:09 step 133460 loss=3.2818 lr=4.91e-05 tok=17.557B dt=8391 wait=141 +11:12:12 step 133480 loss=3.2318 lr=4.91e-05 tok=17.560B dt=8454 wait=142 +11:13:16 step 133500 loss=3.3774 lr=4.91e-05 tok=17.562B dt=8518 wait=142 +11:13:30 pushed ckpt_w124.pt,log_w124.txt +11:14:00 waited 30s for global r=496 +11:14:10 global r=496 load 4.298 -> 4.296 +11:15:15 step 133520 loss=3.2297 lr=4.90e-05 tok=17.565B dt=8637 wait=142 +11:16:33 step 133540 loss=3.3588 lr=4.90e-05 tok=17.568B dt=8715 wait=142 +11:17:43 step 133560 loss=3.3176 lr=4.90e-05 tok=17.570B dt=8784 wait=143 +11:18:46 step 133580 loss=3.2989 lr=4.90e-05 tok=17.573B dt=8848 wait=143 +11:19:50 step 133600 loss=3.2927 lr=4.90e-05 tok=17.575B dt=8912 wait=143 +11:20:53 step 133620 loss=3.4132 lr=4.89e-05 tok=17.578B dt=8975 wait=144 +11:21:57 step 133640 loss=3.2614 lr=4.89e-05 tok=17.581B dt=9039 wait=144 +11:22:41 pushed ckpt_w124.pt,log_w124.txt +11:23:27 waited 45s for global r=497 +11:23:35 global r=497 load 4.302 -> 4.298 +11:24:08 step 133660 loss=3.3356 lr=4.89e-05 tok=17.583B dt=9170 wait=144 +11:25:11 step 133680 loss=3.2971 lr=4.89e-05 tok=17.586B dt=9233 wait=145 +11:26:35 step 133700 loss=3.2314 lr=4.89e-05 tok=17.589B dt=9317 wait=145 +11:27:39 step 133720 loss=3.3637 lr=4.89e-05 tok=17.591B dt=9381 wait=145 +11:28:42 step 133740 loss=3.3158 lr=4.88e-05 tok=17.594B dt=9444 wait=146 +11:29:46 step 133760 loss=3.4123 lr=4.88e-05 tok=17.596B dt=9508 wait=146 +11:30:50 step 133780 loss=3.3252 lr=4.88e-05 tok=17.599B dt=9571 wait=146 +11:31:53 step 133800 loss=3.3276 lr=4.88e-05 tok=17.602B dt=9635 wait=147 +11:32:05 pushed ckpt_w124.pt,log_w124.txt +11:32:35 waited 30s for global r=498 +11:32:43 global r=498 load 4.298 -> 4.295 +11:34:10 step 133820 loss=3.2766 lr=4.88e-05 tok=17.604B dt=9772 wait=147 +11:35:15 step 133840 loss=3.3062 lr=4.87e-05 tok=17.607B dt=9837 wait=147 +11:36:19 step 133860 loss=3.3378 lr=4.87e-05 tok=17.610B dt=9901 wait=147 +11:37:23 step 133880 loss=3.3096 lr=4.87e-05 tok=17.612B dt=9965 wait=148 +11:38:26 step 133900 loss=3.2335 lr=4.87e-05 tok=17.615B dt=10028 wait=148 +11:39:31 step 133920 loss=3.3811 lr=4.87e-05 tok=17.617B dt=10092 wait=148 +11:40:34 step 133940 loss=3.2890 lr=4.87e-05 tok=17.620B dt=10156 wait=149 +11:41:18 pushed ckpt_w124.pt,log_w124.txt +11:42:03 waited 45s for global r=499 +11:42:10 global r=499 load 4.282 -> 4.283 +11:42:50 step 133960 loss=3.3444 lr=4.86e-05 tok=17.623B dt=10292 wait=149 +11:44:06 step 133980 loss=3.2460 lr=4.86e-05 tok=17.625B dt=10368 wait=149 +11:45:10 step 134000 loss=3.2398 lr=4.86e-05 tok=17.628B dt=10432 wait=150 +11:46:13 step 134020 loss=3.3987 lr=4.86e-05 tok=17.630B dt=10495 wait=150 +11:47:17 step 134040 loss=3.2939 lr=4.86e-05 tok=17.633B dt=10559 wait=150 +11:48:20 step 134060 loss=3.3700 lr=4.85e-05 tok=17.636B dt=10622 wait=151 +11:49:24 step 134080 loss=3.2849 lr=4.85e-05 tok=17.638B dt=10686 wait=151 +11:50:28 step 134100 loss=3.3352 lr=4.85e-05 tok=17.641B dt=10750 wait=151 +11:50:41 pushed ckpt_w124.pt,log_w124.txt +11:54:48 step 134120 loss=3.3758 lr=4.85e-05 tok=17.644B dt=11010 wait=152 +11:55:51 step 134140 loss=3.3132 lr=4.85e-05 tok=17.646B dt=11073 wait=152 +11:56:54 step 134160 loss=3.2984 lr=4.85e-05 tok=17.649B dt=11136 wait=152 +11:57:58 step 134180 loss=3.2549 lr=4.84e-05 tok=17.651B dt=11200 wait=153 +11:59:01 step 134200 loss=3.1720 lr=4.84e-05 tok=17.654B dt=11263 wait=153 +12:00:04 step 134220 loss=3.3714 lr=4.84e-05 tok=17.657B dt=11326 wait=153 +12:01:08 step 134240 loss=3.3035 lr=4.84e-05 tok=17.659B dt=11389 wait=153 +12:01:52 pushed ckpt_w124.pt,log_w124.txt +12:05:26 step 134260 loss=3.3105 lr=4.84e-05 tok=17.662B dt=11648 wait=154 +12:06:29 step 134280 loss=3.3786 lr=4.83e-05 tok=17.665B dt=11711 wait=154 +12:07:33 step 134300 loss=3.3268 lr=4.83e-05 tok=17.667B dt=11775 wait=154 +12:08:37 step 134320 loss=3.3841 lr=4.83e-05 tok=17.670B dt=11839 wait=155 +12:09:41 step 134340 loss=3.3450 lr=4.83e-05 tok=17.672B dt=11903 wait=155 +launch +12:10:28 name=w124 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 71/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:10:32 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:10:32 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:10:42 params 1.08B +12:10:43 resumed step=133950 seen=17.62B round=499 +12:10:43 stream salt=133950 +12:12:50 blocks compiled +12:12:50 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:16:24 step 133951 loss=3.0691 lr=4.86e-05 tok=17.621B dt=214 wait=193 +12:16:51 step 133952 loss=3.1970 lr=4.86e-05 tok=17.622B dt=240 wait=193 +12:17:18 step 133960 loss=3.5686 lr=4.86e-05 tok=17.623B dt=268 wait=193 +12:18:29 step 133980 loss=3.3921 lr=4.86e-05 tok=17.625B dt=338 wait=194 +12:19:33 step 134000 loss=3.4215 lr=4.86e-05 tok=17.628B dt=402 wait=194 +12:20:36 step 134020 loss=3.5041 lr=4.86e-05 tok=17.630B dt=466 wait=195 +12:21:40 step 134040 loss=3.4838 lr=4.86e-05 tok=17.633B dt=529 wait=195 +12:22:44 step 134060 loss=3.5946 lr=4.85e-05 tok=17.636B dt=594 wait=195 +12:23:48 step 134080 loss=3.4384 lr=4.85e-05 tok=17.638B dt=657 wait=196 +12:24:51 step 134100 loss=3.4215 lr=4.85e-05 tok=17.641B dt=721 wait=196 +12:25:08 pushed ckpt_w124.pt,log_w124.txt +12:29:17 step 134120 loss=3.5893 lr=4.85e-05 tok=17.644B dt=986 wait=196 +12:30:21 step 134140 loss=3.3668 lr=4.85e-05 tok=17.646B dt=1050 wait=197 +12:31:23 step 134160 loss=3.4839 lr=4.85e-05 tok=17.649B dt=1113 wait=197 +12:32:27 step 134180 loss=3.4263 lr=4.84e-05 tok=17.651B dt=1177 wait=197 +12:33:30 step 134200 loss=3.3739 lr=4.84e-05 tok=17.654B dt=1240 wait=198 +12:34:33 step 134220 loss=3.3647 lr=4.84e-05 tok=17.657B dt=1303 wait=198 +12:35:27 global r=500 load 4.293 -> 4.281 +12:36:39 step 134120 loss=3.3858 lr=4.85e-05 tok=17.661B dt=1428 wait=199 +12:37:54 step 134140 loss=3.4219 lr=4.85e-05 tok=17.663B dt=1504 wait=199 +12:38:57 step 134160 loss=3.4477 lr=4.85e-05 tok=17.666B dt=1567 wait=199 +12:40:01 step 134180 loss=3.4891 lr=4.84e-05 tok=17.669B dt=1630 wait=200 +12:41:04 step 134200 loss=3.3166 lr=4.84e-05 tok=17.671B dt=1694 wait=200 +12:42:07 step 134220 loss=3.5179 lr=4.84e-05 tok=17.674B dt=1757 wait=200 +12:43:11 step 134240 loss=3.3025 lr=4.84e-05 tok=17.676B dt=1821 wait=201 +12:44:07 pushed ckpt_w124.pt,log_w124.txt +12:44:52 waited 46s for global r=501 +12:44:59 global r=501 load 4.292 -> 4.276 +12:45:33 step 134260 loss=3.4827 lr=4.84e-05 tok=17.679B dt=1962 wait=201 +12:46:51 step 134280 loss=3.4032 lr=4.83e-05 tok=17.682B dt=2040 wait=201 +12:47:58 step 134300 loss=3.3694 lr=4.83e-05 tok=17.684B dt=2108 wait=202 +12:49:02 step 134320 loss=3.4571 lr=4.83e-05 tok=17.687B dt=2172 wait=202 +12:50:06 step 134340 loss=3.3950 lr=4.83e-05 tok=17.689B dt=2235 wait=202 +12:51:10 step 134360 loss=3.4612 lr=4.83e-05 tok=17.692B dt=2300 wait=203 +12:52:14 step 134380 loss=3.4295 lr=4.83e-05 tok=17.695B dt=2363 wait=203 +12:53:18 step 134400 loss=3.4344 lr=4.82e-05 tok=17.697B dt=2427 wait=203 +12:53:36 pushed ckpt_w124.pt,log_w124.txt +12:54:06 waited 31s for global r=502 +12:54:17 global r=502 load 4.280 -> 4.274 +12:55:22 step 134420 loss=3.3993 lr=4.82e-05 tok=17.700B dt=2552 wait=204 +12:56:45 step 134440 loss=3.5103 lr=4.82e-05 tok=17.703B dt=2635 wait=204 +12:57:49 step 134460 loss=3.5321 lr=4.82e-05 tok=17.705B dt=2699 wait=204 +12:58:52 step 134480 loss=3.4045 lr=4.82e-05 tok=17.708B dt=2762 wait=205 +12:59:55 step 134500 loss=3.4217 lr=4.81e-05 tok=17.710B dt=2825 wait=205 +13:00:59 step 134520 loss=3.5036 lr=4.81e-05 tok=17.713B dt=2888 wait=205 +13:02:03 step 134540 loss=3.3338 lr=4.81e-05 tok=17.716B dt=2952 wait=206 +13:02:48 pushed ckpt_w124.pt,log_w124.txt +13:05:34 waited 167s for global r=503 +13:05:44 global r=503 load 4.282 -> 4.283 +13:06:17 step 134560 loss=3.4319 lr=4.81e-05 tok=17.718B dt=3207 wait=206 +13:07:33 step 134580 loss=3.3853 lr=4.81e-05 tok=17.721B dt=3283 wait=206 +13:08:48 step 134600 loss=3.3330 lr=4.80e-05 tok=17.724B dt=3358 wait=207 +13:09:52 step 134620 loss=3.5018 lr=4.80e-05 tok=17.726B dt=3422 wait=207 +13:10:56 step 134640 loss=3.3480 lr=4.80e-05 tok=17.729B dt=3485 wait=207 +13:11:59 step 134660 loss=3.4961 lr=4.80e-05 tok=17.731B dt=3549 wait=207 +13:13:03 step 134680 loss=3.4750 lr=4.80e-05 tok=17.734B dt=3612 wait=208 +13:14:07 step 134700 loss=3.4580 lr=4.80e-05 tok=17.737B dt=3676 wait=208 +13:14:20 pushed ckpt_w124.pt,log_w124.txt +13:15:20 waited 61s for global r=504 +13:15:29 global r=504 load 4.288 -> 4.279 +13:16:34 step 134720 loss=3.4802 lr=4.79e-05 tok=17.739B dt=3824 wait=209 +13:17:57 step 134740 loss=3.4374 lr=4.79e-05 tok=17.742B dt=3907 wait=209 +13:19:00 step 134760 loss=3.4521 lr=4.79e-05 tok=17.745B dt=3970 wait=209 +13:20:04 step 134780 loss=3.3793 lr=4.79e-05 tok=17.747B dt=4034 wait=209 +13:21:07 step 134800 loss=3.4316 lr=4.79e-05 tok=17.750B dt=4097 wait=210 +13:22:12 step 134820 loss=3.5151 lr=4.78e-05 tok=17.752B dt=4162 wait=210 +13:23:15 step 134840 loss=3.3946 lr=4.78e-05 tok=17.755B dt=4225 wait=210 +13:24:04 pushed ckpt_w124.pt,log_w124.txt +13:24:49 waited 46s for global r=505 +13:24:55 global r=505 load 4.273 -> 4.282 +13:25:29 step 134860 loss=3.5086 lr=4.78e-05 tok=17.758B dt=4358 wait=211 +13:26:40 step 134880 loss=3.3855 lr=4.78e-05 tok=17.760B dt=4430 wait=211 +13:27:55 step 134900 loss=3.4139 lr=4.78e-05 tok=17.763B dt=4505 wait=211 +13:28:58 step 134920 loss=3.4796 lr=4.78e-05 tok=17.765B dt=4568 wait=212 +13:30:01 step 134940 loss=3.3147 lr=4.77e-05 tok=17.768B dt=4631 wait=212 +13:31:04 step 134960 loss=3.4010 lr=4.77e-05 tok=17.771B dt=4694 wait=212 +13:32:07 step 134980 loss=3.3606 lr=4.77e-05 tok=17.773B dt=4757 wait=213 +13:33:11 step 135000 loss=3.3345 lr=4.77e-05 tok=17.776B dt=4821 wait=213 +13:33:37 pushed ckpt_w124.pt,log_w124.txt +13:34:22 waited 46s for global r=506 +13:34:29 global r=506 load 4.298 -> 4.282 +13:35:34 step 135020 loss=3.3483 lr=4.77e-05 tok=17.779B dt=4964 wait=213 +13:37:10 step 135040 loss=3.3885 lr=4.76e-05 tok=17.781B dt=5060 wait=214 +13:38:13 step 135060 loss=3.4576 lr=4.76e-05 tok=17.784B dt=5123 wait=214 +13:39:17 step 135080 loss=3.3308 lr=4.76e-05 tok=17.786B dt=5186 wait=214 +13:40:20 step 135100 loss=3.3192 lr=4.76e-05 tok=17.789B dt=5250 wait=214 +13:41:23 step 135120 loss=3.4507 lr=4.76e-05 tok=17.792B dt=5313 wait=215 +13:42:26 step 135140 loss=3.3722 lr=4.76e-05 tok=17.794B dt=5375 wait=215 +13:43:14 pushed ckpt_w124.pt,log_w124.txt +13:45:45 waited 151s for global r=507 +13:45:55 global r=507 load 4.289 -> 4.282 +13:46:27 step 135160 loss=3.4353 lr=4.75e-05 tok=17.797B dt=5617 wait=215 +13:47:31 step 135180 loss=3.2288 lr=4.75e-05 tok=17.800B dt=5680 wait=216 +13:48:53 step 135200 loss=3.3261 lr=4.75e-05 tok=17.802B dt=5762 wait=216 +13:49:57 step 135220 loss=3.3388 lr=4.75e-05 tok=17.805B dt=5826 wait=216 +13:50:59 step 135240 loss=3.2786 lr=4.75e-05 tok=17.807B dt=5889 wait=217 +13:52:03 step 135260 loss=3.3472 lr=4.74e-05 tok=17.810B dt=5952 wait=217 +13:53:05 step 135280 loss=3.3602 lr=4.74e-05 tok=17.813B dt=6015 wait=217 +13:54:08 step 135300 loss=3.3101 lr=4.74e-05 tok=17.815B dt=6077 wait=218 +13:54:26 pushed ckpt_w124.pt,log_w124.txt +13:55:58 waited 92s for global r=508 +13:56:07 global r=508 load 4.312 -> 4.283 +13:57:11 step 135320 loss=3.3354 lr=4.74e-05 tok=17.818B dt=6261 wait=218 +13:58:34 step 135340 loss=3.4086 lr=4.74e-05 tok=17.821B dt=6344 wait=218 +13:59:38 step 135360 loss=3.3932 lr=4.74e-05 tok=17.823B dt=6407 wait=219 +14:00:40 step 135380 loss=3.3730 lr=4.73e-05 tok=17.826B dt=6470 wait=219 +14:01:43 step 135400 loss=3.4242 lr=4.73e-05 tok=17.828B dt=6533 wait=219 +14:02:46 step 135420 loss=3.4668 lr=4.73e-05 tok=17.831B dt=6596 wait=219 +14:03:49 step 135440 loss=3.3681 lr=4.73e-05 tok=17.834B dt=6658 wait=220 +14:04:39 pushed ckpt_w124.pt,log_w124.txt +14:05:09 waited 30s for global r=509 +14:05:25 global r=509 load 4.280 -> 4.282 +14:05:58 step 135460 loss=3.3563 lr=4.73e-05 tok=17.836B dt=6788 wait=220 +14:07:07 step 135480 loss=3.3818 lr=4.73e-05 tok=17.839B dt=6857 wait=220 +14:08:23 step 135500 loss=3.3526 lr=4.72e-05 tok=17.842B dt=6933 wait=221 +14:09:26 step 135520 loss=3.4405 lr=4.72e-05 tok=17.844B dt=6996 wait=222 +14:10:29 step 135540 loss=3.2902 lr=4.72e-05 tok=17.847B dt=7059 wait=222 +14:11:33 step 135560 loss=3.3434 lr=4.72e-05 tok=17.849B dt=7123 wait=222 +14:12:36 step 135580 loss=3.3718 lr=4.72e-05 tok=17.852B dt=7185 wait=222 +14:13:39 step 135600 loss=3.3613 lr=4.71e-05 tok=17.855B dt=7248 wait=223 +14:13:57 pushed ckpt_w124.pt,log_w124.txt +14:14:58 waited 61s for global r=510 +14:15:11 global r=510 load 4.315 -> 4.291 +14:16:15 step 135620 loss=3.3232 lr=4.71e-05 tok=17.857B dt=7405 wait=223 +14:17:36 step 135640 loss=3.3650 lr=4.71e-05 tok=17.860B dt=7485 wait=223 +14:18:42 step 135660 loss=3.3759 lr=4.71e-05 tok=17.862B dt=7551 wait=224 +14:19:45 step 135680 loss=3.2785 lr=4.71e-05 tok=17.865B dt=7614 wait=224 +14:20:47 step 135700 loss=3.4130 lr=4.71e-05 tok=17.868B dt=7677 wait=224 +14:21:51 step 135720 loss=3.4098 lr=4.70e-05 tok=17.870B dt=7740 wait=225 +14:22:53 step 135740 loss=3.3848 lr=4.70e-05 tok=17.873B dt=7803 wait=225 +14:23:42 pushed ckpt_w124.pt,log_w124.txt +14:24:28 waited 46s for global r=511 +14:24:37 global r=511 load 4.308 -> 4.289 +14:25:10 step 135760 loss=3.4764 lr=4.70e-05 tok=17.876B dt=7940 wait=225 +14:26:19 step 135780 loss=3.3904 lr=4.70e-05 tok=17.878B dt=8009 wait=226 +14:27:36 step 135800 loss=3.4553 lr=4.70e-05 tok=17.881B dt=8085 wait=226 +14:28:38 step 135820 loss=3.5185 lr=4.69e-05 tok=17.883B dt=8148 wait=226 +14:29:41 step 135840 loss=3.3011 lr=4.69e-05 tok=17.886B dt=8211 wait=226 +14:30:44 step 135860 loss=3.4021 lr=4.69e-05 tok=17.889B dt=8274 wait=227 +14:31:47 step 135880 loss=3.3864 lr=4.69e-05 tok=17.891B dt=8336 wait=227 +14:32:49 step 135900 loss=3.4187 lr=4.69e-05 tok=17.894B dt=8399 wait=227 +14:33:09 pushed ckpt_w124.pt,log_w124.txt +14:33:55 waited 46s for global r=512 +14:34:05 global r=512 load 4.298 -> 4.296 +14:35:09 step 135920 loss=3.3340 lr=4.69e-05 tok=17.897B dt=8539 wait=228 +14:36:20 step 135940 loss=3.2873 lr=4.68e-05 tok=17.899B dt=8610 wait=228 +14:37:35 step 135960 loss=3.3288 lr=4.68e-05 tok=17.902B dt=8685 wait=228 +14:38:38 step 135980 loss=3.3236 lr=4.68e-05 tok=17.904B dt=8748 wait=229 +14:39:41 step 136000 loss=3.3751 lr=4.68e-05 tok=17.907B dt=8811 wait=229 +14:40:45 step 136020 loss=3.4260 lr=4.68e-05 tok=17.910B dt=8874 wait=229 +14:41:47 step 136040 loss=3.3993 lr=4.67e-05 tok=17.912B dt=8937 wait=230 +14:42:42 pushed ckpt_w124.pt,log_w124.txt +14:43:28 waited 46s for global r=513 +14:43:37 global r=513 load 4.302 -> 4.292 +14:44:10 step 136060 loss=3.4321 lr=4.67e-05 tok=17.915B dt=9080 wait=230 +14:45:13 step 136080 loss=3.3330 lr=4.67e-05 tok=17.918B dt=9143 wait=230 +14:46:37 step 136100 loss=3.3231 lr=4.67e-05 tok=17.920B dt=9226 wait=230 +14:47:39 step 136120 loss=3.4073 lr=4.67e-05 tok=17.923B dt=9289 wait=231 +14:48:43 step 136140 loss=3.2888 lr=4.67e-05 tok=17.925B dt=9352 wait=231 +14:49:45 step 136160 loss=3.4371 lr=4.66e-05 tok=17.928B dt=9415 wait=231 +14:50:48 step 136180 loss=3.3456 lr=4.66e-05 tok=17.931B dt=9478 wait=232 +14:51:51 step 136200 loss=3.3625 lr=4.66e-05 tok=17.933B dt=9541 wait=232 +14:52:14 pushed ckpt_w124.pt,log_w124.txt +14:52:59 waited 45s for global r=514 +14:53:08 global r=514 load 4.328 -> 4.295 +14:54:12 step 136220 loss=3.3152 lr=4.66e-05 tok=17.936B dt=9682 wait=232 +14:55:24 step 136240 loss=3.3513 lr=4.66e-05 tok=17.939B dt=9753 wait=233 +14:56:40 step 136260 loss=3.4266 lr=4.65e-05 tok=17.941B dt=9829 wait=233 +14:57:43 step 136280 loss=3.2838 lr=4.65e-05 tok=17.944B dt=9893 wait=233 +14:58:46 step 136300 loss=3.3959 lr=4.65e-05 tok=17.946B dt=9956 wait=234 +14:59:49 step 136320 loss=3.4585 lr=4.65e-05 tok=17.949B dt=10018 wait=234 +15:00:51 step 136340 loss=3.3160 lr=4.65e-05 tok=17.952B dt=10081 wait=234 +15:01:39 pushed ckpt_w124.pt,log_w124.txt +15:02:26 waited 47s for global r=515 +15:02:35 global r=515 load 4.302 -> 4.297 +15:03:08 step 136360 loss=3.4390 lr=4.65e-05 tok=17.954B dt=10218 wait=235 +15:04:11 step 136380 loss=3.3232 lr=4.64e-05 tok=17.957B dt=10281 wait=235 +15:05:28 step 136400 loss=3.3702 lr=4.64e-05 tok=17.959B dt=10357 wait=235 +15:06:36 step 136420 loss=3.4482 lr=4.64e-05 tok=17.962B dt=10426 wait=236 +15:07:39 step 136440 loss=3.3821 lr=4.64e-05 tok=17.965B dt=10489 wait=236 +15:08:42 step 136460 loss=3.3458 lr=4.64e-05 tok=17.967B dt=10552 wait=236 +15:09:45 step 136480 loss=3.3714 lr=4.64e-05 tok=17.970B dt=10615 wait=237 +15:10:48 step 136500 loss=3.3723 lr=4.63e-05 tok=17.973B dt=10678 wait=237 +15:11:08 pushed ckpt_w124.pt,log_w124.txt +15:11:55 waited 47s for global r=516 +15:12:07 global r=516 load 4.300 -> 4.298 +15:13:31 step 136520 loss=3.3622 lr=4.63e-05 tok=17.975B dt=10840 wait=237 +15:14:34 step 136540 loss=3.3976 lr=4.63e-05 tok=17.978B dt=10903 wait=238 +15:15:37 step 136560 loss=3.3220 lr=4.63e-05 tok=17.980B dt=10966 wait=238 +15:16:39 step 136580 loss=3.3396 lr=4.63e-05 tok=17.983B dt=11029 wait=238 +15:17:42 step 136600 loss=3.3873 lr=4.62e-05 tok=17.986B dt=11092 wait=238 +15:18:45 step 136620 loss=3.4444 lr=4.62e-05 tok=17.988B dt=11155 wait=239 +15:19:48 step 136640 loss=3.3231 lr=4.62e-05 tok=17.991B dt=11218 wait=239 +15:20:39 pushed ckpt_w124.pt,log_w124.txt +15:21:24 waited 45s for global r=517 +15:21:35 global r=517 load 4.306 -> 4.295 +15:22:18 step 136660 loss=3.4910 lr=4.62e-05 tok=17.994B dt=11368 wait=239 +15:23:33 step 136680 loss=3.3543 lr=4.62e-05 tok=17.996B dt=11443 wait=240 +15:24:36 step 136700 loss=3.3654 lr=4.62e-05 tok=17.999B dt=11506 wait=240 +15:25:39 step 136720 loss=3.4381 lr=4.61e-05 tok=18.001B dt=11569 wait=240 +15:26:42 step 136740 loss=3.2952 lr=4.61e-05 tok=18.004B dt=11632 wait=241 +15:27:45 step 136760 loss=3.4681 lr=4.61e-05 tok=18.007B dt=11694 wait=241 +15:28:48 step 136780 loss=3.3091 lr=4.61e-05 tok=18.009B dt=11758 wait=241 +15:29:51 step 136800 loss=3.3643 lr=4.61e-05 tok=18.012B dt=11820 wait=242 +15:30:10 pushed ckpt_w124.pt,log_w124.txt +15:30:55 waited 45s for global r=518 +15:31:09 global r=518 load 4.289 -> 4.283 +15:32:29 step 136820 loss=3.3372 lr=4.60e-05 tok=18.015B dt=11979 wait=242 +15:33:37 step 136840 loss=3.4039 lr=4.60e-05 tok=18.017B dt=12047 wait=242 +15:34:40 step 136860 loss=3.3391 lr=4.60e-05 tok=18.020B dt=12110 wait=243 +15:35:43 step 136880 loss=3.3126 lr=4.60e-05 tok=18.022B dt=12173 wait=243 +15:36:46 step 136900 loss=3.3738 lr=4.60e-05 tok=18.025B dt=12235 wait=243 +15:37:48 step 136920 loss=3.5105 lr=4.60e-05 tok=18.028B dt=12298 wait=244 +15:38:51 step 136940 loss=3.2868 lr=4.59e-05 tok=18.030B dt=12360 wait=244 +15:39:40 pushed ckpt_w124.pt,log_w124.txt +15:40:28 waited 48s for global r=519 +15:40:38 global r=519 load 4.277 -> 4.288 +15:41:16 step 136960 loss=3.4580 lr=4.59e-05 tok=18.033B dt=12506 wait=244 +15:42:33 step 136980 loss=3.3304 lr=4.59e-05 tok=18.036B dt=12583 wait=245 +15:43:36 step 137000 loss=3.2803 lr=4.59e-05 tok=18.038B dt=12646 wait=245 +15:44:39 step 137020 loss=3.3525 lr=4.59e-05 tok=18.041B dt=12709 wait=245 +15:45:41 step 137040 loss=3.3292 lr=4.59e-05 tok=18.043B dt=12771 wait=245 +15:46:44 step 137060 loss=3.3967 lr=4.58e-05 tok=18.046B dt=12834 wait=246 +15:47:47 step 137080 loss=3.2235 lr=4.58e-05 tok=18.049B dt=12897 wait=246 +15:48:51 step 137100 loss=3.4190 lr=4.58e-05 tok=18.051B dt=12960 wait=246 diff --git a/log_w125.txt b/log_w125.txt new file mode 100644 index 0000000000000000000000000000000000000000..5c7816de7550a76855d8d36ab66c2d6f558e8867 --- /dev/null +++ b/log_w125.txt @@ -0,0 +1,197 @@ +launch +08:49:21 name=w125 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 72/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +08:49:27 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +08:49:27 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +08:49:37 params 1.08B +08:49:38 resumed step=131250 seen=17.27B round=481 +08:49:38 stream salt=131250 +08:51:19 blocks compiled +08:51:19 start: accum=4, 131072 tokens per optimiser step +08:53:25 step 131251 loss=3.0930 lr=5.11e-05 tok=17.268B dt=126 wait=81 +08:53:28 step 131252 loss=3.1234 lr=5.11e-05 tok=17.268B dt=129 wait=81 +08:53:56 step 131260 loss=3.4399 lr=5.11e-05 tok=17.269B dt=157 wait=81 +08:55:07 step 131280 loss=3.3533 lr=5.11e-05 tok=17.271B dt=228 wait=82 +08:56:11 step 131300 loss=3.4415 lr=5.11e-05 tok=17.274B dt=292 wait=82 +08:57:15 step 131320 loss=3.3992 lr=5.11e-05 tok=17.277B dt=356 wait=82 +08:58:19 step 131340 loss=3.4373 lr=5.11e-05 tok=17.279B dt=420 wait=83 +08:59:24 step 131360 loss=3.5330 lr=5.10e-05 tok=17.282B dt=484 wait=83 +09:00:27 step 131380 loss=3.3932 lr=5.10e-05 tok=17.284B dt=548 wait=83 +09:01:31 step 131400 loss=3.4643 lr=5.10e-05 tok=17.287B dt=612 wait=84 +09:01:48 pushed ckpt_w125.pt,log_w125.txt +09:02:48 waited 61s for global r=482 +09:03:01 global r=482 load 4.284 -> 4.279 +09:04:08 step 131420 loss=3.3284 lr=5.10e-05 tok=17.290B dt=769 wait=84 +09:05:12 step 131440 loss=3.3235 lr=5.10e-05 tok=17.292B dt=833 wait=84 +09:06:16 step 131460 loss=3.3059 lr=5.10e-05 tok=17.295B dt=897 wait=85 +09:07:20 step 131480 loss=3.3102 lr=5.09e-05 tok=17.298B dt=961 wait=85 +09:08:24 step 131500 loss=3.3121 lr=5.09e-05 tok=17.300B dt=1024 wait=85 +09:09:28 step 131520 loss=3.3870 lr=5.09e-05 tok=17.303B dt=1089 wait=86 +09:10:32 step 131540 loss=3.1930 lr=5.09e-05 tok=17.305B dt=1153 wait=86 +09:11:18 pushed ckpt_w125.pt,log_w125.txt +09:12:18 waited 60s for global r=483 +09:12:25 global r=483 load 4.279 -> 4.282 +09:13:00 step 131560 loss=3.4358 lr=5.09e-05 tok=17.308B dt=1300 wait=86 +09:14:03 step 131580 loss=3.3280 lr=5.08e-05 tok=17.311B dt=1364 wait=86 +09:15:08 step 131600 loss=3.3291 lr=5.08e-05 tok=17.313B dt=1429 wait=87 +09:16:12 step 131620 loss=3.4658 lr=5.08e-05 tok=17.316B dt=1492 wait=87 +09:17:15 step 131640 loss=3.3106 lr=5.08e-05 tok=17.319B dt=1556 wait=87 +09:18:19 step 131660 loss=3.3911 lr=5.08e-05 tok=17.321B dt=1620 wait=88 +09:19:22 step 131680 loss=3.3609 lr=5.07e-05 tok=17.324B dt=1683 wait=88 +09:20:27 step 131700 loss=3.4387 lr=5.07e-05 tok=17.326B dt=1747 wait=88 +09:20:39 pushed ckpt_w125.pt,log_w125.txt +09:21:40 waited 60s for global r=484 +09:21:47 global r=484 load 4.266 -> 4.286 +09:22:52 step 131720 loss=3.3345 lr=5.07e-05 tok=17.329B dt=1893 wait=89 +09:23:56 step 131740 loss=3.3347 lr=5.07e-05 tok=17.332B dt=1957 wait=89 +09:25:00 step 131760 loss=3.3858 lr=5.07e-05 tok=17.334B dt=2020 wait=89 +09:26:04 step 131780 loss=3.3504 lr=5.07e-05 tok=17.337B dt=2084 wait=90 +09:27:08 step 131800 loss=3.4019 lr=5.06e-05 tok=17.340B dt=2148 wait=90 +09:28:12 step 131820 loss=3.3853 lr=5.06e-05 tok=17.342B dt=2212 wait=90 +09:29:15 step 131840 loss=3.2965 lr=5.06e-05 tok=17.345B dt=2276 wait=91 +09:29:59 pushed ckpt_w125.pt,log_w125.txt +09:31:00 waited 60s for global r=485 +09:31:07 global r=485 load 4.290 -> 4.296 +09:31:41 step 131860 loss=3.4756 lr=5.06e-05 tok=17.347B dt=2422 wait=91 +09:32:45 step 131880 loss=3.3509 lr=5.06e-05 tok=17.350B dt=2486 wait=91 +09:33:49 step 131900 loss=3.3691 lr=5.05e-05 tok=17.353B dt=2550 wait=92 +09:34:53 step 131920 loss=3.4284 lr=5.05e-05 tok=17.355B dt=2614 wait=92 +09:35:57 step 131940 loss=3.3212 lr=5.05e-05 tok=17.358B dt=2678 wait=92 +09:37:02 step 131960 loss=3.3772 lr=5.05e-05 tok=17.360B dt=2743 wait=93 +09:38:06 step 131980 loss=3.3847 lr=5.05e-05 tok=17.363B dt=2807 wait=93 +09:39:10 step 132000 loss=3.4206 lr=5.04e-05 tok=17.366B dt=2871 wait=93 +09:39:22 pushed ckpt_w125.pt,log_w125.txt +09:40:23 waited 61s for global r=486 +09:40:30 global r=486 load 4.268 -> 4.263 +09:41:35 step 132020 loss=3.3749 lr=5.04e-05 tok=17.368B dt=3016 wait=94 +09:42:39 step 132040 loss=3.4060 lr=5.04e-05 tok=17.371B dt=3080 wait=94 +09:43:43 step 132060 loss=3.3768 lr=5.04e-05 tok=17.374B dt=3144 wait=94 +09:44:48 step 132080 loss=3.3316 lr=5.04e-05 tok=17.376B dt=3208 wait=95 +09:45:52 step 132100 loss=3.2940 lr=5.04e-05 tok=17.379B dt=3273 wait=95 +09:46:57 step 132120 loss=3.4298 lr=5.03e-05 tok=17.381B dt=3338 wait=95 +09:48:01 step 132140 loss=3.3465 lr=5.03e-05 tok=17.384B dt=3402 wait=95 +09:48:47 pushed ckpt_w125.pt,log_w125.txt +09:49:48 waited 61s for global r=487 +09:49:57 global r=487 load 4.294 -> 4.277 +09:50:30 step 132160 loss=3.4614 lr=5.03e-05 tok=17.387B dt=3551 wait=96 +09:51:34 step 132180 loss=3.3293 lr=5.03e-05 tok=17.389B dt=3615 wait=96 +09:52:39 step 132200 loss=3.3218 lr=5.03e-05 tok=17.392B dt=3679 wait=97 +09:53:43 step 132220 loss=3.4384 lr=5.02e-05 tok=17.395B dt=3744 wait=97 +09:54:47 step 132240 loss=3.3602 lr=5.02e-05 tok=17.397B dt=3808 wait=97 +09:55:51 step 132260 loss=3.4198 lr=5.02e-05 tok=17.400B dt=3871 wait=97 +09:56:54 step 132280 loss=3.4017 lr=5.02e-05 tok=17.402B dt=3935 wait=98 +09:57:58 step 132300 loss=3.3308 lr=5.02e-05 tok=17.405B dt=3999 wait=98 +09:58:14 pushed ckpt_w125.pt,log_w125.txt +09:59:14 waited 60s for global r=488 +09:59:21 global r=488 load 4.293 -> 4.281 +10:00:26 step 132320 loss=3.3417 lr=5.01e-05 tok=17.408B dt=4147 wait=98 +10:01:30 step 132340 loss=3.3866 lr=5.01e-05 tok=17.410B dt=4211 wait=99 +10:02:34 step 132360 loss=3.3573 lr=5.01e-05 tok=17.413B dt=4275 wait=99 +10:03:38 step 132380 loss=3.2873 lr=5.01e-05 tok=17.416B dt=4338 wait=99 +10:04:41 step 132400 loss=3.3277 lr=5.01e-05 tok=17.418B dt=4402 wait=100 +10:05:44 step 132420 loss=3.4148 lr=5.01e-05 tok=17.421B dt=4465 wait=100 +10:06:47 step 132440 loss=3.2633 lr=5.00e-05 tok=17.423B dt=4528 wait=100 +10:07:32 pushed ckpt_w125.pt,log_w125.txt +10:08:48 waited 76s for global r=489 +10:08:54 global r=489 load 4.276 -> 4.281 +10:09:27 step 132460 loss=3.3097 lr=5.00e-05 tok=17.426B dt=4688 wait=101 +10:10:30 step 132480 loss=3.2666 lr=5.00e-05 tok=17.429B dt=4751 wait=101 +10:11:33 step 132500 loss=3.3083 lr=5.00e-05 tok=17.431B dt=4814 wait=101 +10:12:37 step 132520 loss=3.2919 lr=5.00e-05 tok=17.434B dt=4877 wait=102 +10:13:40 step 132540 loss=3.2887 lr=4.99e-05 tok=17.437B dt=4941 wait=102 +10:14:43 step 132560 loss=3.3421 lr=4.99e-05 tok=17.439B dt=5004 wait=102 +10:15:46 step 132580 loss=3.2756 lr=4.99e-05 tok=17.442B dt=5067 wait=103 +10:16:49 step 132600 loss=3.3389 lr=4.99e-05 tok=17.444B dt=5130 wait=103 +10:17:03 pushed ckpt_w125.pt,log_w125.txt +10:18:04 waited 61s for global r=490 +10:18:10 global r=490 load 4.287 -> 4.287 +10:19:14 step 132620 loss=3.2868 lr=4.99e-05 tok=17.447B dt=5275 wait=103 +10:20:18 step 132640 loss=3.3341 lr=4.99e-05 tok=17.450B dt=5339 wait=104 +10:21:21 step 132660 loss=3.3407 lr=4.98e-05 tok=17.452B dt=5402 wait=104 +10:22:25 step 132680 loss=3.2635 lr=4.98e-05 tok=17.455B dt=5466 wait=104 +10:23:28 step 132700 loss=3.2807 lr=4.98e-05 tok=17.457B dt=5529 wait=104 +10:24:31 step 132720 loss=3.3897 lr=4.98e-05 tok=17.460B dt=5592 wait=105 +10:25:34 step 132740 loss=3.2789 lr=4.98e-05 tok=17.463B dt=5655 wait=105 +10:26:18 pushed ckpt_w125.pt,log_w125.txt +10:27:34 waited 76s for global r=491 +10:27:39 global r=491 load 4.295 -> 4.299 +10:28:12 step 132760 loss=3.3846 lr=4.97e-05 tok=17.465B dt=5813 wait=105 +10:29:15 step 132780 loss=3.3075 lr=4.97e-05 tok=17.468B dt=5876 wait=106 +10:30:18 step 132800 loss=3.2902 lr=4.97e-05 tok=17.471B dt=5939 wait=106 +10:31:22 step 132820 loss=3.3653 lr=4.97e-05 tok=17.473B dt=6003 wait=106 +10:32:26 step 132840 loss=3.2822 lr=4.97e-05 tok=17.476B dt=6066 wait=107 +10:33:29 step 132860 loss=3.3921 lr=4.96e-05 tok=17.478B dt=6130 wait=107 +10:34:32 step 132880 loss=3.3146 lr=4.96e-05 tok=17.481B dt=6193 wait=107 +10:35:36 step 132900 loss=3.3084 lr=4.96e-05 tok=17.484B dt=6256 wait=108 +10:35:47 pushed ckpt_w125.pt,log_w125.txt +10:36:48 waited 61s for global r=492 +10:36:53 global r=492 load 4.281 -> 4.280 +10:37:58 step 132920 loss=3.3098 lr=4.96e-05 tok=17.486B dt=6399 wait=108 +10:39:02 step 132940 loss=3.2608 lr=4.96e-05 tok=17.489B dt=6462 wait=108 +10:40:05 step 132960 loss=3.2692 lr=4.96e-05 tok=17.492B dt=6526 wait=109 +10:41:08 step 132980 loss=3.2923 lr=4.95e-05 tok=17.494B dt=6589 wait=109 +10:42:11 step 133000 loss=3.2678 lr=4.95e-05 tok=17.497B dt=6652 wait=109 +10:43:15 step 133020 loss=3.3933 lr=4.95e-05 tok=17.499B dt=6716 wait=110 +10:44:18 step 133040 loss=3.2391 lr=4.95e-05 tok=17.502B dt=6779 wait=110 +10:45:04 pushed ckpt_w125.pt,log_w125.txt +10:46:05 waited 60s for global r=493 +10:46:12 global r=493 load 4.300 -> 4.287 +10:46:46 step 133060 loss=3.3167 lr=4.95e-05 tok=17.505B dt=6926 wait=110 +10:47:49 step 133080 loss=3.2339 lr=4.94e-05 tok=17.507B dt=6990 wait=111 +10:48:53 step 133100 loss=3.2949 lr=4.94e-05 tok=17.510B dt=7053 wait=111 +10:49:56 step 133120 loss=3.3786 lr=4.94e-05 tok=17.513B dt=7117 wait=111 +10:50:59 step 133140 loss=3.3073 lr=4.94e-05 tok=17.515B dt=7180 wait=112 +10:52:03 step 133160 loss=3.3826 lr=4.94e-05 tok=17.518B dt=7243 wait=112 +10:53:06 step 133180 loss=3.3014 lr=4.94e-05 tok=17.520B dt=7306 wait=112 +10:54:09 step 133200 loss=3.3189 lr=4.93e-05 tok=17.523B dt=7370 wait=113 +10:54:21 pushed ckpt_w125.pt,log_w125.txt +10:55:22 waited 61s for global r=494 +10:55:28 global r=494 load 4.298 -> 4.283 +10:56:33 step 133220 loss=3.3207 lr=4.93e-05 tok=17.526B dt=7513 wait=113 +10:57:36 step 133240 loss=3.3451 lr=4.93e-05 tok=17.528B dt=7576 wait=113 +10:58:39 step 133260 loss=3.2689 lr=4.93e-05 tok=17.531B dt=7640 wait=113 +10:59:43 step 133280 loss=3.2662 lr=4.93e-05 tok=17.534B dt=7704 wait=114 +11:00:46 step 133300 loss=3.2861 lr=4.92e-05 tok=17.536B dt=7767 wait=114 +11:01:49 step 133320 loss=3.3623 lr=4.92e-05 tok=17.539B dt=7830 wait=114 +11:02:52 step 133340 loss=3.2637 lr=4.92e-05 tok=17.541B dt=7893 wait=115 +11:03:35 pushed ckpt_w125.pt,log_w125.txt +11:04:51 waited 76s for global r=495 +11:04:57 global r=495 load 4.320 -> 4.289 +11:05:30 step 133360 loss=3.3856 lr=4.92e-05 tok=17.544B dt=8051 wait=115 +11:06:33 step 133380 loss=3.2826 lr=4.92e-05 tok=17.547B dt=8114 wait=115 +11:07:37 step 133400 loss=3.2643 lr=4.91e-05 tok=17.549B dt=8178 wait=116 +11:08:40 step 133420 loss=3.3793 lr=4.91e-05 tok=17.552B dt=8241 wait=116 +11:09:44 step 133440 loss=3.2941 lr=4.91e-05 tok=17.554B dt=8305 wait=116 +11:10:47 step 133460 loss=3.3890 lr=4.91e-05 tok=17.557B dt=8368 wait=117 +11:11:50 step 133480 loss=3.2861 lr=4.91e-05 tok=17.560B dt=8431 wait=117 +11:12:54 step 133500 loss=3.3707 lr=4.91e-05 tok=17.562B dt=8495 wait=117 +11:13:11 pushed ckpt_w125.pt,log_w125.txt +11:13:56 waited 46s for global r=496 +11:14:08 global r=496 load 4.301 -> 4.296 +11:15:12 step 133520 loss=3.3433 lr=4.90e-05 tok=17.565B dt=8633 wait=118 +11:16:15 step 133540 loss=3.3653 lr=4.90e-05 tok=17.568B dt=8696 wait=118 +11:17:18 step 133560 loss=3.3433 lr=4.90e-05 tok=17.570B dt=8759 wait=118 +11:18:21 step 133580 loss=3.3211 lr=4.90e-05 tok=17.573B dt=8822 wait=119 +11:19:24 step 133600 loss=3.2647 lr=4.90e-05 tok=17.575B dt=8885 wait=119 +11:20:28 step 133620 loss=3.3662 lr=4.89e-05 tok=17.578B dt=8948 wait=119 +11:21:31 step 133640 loss=3.2473 lr=4.89e-05 tok=17.581B dt=9012 wait=120 +11:22:15 pushed ckpt_w125.pt,log_w125.txt +11:23:16 waited 61s for global r=497 +11:23:24 global r=497 load 4.301 -> 4.298 +11:23:57 step 133660 loss=3.3494 lr=4.89e-05 tok=17.583B dt=9158 wait=120 +11:25:00 step 133680 loss=3.2617 lr=4.89e-05 tok=17.586B dt=9221 wait=120 +11:26:03 step 133700 loss=3.2653 lr=4.89e-05 tok=17.589B dt=9284 wait=121 +11:27:07 step 133720 loss=3.3623 lr=4.89e-05 tok=17.591B dt=9348 wait=121 +11:28:10 step 133740 loss=3.2429 lr=4.88e-05 tok=17.594B dt=9411 wait=121 +11:29:14 step 133760 loss=3.3853 lr=4.88e-05 tok=17.596B dt=9474 wait=122 +11:30:17 step 133780 loss=3.3441 lr=4.88e-05 tok=17.599B dt=9538 wait=122 +11:31:20 step 133800 loss=3.3609 lr=4.88e-05 tok=17.602B dt=9601 wait=122 +11:31:32 pushed ckpt_w125.pt,log_w125.txt +11:32:33 waited 61s for global r=498 +11:32:41 global r=498 load 4.296 -> 4.295 +11:33:45 step 133820 loss=3.3371 lr=4.88e-05 tok=17.604B dt=9746 wait=122 +11:34:49 step 133840 loss=3.3501 lr=4.87e-05 tok=17.607B dt=9810 wait=123 +11:35:52 step 133860 loss=3.3107 lr=4.87e-05 tok=17.610B dt=9873 wait=123 +11:36:55 step 133880 loss=3.2677 lr=4.87e-05 tok=17.612B dt=9936 wait=123 +11:37:59 step 133900 loss=3.3138 lr=4.87e-05 tok=17.615B dt=10000 wait=124 +11:39:02 step 133920 loss=3.3613 lr=4.87e-05 tok=17.617B dt=10063 wait=124 +11:40:06 step 133940 loss=3.2959 lr=4.87e-05 tok=17.620B dt=10126 wait=124 diff --git a/log_w127.txt b/log_w127.txt new file mode 100644 index 0000000000000000000000000000000000000000..60db11410a1e0d84a44820a18501e1227150965d --- /dev/null +++ b/log_w127.txt @@ -0,0 +1,430 @@ +launch +12:22:52 name=w127 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 74/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:22:58 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:22:58 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:23:07 params 1.08B +12:23:09 resumed step=133950 seen=17.62B round=499 +12:23:09 stream salt=133950 +12:25:06 blocks compiled +12:25:06 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:28:34 step 133951 loss=3.1319 lr=4.86e-05 tok=17.621B dt=208 wait=161 +12:29:02 step 133952 loss=3.2731 lr=4.86e-05 tok=17.622B dt=236 wait=161 +12:29:30 step 133960 loss=3.5391 lr=4.86e-05 tok=17.623B dt=264 wait=161 +12:30:36 step 133980 loss=3.4193 lr=4.86e-05 tok=17.625B dt=330 wait=161 +12:31:40 step 134000 loss=3.4134 lr=4.86e-05 tok=17.628B dt=394 wait=162 +12:32:43 step 134020 loss=3.4441 lr=4.86e-05 tok=17.630B dt=457 wait=162 +12:33:46 step 134040 loss=3.5093 lr=4.86e-05 tok=17.633B dt=520 wait=162 +12:34:50 step 134060 loss=3.6387 lr=4.85e-05 tok=17.636B dt=584 wait=163 +12:35:39 global r=500 load 4.268 -> 4.281 +12:36:43 step 134120 loss=3.4066 lr=4.85e-05 tok=17.640B dt=697 wait=163 +12:37:47 step 134140 loss=3.4675 lr=4.85e-05 tok=17.642B dt=761 wait=164 +12:38:51 step 134160 loss=3.3900 lr=4.85e-05 tok=17.645B dt=825 wait=164 +12:39:54 step 134180 loss=3.5155 lr=4.84e-05 tok=17.648B dt=888 wait=164 +12:40:58 step 134200 loss=3.3829 lr=4.84e-05 tok=17.650B dt=952 wait=165 +12:42:01 step 134220 loss=3.5178 lr=4.84e-05 tok=17.653B dt=1015 wait=165 +12:43:04 step 134240 loss=3.3429 lr=4.84e-05 tok=17.655B dt=1078 wait=165 +12:43:51 pushed ckpt_w127.pt,log_w127.txt +12:44:51 waited 61s for global r=501 +12:44:59 global r=501 load 4.278 -> 4.276 +12:45:32 step 134260 loss=3.5006 lr=4.84e-05 tok=17.658B dt=1226 wait=166 +12:46:36 step 134280 loss=3.4963 lr=4.83e-05 tok=17.661B dt=1290 wait=166 +12:47:39 step 134300 loss=3.3634 lr=4.83e-05 tok=17.663B dt=1353 wait=166 +12:48:42 step 134320 loss=3.4276 lr=4.83e-05 tok=17.666B dt=1416 wait=167 +12:49:46 step 134340 loss=3.3981 lr=4.83e-05 tok=17.669B dt=1480 wait=167 +12:50:49 step 134360 loss=3.4475 lr=4.83e-05 tok=17.671B dt=1543 wait=168 +12:51:52 step 134380 loss=3.4276 lr=4.83e-05 tok=17.674B dt=1606 wait=168 +12:52:55 step 134400 loss=3.4469 lr=4.82e-05 tok=17.676B dt=1669 wait=168 +12:53:07 pushed ckpt_w127.pt,log_w127.txt +12:54:07 waited 61s for global r=502 +12:54:15 global r=502 load 4.266 -> 4.274 +12:55:19 step 134420 loss=3.3877 lr=4.82e-05 tok=17.679B dt=1813 wait=169 +12:56:22 step 134440 loss=3.4706 lr=4.82e-05 tok=17.682B dt=1876 wait=169 +12:57:26 step 134460 loss=3.4454 lr=4.82e-05 tok=17.684B dt=1940 wait=169 +12:58:30 step 134480 loss=3.3651 lr=4.82e-05 tok=17.687B dt=2004 wait=170 +12:59:33 step 134500 loss=3.3726 lr=4.81e-05 tok=17.689B dt=2067 wait=170 +13:00:37 step 134520 loss=3.4737 lr=4.81e-05 tok=17.692B dt=2131 wait=170 +13:01:40 step 134540 loss=3.3840 lr=4.81e-05 tok=17.695B dt=2194 wait=170 +13:02:24 pushed ckpt_w127.pt,log_w127.txt +13:05:58 step 134560 loss=3.4896 lr=4.81e-05 tok=17.697B dt=2452 wait=171 +13:06:03 global r=503 load 4.278 -> 4.283 +13:06:35 step 134560 loss=3.4900 lr=4.81e-05 tok=17.699B dt=2489 wait=171 +13:07:38 step 134580 loss=3.4204 lr=4.81e-05 tok=17.701B dt=2552 wait=171 +13:08:41 step 134600 loss=3.4491 lr=4.80e-05 tok=17.704B dt=2615 wait=172 +13:09:44 step 134620 loss=3.5453 lr=4.80e-05 tok=17.707B dt=2678 wait=172 +13:10:48 step 134640 loss=3.4058 lr=4.80e-05 tok=17.709B dt=2742 wait=172 +13:11:51 step 134660 loss=3.4732 lr=4.80e-05 tok=17.712B dt=2805 wait=172 +13:12:55 step 134680 loss=3.4966 lr=4.80e-05 tok=17.714B dt=2869 wait=173 +13:13:59 step 134700 loss=3.4337 lr=4.80e-05 tok=17.717B dt=2933 wait=173 +13:14:13 pushed ckpt_w127.pt,log_w127.txt +13:15:14 waited 60s for global r=504 +13:15:21 global r=504 load 4.275 -> 4.279 +13:16:26 step 134720 loss=3.4808 lr=4.79e-05 tok=17.720B dt=3080 wait=173 +13:17:29 step 134740 loss=3.5147 lr=4.79e-05 tok=17.722B dt=3143 wait=174 +13:18:33 step 134760 loss=3.4452 lr=4.79e-05 tok=17.725B dt=3207 wait=174 +13:19:37 step 134780 loss=3.3613 lr=4.79e-05 tok=17.727B dt=3271 wait=174 +13:20:40 step 134800 loss=3.4376 lr=4.79e-05 tok=17.730B dt=3334 wait=175 +13:21:44 step 134820 loss=3.4930 lr=4.78e-05 tok=17.733B dt=3398 wait=175 +13:22:47 step 134840 loss=3.4102 lr=4.78e-05 tok=17.735B dt=3461 wait=175 +13:23:33 pushed ckpt_w127.pt,log_w127.txt +13:24:48 waited 76s for global r=505 +13:24:55 global r=505 load 4.278 -> 4.282 +13:25:28 step 134860 loss=3.4229 lr=4.78e-05 tok=17.738B dt=3622 wait=176 +13:26:31 step 134880 loss=3.3915 lr=4.78e-05 tok=17.741B dt=3685 wait=176 +13:27:34 step 134900 loss=3.3632 lr=4.78e-05 tok=17.743B dt=3748 wait=176 +13:28:38 step 134920 loss=3.4897 lr=4.78e-05 tok=17.746B dt=3812 wait=177 +13:29:41 step 134940 loss=3.3474 lr=4.77e-05 tok=17.748B dt=3875 wait=177 +13:30:45 step 134960 loss=3.4875 lr=4.77e-05 tok=17.751B dt=3939 wait=177 +13:31:48 step 134980 loss=3.3606 lr=4.77e-05 tok=17.754B dt=4002 wait=177 +13:32:52 step 135000 loss=3.4194 lr=4.77e-05 tok=17.756B dt=4066 wait=178 +13:33:08 pushed ckpt_w127.pt,log_w127.txt +13:34:23 waited 76s for global r=506 +13:34:30 global r=506 load 4.277 -> 4.282 +13:35:34 step 135020 loss=3.4263 lr=4.77e-05 tok=17.759B dt=4228 wait=178 +13:36:38 step 135040 loss=3.4670 lr=4.76e-05 tok=17.762B dt=4292 wait=178 +13:37:41 step 135060 loss=3.5005 lr=4.76e-05 tok=17.764B dt=4355 wait=179 +13:38:44 step 135080 loss=3.4219 lr=4.76e-05 tok=17.767B dt=4418 wait=179 +13:39:47 step 135100 loss=3.4313 lr=4.76e-05 tok=17.769B dt=4481 wait=179 +13:40:51 step 135120 loss=3.4701 lr=4.76e-05 tok=17.772B dt=4545 wait=180 +13:41:54 step 135140 loss=3.3779 lr=4.76e-05 tok=17.775B dt=4608 wait=180 +13:42:38 pushed ckpt_w127.pt,log_w127.txt +13:46:11 step 135160 loss=3.5231 lr=4.75e-05 tok=17.777B dt=4865 wait=180 +13:46:21 global r=507 load 4.290 -> 4.282 +13:46:52 step 135160 loss=3.5141 lr=4.75e-05 tok=17.779B dt=4906 wait=180 +13:47:55 step 135180 loss=3.3208 lr=4.75e-05 tok=17.781B dt=4969 wait=181 +13:48:58 step 135200 loss=3.4341 lr=4.75e-05 tok=17.784B dt=5032 wait=181 +13:50:01 step 135220 loss=3.4109 lr=4.75e-05 tok=17.786B dt=5095 wait=181 +13:51:04 step 135240 loss=3.2383 lr=4.75e-05 tok=17.789B dt=5158 wait=182 +13:52:07 step 135260 loss=3.4891 lr=4.74e-05 tok=17.792B dt=5221 wait=182 +13:53:09 step 135280 loss=3.3615 lr=4.74e-05 tok=17.794B dt=5283 wait=182 +13:54:12 step 135300 loss=3.3469 lr=4.74e-05 tok=17.797B dt=5346 wait=183 +13:54:26 pushed ckpt_w127.pt,log_w127.txt +13:55:56 waited 91s for global r=508 +13:56:05 global r=508 load 4.280 -> 4.283 +13:57:09 step 135320 loss=3.3283 lr=4.74e-05 tok=17.800B dt=5523 wait=183 +13:58:12 step 135340 loss=3.4110 lr=4.74e-05 tok=17.802B dt=5586 wait=183 +13:59:15 step 135360 loss=3.3967 lr=4.74e-05 tok=17.805B dt=5649 wait=184 +14:00:18 step 135380 loss=3.2994 lr=4.73e-05 tok=17.807B dt=5712 wait=184 +14:01:21 step 135400 loss=3.3334 lr=4.73e-05 tok=17.810B dt=5775 wait=184 +14:02:24 step 135420 loss=3.4764 lr=4.73e-05 tok=17.813B dt=5838 wait=185 +14:03:27 step 135440 loss=3.3651 lr=4.73e-05 tok=17.815B dt=5901 wait=185 +14:04:13 pushed ckpt_w127.pt,log_w127.txt +14:05:13 waited 60s for global r=509 +14:05:20 global r=509 load 4.272 -> 4.282 +14:05:53 step 135460 loss=3.4324 lr=4.73e-05 tok=17.818B dt=6047 wait=185 +14:06:56 step 135480 loss=3.2785 lr=4.73e-05 tok=17.821B dt=6110 wait=185 +14:07:58 step 135500 loss=3.3690 lr=4.72e-05 tok=17.823B dt=6172 wait=186 +14:09:01 step 135520 loss=3.4540 lr=4.72e-05 tok=17.826B dt=6235 wait=186 +14:10:04 step 135540 loss=3.3323 lr=4.72e-05 tok=17.828B dt=6298 wait=186 +14:11:07 step 135560 loss=3.3999 lr=4.72e-05 tok=17.831B dt=6361 wait=187 +14:12:10 step 135580 loss=3.3768 lr=4.72e-05 tok=17.834B dt=6424 wait=187 +14:13:13 step 135600 loss=3.3524 lr=4.71e-05 tok=17.836B dt=6487 wait=187 +14:13:28 pushed ckpt_w127.pt,log_w127.txt +14:14:43 waited 76s for global r=510 +14:14:54 global r=510 load 4.300 -> 4.291 +14:15:58 step 135620 loss=3.3786 lr=4.71e-05 tok=17.839B dt=6652 wait=188 +14:17:00 step 135640 loss=3.3599 lr=4.71e-05 tok=17.842B dt=6714 wait=188 +14:18:03 step 135660 loss=3.3916 lr=4.71e-05 tok=17.844B dt=6777 wait=188 +14:19:06 step 135680 loss=3.3579 lr=4.71e-05 tok=17.847B dt=6840 wait=189 +14:20:10 step 135700 loss=3.3482 lr=4.71e-05 tok=17.849B dt=6904 wait=189 +14:21:12 step 135720 loss=3.3968 lr=4.70e-05 tok=17.852B dt=6966 wait=189 +14:22:15 step 135740 loss=3.3419 lr=4.70e-05 tok=17.855B dt=7029 wait=189 +14:22:59 pushed ckpt_w127.pt,log_w127.txt +14:24:29 waited 91s for global r=511 +14:24:38 global r=511 load 4.306 -> 4.289 +14:25:11 step 135760 loss=3.4493 lr=4.70e-05 tok=17.857B dt=7205 wait=190 +14:26:14 step 135780 loss=3.3137 lr=4.70e-05 tok=17.860B dt=7268 wait=190 +14:27:17 step 135800 loss=3.3467 lr=4.70e-05 tok=17.862B dt=7331 wait=190 +14:28:20 step 135820 loss=3.4132 lr=4.69e-05 tok=17.865B dt=7394 wait=191 +14:29:23 step 135840 loss=3.3052 lr=4.69e-05 tok=17.868B dt=7457 wait=191 +14:30:26 step 135860 loss=3.4169 lr=4.69e-05 tok=17.870B dt=7520 wait=191 +14:31:29 step 135880 loss=3.3791 lr=4.69e-05 tok=17.873B dt=7583 wait=192 +14:32:31 step 135900 loss=3.3759 lr=4.69e-05 tok=17.876B dt=7645 wait=192 +14:32:43 pushed ckpt_w127.pt,log_w127.txt +14:33:59 waited 76s for global r=512 +14:34:06 global r=512 load 4.307 -> 4.296 +14:35:11 step 135920 loss=3.3700 lr=4.69e-05 tok=17.878B dt=7805 wait=192 +14:36:13 step 135940 loss=3.3896 lr=4.68e-05 tok=17.881B dt=7867 wait=193 +14:37:16 step 135960 loss=3.4426 lr=4.68e-05 tok=17.883B dt=7930 wait=193 +14:38:19 step 135980 loss=3.3453 lr=4.68e-05 tok=17.886B dt=7993 wait=193 +14:39:22 step 136000 loss=3.3278 lr=4.68e-05 tok=17.889B dt=8056 wait=194 +14:40:24 step 136020 loss=3.4573 lr=4.68e-05 tok=17.891B dt=8118 wait=194 +14:41:27 step 136040 loss=3.3704 lr=4.67e-05 tok=17.894B dt=8181 wait=194 +14:42:14 pushed ckpt_w127.pt,log_w127.txt +14:43:30 waited 76s for global r=513 +14:43:36 global r=513 load 4.295 -> 4.292 +14:44:08 step 136060 loss=3.1939 lr=4.67e-05 tok=17.897B dt=8342 wait=195 +14:45:11 step 136080 loss=3.3553 lr=4.67e-05 tok=17.899B dt=8405 wait=195 +14:46:14 step 136100 loss=3.3736 lr=4.67e-05 tok=17.902B dt=8468 wait=195 +14:47:18 step 136120 loss=3.2955 lr=4.67e-05 tok=17.904B dt=8532 wait=196 +14:48:20 step 136140 loss=3.3082 lr=4.67e-05 tok=17.907B dt=8594 wait=196 +14:49:24 step 136160 loss=3.5233 lr=4.66e-05 tok=17.910B dt=8658 wait=196 +14:50:26 step 136180 loss=3.4188 lr=4.66e-05 tok=17.912B dt=8720 wait=196 +14:51:29 step 136200 loss=3.3771 lr=4.66e-05 tok=17.915B dt=8783 wait=197 +14:51:42 pushed ckpt_w127.pt,log_w127.txt +14:52:58 waited 76s for global r=514 +14:53:05 global r=514 load 4.302 -> 4.295 +14:54:09 step 136220 loss=3.3892 lr=4.66e-05 tok=17.918B dt=8943 wait=197 +14:55:12 step 136240 loss=3.3801 lr=4.66e-05 tok=17.920B dt=9006 wait=197 +14:56:15 step 136260 loss=3.3483 lr=4.65e-05 tok=17.923B dt=9069 wait=198 +14:57:18 step 136280 loss=3.3675 lr=4.65e-05 tok=17.925B dt=9132 wait=198 +14:58:21 step 136300 loss=3.3897 lr=4.65e-05 tok=17.928B dt=9195 wait=198 +14:59:23 step 136320 loss=3.4351 lr=4.65e-05 tok=17.931B dt=9257 wait=199 +15:00:26 step 136340 loss=3.3452 lr=4.65e-05 tok=17.933B dt=9320 wait=199 +15:01:09 pushed ckpt_w127.pt,log_w127.txt +15:02:25 waited 76s for global r=515 +15:02:30 global r=515 load 4.318 -> 4.297 +15:03:03 step 136360 loss=3.4133 lr=4.65e-05 tok=17.936B dt=9477 wait=199 +15:04:05 step 136380 loss=3.3236 lr=4.64e-05 tok=17.939B dt=9539 wait=200 +15:05:08 step 136400 loss=3.4130 lr=4.64e-05 tok=17.941B dt=9602 wait=200 +15:06:11 step 136420 loss=3.3957 lr=4.64e-05 tok=17.944B dt=9665 wait=200 +15:07:14 step 136440 loss=3.3731 lr=4.64e-05 tok=17.946B dt=9728 wait=200 +15:08:16 step 136460 loss=3.4164 lr=4.64e-05 tok=17.949B dt=9790 wait=201 +15:09:19 step 136480 loss=3.3744 lr=4.64e-05 tok=17.952B dt=9853 wait=201 +15:10:23 step 136500 loss=3.3771 lr=4.63e-05 tok=17.954B dt=9917 wait=201 +15:10:34 pushed ckpt_w127.pt,log_w127.txt +15:11:50 waited 76s for global r=516 +15:11:57 global r=516 load 4.295 -> 4.298 +15:13:01 step 136520 loss=3.3216 lr=4.63e-05 tok=17.957B dt=10075 wait=202 +15:14:04 step 136540 loss=3.4042 lr=4.63e-05 tok=17.959B dt=10138 wait=202 +15:15:07 step 136560 loss=3.3987 lr=4.63e-05 tok=17.962B dt=10201 wait=202 +15:16:09 step 136580 loss=3.3885 lr=4.63e-05 tok=17.965B dt=10263 wait=203 +15:17:12 step 136600 loss=3.2946 lr=4.62e-05 tok=17.967B dt=10326 wait=203 +15:18:15 step 136620 loss=3.4510 lr=4.62e-05 tok=17.970B dt=10389 wait=203 +15:19:18 step 136640 loss=3.3782 lr=4.62e-05 tok=17.973B dt=10452 wait=204 +15:20:02 pushed ckpt_w127.pt,log_w127.txt +15:21:18 waited 76s for global r=517 +15:21:25 global r=517 load 4.298 -> 4.295 +15:21:57 step 136660 loss=3.4233 lr=4.62e-05 tok=17.975B dt=10611 wait=204 +15:23:00 step 136680 loss=3.3505 lr=4.62e-05 tok=17.978B dt=10674 wait=204 +15:24:03 step 136700 loss=3.3014 lr=4.62e-05 tok=17.980B dt=10737 wait=205 +15:25:06 step 136720 loss=3.4409 lr=4.61e-05 tok=17.983B dt=10800 wait=205 +15:26:10 step 136740 loss=3.2886 lr=4.61e-05 tok=17.986B dt=10864 wait=205 +15:27:13 step 136760 loss=3.4424 lr=4.61e-05 tok=17.988B dt=10927 wait=206 +15:28:15 step 136780 loss=3.3412 lr=4.61e-05 tok=17.991B dt=10989 wait=206 +15:29:18 step 136800 loss=3.3580 lr=4.61e-05 tok=17.994B dt=11052 wait=206 +15:29:32 pushed ckpt_w127.pt,log_w127.txt +15:30:48 waited 76s for global r=518 +15:30:54 global r=518 load 4.294 -> 4.283 +15:31:59 step 136820 loss=3.2906 lr=4.60e-05 tok=17.996B dt=11213 wait=207 +15:33:01 step 136840 loss=3.3991 lr=4.60e-05 tok=17.999B dt=11275 wait=207 +15:34:04 step 136860 loss=3.3764 lr=4.60e-05 tok=18.001B dt=11338 wait=207 +15:35:07 step 136880 loss=3.3340 lr=4.60e-05 tok=18.004B dt=11401 wait=208 +15:36:10 step 136900 loss=3.3399 lr=4.60e-05 tok=18.007B dt=11464 wait=208 +15:37:13 step 136920 loss=3.4337 lr=4.60e-05 tok=18.009B dt=11527 wait=208 +15:38:15 step 136940 loss=3.3265 lr=4.59e-05 tok=18.012B dt=11589 wait=208 +15:39:04 pushed ckpt_w127.pt,log_w127.txt +15:40:35 waited 91s for global r=519 +15:40:46 global r=519 load 4.296 -> 4.288 +15:41:19 step 136960 loss=3.4058 lr=4.59e-05 tok=18.015B dt=11773 wait=209 +15:42:21 step 136980 loss=3.3300 lr=4.59e-05 tok=18.017B dt=11835 wait=209 +15:43:24 step 137000 loss=3.3128 lr=4.59e-05 tok=18.020B dt=11898 wait=210 +15:44:28 step 137020 loss=3.4040 lr=4.59e-05 tok=18.022B dt=11962 wait=210 +15:45:30 step 137040 loss=3.3008 lr=4.59e-05 tok=18.025B dt=12024 wait=210 +15:46:33 step 137060 loss=3.4755 lr=4.58e-05 tok=18.028B dt=12087 wait=210 +15:47:36 step 137080 loss=3.3109 lr=4.58e-05 tok=18.030B dt=12150 wait=211 +15:48:39 step 137100 loss=3.4273 lr=4.58e-05 tok=18.033B dt=12213 wait=211 +15:48:52 pushed ckpt_w127.pt,log_w127.txt +15:49:52 waited 60s for global r=520 +15:50:01 global r=520 load 4.278 -> 4.286 +15:51:06 step 137120 loss=3.3199 lr=4.58e-05 tok=18.036B dt=12360 wait=211 +15:52:08 step 137140 loss=3.3700 lr=4.58e-05 tok=18.038B dt=12422 wait=212 +15:53:12 step 137160 loss=3.4068 lr=4.57e-05 tok=18.041B dt=12486 wait=212 +15:54:14 step 137180 loss=3.3696 lr=4.57e-05 tok=18.043B dt=12548 wait=212 +15:55:17 step 137200 loss=3.3733 lr=4.57e-05 tok=18.046B dt=12611 wait=213 +15:56:20 step 137220 loss=3.3700 lr=4.57e-05 tok=18.049B dt=12674 wait=213 +15:57:23 step 137240 loss=3.2464 lr=4.57e-05 tok=18.051B dt=12737 wait=213 +15:58:07 pushed ckpt_w127.pt,log_w127.txt +16:01:40 step 137260 loss=3.4393 lr=4.57e-05 tok=18.054B dt=12994 wait=214 +16:02:43 step 137280 loss=3.3836 lr=4.56e-05 tok=18.056B dt=13057 wait=214 +16:03:46 step 137300 loss=3.3538 lr=4.56e-05 tok=18.059B dt=13120 wait=214 +16:04:48 step 137320 loss=3.2752 lr=4.56e-05 tok=18.062B dt=13182 wait=214 +16:05:51 step 137340 loss=3.2925 lr=4.56e-05 tok=18.064B dt=13245 wait=215 +16:06:54 step 137360 loss=3.3260 lr=4.56e-05 tok=18.067B dt=13308 wait=215 +16:07:56 step 137380 loss=3.2878 lr=4.55e-05 tok=18.070B dt=13370 wait=215 +16:08:59 step 137400 loss=3.4508 lr=4.55e-05 tok=18.072B dt=13433 wait=216 +16:09:11 pushed ckpt_w127.pt,log_w127.txt +16:13:15 step 137420 loss=3.3835 lr=4.55e-05 tok=18.075B dt=13689 wait=216 +16:14:18 step 137440 loss=3.5036 lr=4.55e-05 tok=18.077B dt=13752 wait=216 +16:15:21 step 137460 loss=3.3902 lr=4.55e-05 tok=18.080B dt=13815 wait=217 +16:16:23 step 137480 loss=3.4467 lr=4.55e-05 tok=18.083B dt=13877 wait=217 +16:17:26 step 137500 loss=3.4209 lr=4.54e-05 tok=18.085B dt=13940 wait=217 +16:18:29 step 137520 loss=3.4086 lr=4.54e-05 tok=18.088B dt=14003 wait=217 +16:19:31 step 137540 loss=3.4318 lr=4.54e-05 tok=18.091B dt=14065 wait=218 +16:20:14 pushed ckpt_w127.pt,log_w127.txt +16:23:48 step 137560 loss=3.3881 lr=4.54e-05 tok=18.093B dt=14322 wait=218 +16:24:50 step 137580 loss=3.3600 lr=4.54e-05 tok=18.096B dt=14384 wait=218 +16:25:52 step 137600 loss=3.4018 lr=4.54e-05 tok=18.098B dt=14446 wait=219 +16:26:55 step 137620 loss=3.4189 lr=4.53e-05 tok=18.101B dt=14509 wait=219 +16:27:58 step 137640 loss=3.2794 lr=4.53e-05 tok=18.104B dt=14572 wait=219 +16:29:00 step 137660 loss=3.4285 lr=4.53e-05 tok=18.106B dt=14634 wait=220 +16:30:03 step 137680 loss=3.3509 lr=4.53e-05 tok=18.109B dt=14697 wait=220 +16:31:06 step 137700 loss=3.3994 lr=4.53e-05 tok=18.112B dt=14760 wait=220 +16:31:20 pushed ckpt_w127.pt,log_w127.txt +16:35:27 step 137720 loss=3.3305 lr=4.52e-05 tok=18.114B dt=15021 wait=221 +16:36:30 step 137740 loss=3.4019 lr=4.52e-05 tok=18.117B dt=15084 wait=221 +16:37:33 step 137760 loss=3.3594 lr=4.52e-05 tok=18.119B dt=15147 wait=221 +16:38:35 step 137780 loss=3.3681 lr=4.52e-05 tok=18.122B dt=15209 wait=222 +16:38:44 global r=521 load 4.306 -> 4.280 +16:39:47 step 137720 loss=3.3286 lr=4.52e-05 tok=18.125B dt=15281 wait=222 +16:40:50 step 137740 loss=3.3267 lr=4.52e-05 tok=18.127B dt=15344 wait=222 +16:41:53 step 137760 loss=3.3537 lr=4.52e-05 tok=18.130B dt=15407 wait=223 +16:42:57 step 137780 loss=3.2308 lr=4.52e-05 tok=18.133B dt=15471 wait=223 +16:43:59 step 137800 loss=3.3222 lr=4.52e-05 tok=18.135B dt=15533 wait=223 +16:45:03 step 137820 loss=3.3996 lr=4.52e-05 tok=18.138B dt=15597 wait=224 +16:46:04 step 137840 loss=3.4061 lr=4.51e-05 tok=18.140B dt=15658 wait=224 +16:46:49 pushed ckpt_w127.pt,log_w127.txt +16:47:50 waited 61s for global r=522 +16:47:57 global r=522 load 4.292 -> 4.274 +16:48:30 step 137860 loss=3.4579 lr=4.51e-05 tok=18.143B dt=15804 wait=224 +16:49:33 step 137880 loss=3.3617 lr=4.51e-05 tok=18.146B dt=15867 wait=224 +16:50:36 step 137900 loss=3.3251 lr=4.51e-05 tok=18.148B dt=15930 wait=225 +16:51:39 step 137920 loss=3.4557 lr=4.51e-05 tok=18.151B dt=15993 wait=225 +16:52:42 step 137940 loss=3.3781 lr=4.51e-05 tok=18.153B dt=16056 wait=225 +16:53:45 step 137960 loss=3.3817 lr=4.50e-05 tok=18.156B dt=16119 wait=226 +16:54:48 step 137980 loss=3.3623 lr=4.50e-05 tok=18.159B dt=16182 wait=226 +16:55:51 step 138000 loss=3.3584 lr=4.50e-05 tok=18.161B dt=16245 wait=226 +16:56:11 pushed ckpt_w127.pt,log_w127.txt +16:57:11 waited 61s for global r=523 +16:57:20 global r=523 load 4.284 -> 4.289 +16:58:24 step 138020 loss=3.3791 lr=4.50e-05 tok=18.164B dt=16398 wait=227 +16:59:27 step 138040 loss=3.3759 lr=4.50e-05 tok=18.167B dt=16461 wait=227 +17:00:31 step 138060 loss=3.2334 lr=4.49e-05 tok=18.169B dt=16525 wait=227 +17:01:34 step 138080 loss=3.3341 lr=4.49e-05 tok=18.172B dt=16588 wait=228 +17:02:37 step 138100 loss=3.3604 lr=4.49e-05 tok=18.174B dt=16651 wait=228 +17:03:40 step 138120 loss=3.4275 lr=4.49e-05 tok=18.177B dt=16714 wait=228 +17:04:43 step 138140 loss=3.2997 lr=4.49e-05 tok=18.180B dt=16777 wait=229 +17:05:26 pushed ckpt_w127.pt,log_w127.txt +17:06:27 waited 61s for global r=524 +17:06:35 global r=524 load 4.300 -> 4.284 +17:07:08 step 138160 loss=3.4451 lr=4.49e-05 tok=18.182B dt=16922 wait=229 +17:08:10 step 138180 loss=3.3373 lr=4.48e-05 tok=18.185B dt=16984 wait=229 +17:09:13 step 138200 loss=3.3019 lr=4.48e-05 tok=18.188B dt=17047 wait=230 +17:10:16 step 138220 loss=3.4404 lr=4.48e-05 tok=18.190B dt=17110 wait=230 +17:11:19 step 138240 loss=3.3602 lr=4.48e-05 tok=18.193B dt=17173 wait=230 +17:12:21 step 138260 loss=3.4450 lr=4.48e-05 tok=18.195B dt=17235 wait=231 +17:13:24 step 138280 loss=3.3015 lr=4.48e-05 tok=18.198B dt=17298 wait=231 +17:14:28 step 138300 loss=3.3794 lr=4.47e-05 tok=18.201B dt=17362 wait=231 +17:14:49 pushed ckpt_w127.pt,log_w127.txt +17:15:35 waited 45s for global r=525 +17:15:44 global r=525 load 4.285 -> 4.278 +17:16:49 step 138320 loss=3.3908 lr=4.47e-05 tok=18.203B dt=17503 wait=232 +17:17:51 step 138340 loss=3.4030 lr=4.47e-05 tok=18.206B dt=17565 wait=232 +17:18:54 step 138360 loss=3.4086 lr=4.47e-05 tok=18.209B dt=17628 wait=232 +17:19:57 step 138380 loss=3.3594 lr=4.47e-05 tok=18.211B dt=17691 wait=233 +17:21:00 step 138400 loss=3.3161 lr=4.46e-05 tok=18.214B dt=17754 wait=233 +17:22:02 step 138420 loss=3.4944 lr=4.46e-05 tok=18.216B dt=17816 wait=233 +17:23:05 step 138440 loss=3.3598 lr=4.46e-05 tok=18.219B dt=17879 wait=234 +17:23:52 pushed ckpt_w127.pt,log_w127.txt +17:24:53 waited 61s for global r=526 +17:25:07 global r=526 load 4.329 -> 4.291 +17:25:40 step 138460 loss=3.4065 lr=4.46e-05 tok=18.222B dt=18034 wait=234 +17:26:45 step 138480 loss=2.3989 lr=4.46e-05 tok=18.224B dt=18099 wait=234 +17:27:48 step 138500 loss=3.3508 lr=4.46e-05 tok=18.227B dt=18162 wait=234 +17:28:51 step 138520 loss=3.3583 lr=4.45e-05 tok=18.229B dt=18225 wait=235 +17:29:54 step 138540 loss=3.3211 lr=4.45e-05 tok=18.232B dt=18288 wait=235 +17:30:57 step 138560 loss=3.3715 lr=4.45e-05 tok=18.235B dt=18351 wait=235 +17:31:59 step 138580 loss=3.4233 lr=4.45e-05 tok=18.237B dt=18413 wait=236 +17:33:02 step 138600 loss=3.3965 lr=4.45e-05 tok=18.240B dt=18476 wait=236 +17:33:23 pushed ckpt_w127.pt,log_w127.txt +17:34:10 waited 47s for global r=527 +17:34:22 global r=527 load 4.303 -> 4.298 +17:35:27 step 138620 loss=3.3375 lr=4.45e-05 tok=18.243B dt=18621 wait=236 +17:36:30 step 138640 loss=3.3721 lr=4.44e-05 tok=18.245B dt=18684 wait=237 +17:37:32 step 138660 loss=3.3947 lr=4.44e-05 tok=18.248B dt=18746 wait=237 +17:38:34 step 138680 loss=3.4167 lr=4.44e-05 tok=18.250B dt=18808 wait=237 +17:39:37 step 138700 loss=3.3799 lr=4.44e-05 tok=18.253B dt=18871 wait=238 +17:40:40 step 138720 loss=3.4429 lr=4.44e-05 tok=18.256B dt=18934 wait=238 +17:41:43 step 138740 loss=3.3846 lr=4.44e-05 tok=18.258B dt=18997 wait=238 +17:42:29 pushed ckpt_w127.pt,log_w127.txt +17:43:29 waited 61s for global r=528 +17:43:39 global r=528 load 4.297 -> 4.296 +17:44:12 step 138760 loss=3.4976 lr=4.43e-05 tok=18.261B dt=19146 wait=238 +17:45:15 step 138780 loss=3.3164 lr=4.43e-05 tok=18.264B dt=19209 wait=239 +17:46:17 step 138800 loss=3.3051 lr=4.43e-05 tok=18.266B dt=19271 wait=239 +17:47:20 step 138820 loss=3.4336 lr=4.43e-05 tok=18.269B dt=19334 wait=239 +17:48:23 step 138840 loss=3.3849 lr=4.43e-05 tok=18.271B dt=19397 wait=240 +17:49:26 step 138860 loss=3.3509 lr=4.42e-05 tok=18.274B dt=19460 wait=240 +17:50:29 step 138880 loss=3.3717 lr=4.42e-05 tok=18.277B dt=19523 wait=240 +17:51:32 step 138900 loss=3.3539 lr=4.42e-05 tok=18.279B dt=19586 wait=241 +17:51:45 pushed ckpt_w127.pt,log_w127.txt +17:52:47 waited 61s for global r=529 +17:52:54 global r=529 load 4.321 -> 4.299 +17:54:01 step 138920 loss=3.3664 lr=4.42e-05 tok=18.282B dt=19736 wait=241 +17:55:05 step 138940 loss=3.3854 lr=4.42e-05 tok=18.285B dt=19799 wait=241 +17:56:07 step 138960 loss=3.4157 lr=4.42e-05 tok=18.287B dt=19861 wait=242 +17:57:10 step 138980 loss=3.4165 lr=4.41e-05 tok=18.290B dt=19924 wait=242 +17:58:13 step 139000 loss=3.3287 lr=4.41e-05 tok=18.292B dt=19987 wait=242 +17:59:16 step 139020 loss=3.4354 lr=4.41e-05 tok=18.295B dt=20050 wait=243 +18:00:18 step 139040 loss=3.3498 lr=4.41e-05 tok=18.298B dt=20112 wait=243 +18:01:04 pushed ckpt_w127.pt,log_w127.txt +18:02:05 waited 61s for global r=530 +18:02:13 global r=530 load 4.303 -> 4.301 +18:02:46 step 139060 loss=3.3806 lr=4.41e-05 tok=18.300B dt=20260 wait=243 +18:03:49 step 139080 loss=3.3652 lr=4.41e-05 tok=18.303B dt=20323 wait=243 +18:04:52 step 139100 loss=3.3377 lr=4.40e-05 tok=18.306B dt=20386 wait=244 +18:05:55 step 139120 loss=3.4167 lr=4.40e-05 tok=18.308B dt=20449 wait=244 +18:06:58 step 139140 loss=3.3177 lr=4.40e-05 tok=18.311B dt=20512 wait=244 +18:08:01 step 139160 loss=3.3554 lr=4.40e-05 tok=18.313B dt=20575 wait=245 +18:09:04 step 139180 loss=3.3653 lr=4.40e-05 tok=18.316B dt=20638 wait=245 +18:10:07 step 139200 loss=3.3744 lr=4.39e-05 tok=18.319B dt=20701 wait=245 +18:10:19 pushed ckpt_w127.pt,log_w127.txt +18:11:20 waited 61s for global r=531 +18:11:27 global r=531 load 4.300 -> 4.297 +18:12:31 step 139220 loss=3.3700 lr=4.39e-05 tok=18.321B dt=20845 wait=246 +18:13:34 step 139240 loss=3.3708 lr=4.39e-05 tok=18.324B dt=20908 wait=246 +18:14:37 step 139260 loss=3.3218 lr=4.39e-05 tok=18.326B dt=20971 wait=246 +18:15:40 step 139280 loss=3.3701 lr=4.39e-05 tok=18.329B dt=21034 wait=247 +18:16:43 step 139300 loss=3.3889 lr=4.39e-05 tok=18.332B dt=21097 wait=247 +18:17:46 step 139320 loss=3.3378 lr=4.38e-05 tok=18.334B dt=21160 wait=247 +18:18:50 step 139340 loss=3.1649 lr=4.38e-05 tok=18.337B dt=21224 wait=248 +18:19:33 pushed ckpt_w127.pt,log_w127.txt +18:23:07 step 139360 loss=3.4884 lr=4.38e-05 tok=18.340B dt=21481 wait=248 +18:23:13 global r=532 load 4.323 -> 4.315 +18:23:45 step 139360 loss=3.4399 lr=4.38e-05 tok=18.341B dt=21519 wait=248 +18:24:47 step 139380 loss=3.3667 lr=4.38e-05 tok=18.344B dt=21581 wait=249 +18:25:50 step 139400 loss=3.3905 lr=4.38e-05 tok=18.346B dt=21644 wait=249 +18:26:53 step 139420 loss=3.4759 lr=4.38e-05 tok=18.349B dt=21707 wait=249 +18:27:56 step 139440 loss=3.4080 lr=4.37e-05 tok=18.351B dt=21770 wait=249 +18:28:59 step 139460 loss=3.4718 lr=4.37e-05 tok=18.354B dt=21833 wait=250 +18:30:02 step 139480 loss=3.4271 lr=4.37e-05 tok=18.357B dt=21896 wait=250 +18:31:05 step 139500 loss=3.3589 lr=4.37e-05 tok=18.359B dt=21959 wait=250 +18:31:23 pushed ckpt_w127.pt,log_w127.txt +18:35:06 global r=533 load 4.318 -> 4.303 +18:36:09 step 139520 loss=3.3595 lr=4.37e-05 tok=18.363B dt=22263 wait=251 +18:37:12 step 139540 loss=3.4500 lr=4.37e-05 tok=18.366B dt=22326 wait=251 +18:38:15 step 139560 loss=3.4223 lr=4.36e-05 tok=18.368B dt=22389 wait=252 +18:39:18 step 139580 loss=3.3429 lr=4.36e-05 tok=18.371B dt=22452 wait=252 +18:40:20 step 139600 loss=3.3885 lr=4.36e-05 tok=18.374B dt=22514 wait=252 +18:41:24 step 139620 loss=3.4856 lr=4.36e-05 tok=18.376B dt=22578 wait=253 +18:42:27 step 139640 loss=3.3118 lr=4.36e-05 tok=18.379B dt=22641 wait=253 +18:43:10 pushed ckpt_w127.pt,log_w127.txt +18:46:44 step 139660 loss=3.4497 lr=4.35e-05 tok=18.382B dt=22898 wait=253 +18:46:52 global r=534 load 4.299 -> 4.295 +18:47:24 step 139660 loss=3.4366 lr=4.35e-05 tok=18.383B dt=22938 wait=253 +18:48:27 step 139680 loss=3.2719 lr=4.35e-05 tok=18.385B dt=23001 wait=254 +18:49:30 step 139700 loss=3.3518 lr=4.35e-05 tok=18.388B dt=23064 wait=254 +18:50:32 step 139720 loss=3.4913 lr=4.35e-05 tok=18.391B dt=23126 wait=254 +18:51:35 step 139740 loss=3.3490 lr=4.35e-05 tok=18.393B dt=23189 wait=255 +18:52:38 step 139760 loss=3.4537 lr=4.35e-05 tok=18.396B dt=23252 wait=255 +18:53:41 step 139780 loss=3.4155 lr=4.34e-05 tok=18.399B dt=23315 wait=255 +18:54:44 step 139800 loss=3.3757 lr=4.34e-05 tok=18.401B dt=23378 wait=256 +18:54:56 pushed ckpt_w127.pt,log_w127.txt +18:58:36 global r=535 load 4.307 -> 4.308 +18:59:39 step 139820 loss=3.3568 lr=4.34e-05 tok=18.405B dt=23673 wait=256 +19:00:41 step 139840 loss=3.4092 lr=4.34e-05 tok=18.408B dt=23735 wait=256 +19:01:45 step 139860 loss=3.4186 lr=4.34e-05 tok=18.410B dt=23799 wait=257 +19:02:48 step 139880 loss=3.3511 lr=4.34e-05 tok=18.413B dt=23862 wait=257 +19:03:51 step 139900 loss=3.3925 lr=4.33e-05 tok=18.416B dt=23925 wait=257 +19:04:54 step 139920 loss=3.4550 lr=4.33e-05 tok=18.418B dt=23988 wait=258 +19:05:58 step 139940 loss=3.2971 lr=4.33e-05 tok=18.421B dt=24052 wait=258 +19:06:41 pushed ckpt_w127.pt,log_w127.txt +19:10:15 step 139960 loss=3.4390 lr=4.33e-05 tok=18.423B dt=24309 wait=258 +19:10:23 global r=536 load 4.310 -> 4.310 +19:10:54 step 139960 loss=3.4320 lr=4.33e-05 tok=18.425B dt=24348 wait=258 +19:11:57 step 139980 loss=3.3343 lr=4.33e-05 tok=18.427B dt=24411 wait=259 +19:13:00 step 140000 loss=3.3958 lr=4.33e-05 tok=18.430B dt=24474 wait=259 +19:14:04 step 140020 loss=3.4692 lr=4.32e-05 tok=18.433B dt=24538 wait=259 +19:15:06 step 140040 loss=3.4339 lr=4.32e-05 tok=18.435B dt=24600 wait=260 +19:16:10 step 140060 loss=3.3612 lr=4.32e-05 tok=18.438B dt=24664 wait=260 +19:17:12 step 140080 loss=3.3883 lr=4.32e-05 tok=18.441B dt=24726 wait=260 +19:18:14 step 140100 loss=3.4308 lr=4.32e-05 tok=18.443B dt=24788 wait=261 diff --git a/log_w128.txt b/log_w128.txt new file mode 100644 index 0000000000000000000000000000000000000000..0ac148b742784cd396059678c54bf577a01c2f5c --- /dev/null +++ b/log_w128.txt @@ -0,0 +1,633 @@ +launch +12:20:41 name=w128 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 75/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:20:47 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:20:47 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:20:56 params 1.08B +12:20:58 resumed step=133950 seen=17.62B round=499 +12:20:58 stream salt=133950 +12:23:03 blocks compiled +12:23:03 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:25:58 step 133951 loss=3.0819 lr=4.86e-05 tok=17.621B dt=175 wait=141 +12:26:19 step 133952 loss=3.3121 lr=4.86e-05 tok=17.622B dt=196 wait=141 +12:26:46 step 133960 loss=3.5252 lr=4.86e-05 tok=17.623B dt=223 wait=142 +12:27:54 step 133980 loss=3.3463 lr=4.86e-05 tok=17.625B dt=291 wait=142 +12:28:57 step 134000 loss=3.3483 lr=4.86e-05 tok=17.628B dt=354 wait=143 +12:30:00 step 134020 loss=3.4361 lr=4.86e-05 tok=17.630B dt=417 wait=143 +12:31:02 step 134040 loss=3.4899 lr=4.86e-05 tok=17.633B dt=479 wait=143 +12:32:06 step 134060 loss=3.5671 lr=4.85e-05 tok=17.636B dt=543 wait=144 +12:33:09 step 134080 loss=3.3557 lr=4.85e-05 tok=17.638B dt=606 wait=144 +12:34:12 step 134100 loss=3.4579 lr=4.85e-05 tok=17.641B dt=669 wait=144 +12:34:27 pushed ckpt_w128.pt,log_w128.txt +12:35:12 waited 45s for global r=500 +12:35:28 global r=500 load 4.293 -> 4.281 +12:36:34 step 134120 loss=3.3692 lr=4.85e-05 tok=17.644B dt=811 wait=145 +12:37:37 step 134140 loss=3.4152 lr=4.85e-05 tok=17.646B dt=874 wait=145 +12:38:40 step 134160 loss=3.4022 lr=4.85e-05 tok=17.649B dt=937 wait=145 +12:39:43 step 134180 loss=3.3672 lr=4.84e-05 tok=17.651B dt=1000 wait=146 +12:40:46 step 134200 loss=3.3884 lr=4.84e-05 tok=17.654B dt=1063 wait=146 +12:41:49 step 134220 loss=3.4445 lr=4.84e-05 tok=17.657B dt=1127 wait=147 +12:42:52 step 134240 loss=3.2325 lr=4.84e-05 tok=17.659B dt=1189 wait=147 +12:43:38 pushed ckpt_w128.pt,log_w128.txt +12:44:53 waited 76s for global r=501 +12:44:59 global r=501 load 4.277 -> 4.276 +12:45:33 step 134260 loss=3.4207 lr=4.84e-05 tok=17.662B dt=1350 wait=147 +12:46:35 step 134280 loss=3.2937 lr=4.83e-05 tok=17.665B dt=1412 wait=148 +12:47:38 step 134300 loss=3.3846 lr=4.83e-05 tok=17.667B dt=1475 wait=148 +12:48:41 step 134320 loss=3.4679 lr=4.83e-05 tok=17.670B dt=1538 wait=148 +12:49:43 step 134340 loss=3.3876 lr=4.83e-05 tok=17.672B dt=1600 wait=148 +12:50:46 step 134360 loss=3.4447 lr=4.83e-05 tok=17.675B dt=1663 wait=149 +12:51:49 step 134380 loss=3.2942 lr=4.83e-05 tok=17.678B dt=1726 wait=149 +12:52:52 step 134400 loss=3.3878 lr=4.82e-05 tok=17.680B dt=1789 wait=149 +12:53:03 pushed ckpt_w128.pt,log_w128.txt +12:54:19 waited 76s for global r=502 +12:54:26 global r=502 load 4.293 -> 4.274 +12:55:30 step 134420 loss=3.3310 lr=4.82e-05 tok=17.683B dt=1947 wait=150 +12:56:33 step 134440 loss=3.4093 lr=4.82e-05 tok=17.686B dt=2010 wait=150 +12:57:36 step 134460 loss=3.3844 lr=4.82e-05 tok=17.688B dt=2073 wait=150 +12:58:40 step 134480 loss=3.2957 lr=4.82e-05 tok=17.691B dt=2137 wait=151 +12:59:43 step 134500 loss=3.3455 lr=4.81e-05 tok=17.693B dt=2200 wait=151 +13:00:46 step 134520 loss=3.3504 lr=4.81e-05 tok=17.696B dt=2263 wait=151 +13:01:48 step 134540 loss=3.2726 lr=4.81e-05 tok=17.699B dt=2325 wait=152 +13:02:33 pushed ckpt_w128.pt,log_w128.txt +13:05:34 waited 181s for global r=503 +13:05:41 global r=503 load 4.307 -> 4.283 +13:06:14 step 134560 loss=3.4537 lr=4.81e-05 tok=17.701B dt=2591 wait=152 +13:07:16 step 134580 loss=3.3125 lr=4.81e-05 tok=17.704B dt=2653 wait=152 +13:08:19 step 134600 loss=3.3693 lr=4.80e-05 tok=17.707B dt=2716 wait=153 +13:09:22 step 134620 loss=3.4575 lr=4.80e-05 tok=17.709B dt=2779 wait=153 +13:10:25 step 134640 loss=3.3268 lr=4.80e-05 tok=17.712B dt=2842 wait=153 +13:11:28 step 134660 loss=3.4393 lr=4.80e-05 tok=17.714B dt=2905 wait=154 +13:12:31 step 134680 loss=3.3927 lr=4.80e-05 tok=17.717B dt=2968 wait=154 +13:13:34 step 134700 loss=3.4208 lr=4.80e-05 tok=17.720B dt=3031 wait=154 +13:13:46 pushed ckpt_w128.pt,log_w128.txt +13:15:17 waited 91s for global r=504 +13:15:23 global r=504 load 4.294 -> 4.279 +13:16:27 step 134720 loss=3.3857 lr=4.79e-05 tok=17.722B dt=3204 wait=155 +13:17:29 step 134740 loss=3.3398 lr=4.79e-05 tok=17.725B dt=3266 wait=155 +13:18:33 step 134760 loss=3.3135 lr=4.79e-05 tok=17.727B dt=3330 wait=155 +13:19:35 step 134780 loss=3.3533 lr=4.79e-05 tok=17.730B dt=3392 wait=156 +13:20:38 step 134800 loss=3.3953 lr=4.79e-05 tok=17.733B dt=3455 wait=156 +13:21:41 step 134820 loss=3.3965 lr=4.78e-05 tok=17.735B dt=3518 wait=156 +13:22:44 step 134840 loss=3.3399 lr=4.78e-05 tok=17.738B dt=3581 wait=156 +13:23:29 pushed ckpt_w128.pt,log_w128.txt +13:24:45 waited 76s for global r=505 +13:24:55 global r=505 load 4.303 -> 4.282 +13:25:28 step 134860 loss=3.4110 lr=4.78e-05 tok=17.741B dt=3745 wait=157 +13:26:30 step 134880 loss=3.3381 lr=4.78e-05 tok=17.743B dt=3807 wait=157 +13:27:33 step 134900 loss=3.3466 lr=4.78e-05 tok=17.746B dt=3870 wait=157 +13:28:36 step 134920 loss=3.4429 lr=4.78e-05 tok=17.748B dt=3933 wait=158 +13:29:39 step 134940 loss=3.3185 lr=4.77e-05 tok=17.751B dt=3996 wait=158 +13:30:41 step 134960 loss=3.4149 lr=4.77e-05 tok=17.754B dt=4058 wait=158 +13:31:44 step 134980 loss=3.3109 lr=4.77e-05 tok=17.756B dt=4122 wait=159 +13:32:48 step 135000 loss=3.4167 lr=4.77e-05 tok=17.759B dt=4185 wait=159 +13:33:01 pushed ckpt_w128.pt,log_w128.txt +13:34:32 waited 91s for global r=506 +13:34:37 global r=506 load 4.294 -> 4.282 +13:35:41 step 135020 loss=3.3185 lr=4.77e-05 tok=17.762B dt=4358 wait=159 +13:36:44 step 135040 loss=3.4054 lr=4.76e-05 tok=17.764B dt=4421 wait=160 +13:37:47 step 135060 loss=3.3886 lr=4.76e-05 tok=17.767B dt=4484 wait=160 +13:38:50 step 135080 loss=3.3043 lr=4.76e-05 tok=17.769B dt=4547 wait=160 +13:39:53 step 135100 loss=3.3096 lr=4.76e-05 tok=17.772B dt=4610 wait=161 +13:40:56 step 135120 loss=3.3819 lr=4.76e-05 tok=17.775B dt=4673 wait=161 +13:41:58 step 135140 loss=3.2669 lr=4.76e-05 tok=17.777B dt=4735 wait=161 +13:42:41 pushed ckpt_w128.pt,log_w128.txt +13:45:42 waited 181s for global r=507 +13:45:51 global r=507 load 4.316 -> 4.282 +13:46:23 step 135160 loss=3.3749 lr=4.75e-05 tok=17.780B dt=5000 wait=162 +13:47:26 step 135180 loss=3.3647 lr=4.75e-05 tok=17.783B dt=5063 wait=162 +13:48:28 step 135200 loss=3.3067 lr=4.75e-05 tok=17.785B dt=5125 wait=162 +13:49:32 step 135220 loss=3.3498 lr=4.75e-05 tok=17.788B dt=5189 wait=163 +13:50:35 step 135240 loss=3.2673 lr=4.75e-05 tok=17.790B dt=5252 wait=163 +13:51:37 step 135260 loss=3.4421 lr=4.74e-05 tok=17.793B dt=5314 wait=163 +13:52:40 step 135280 loss=3.3502 lr=4.74e-05 tok=17.796B dt=5377 wait=163 +13:53:42 step 135300 loss=3.3774 lr=4.74e-05 tok=17.798B dt=5439 wait=164 +13:53:56 pushed ckpt_w128.pt,log_w128.txt +13:55:57 waited 121s for global r=508 +13:56:03 global r=508 load 4.303 -> 4.283 +13:57:07 step 135320 loss=3.3568 lr=4.74e-05 tok=17.801B dt=5644 wait=164 +13:58:10 step 135340 loss=3.3699 lr=4.74e-05 tok=17.804B dt=5707 wait=164 +13:59:13 step 135360 loss=3.3708 lr=4.74e-05 tok=17.806B dt=5770 wait=165 +14:00:16 step 135380 loss=3.3965 lr=4.73e-05 tok=17.809B dt=5833 wait=165 +14:01:19 step 135400 loss=3.3942 lr=4.73e-05 tok=17.811B dt=5896 wait=165 +14:02:22 step 135420 loss=3.4570 lr=4.73e-05 tok=17.814B dt=5959 wait=166 +14:03:25 step 135440 loss=3.3262 lr=4.73e-05 tok=17.817B dt=6022 wait=166 +14:04:09 pushed ckpt_w128.pt,log_w128.txt +14:05:10 waited 60s for global r=509 +14:05:19 global r=509 load 4.286 -> 4.282 +14:05:52 step 135460 loss=3.4775 lr=4.73e-05 tok=17.819B dt=6169 wait=166 +14:06:55 step 135480 loss=3.3449 lr=4.73e-05 tok=17.822B dt=6232 wait=167 +14:07:57 step 135500 loss=3.3808 lr=4.72e-05 tok=17.824B dt=6294 wait=167 +14:09:00 step 135520 loss=3.4738 lr=4.72e-05 tok=17.827B dt=6357 wait=167 +14:10:03 step 135540 loss=3.2857 lr=4.72e-05 tok=17.830B dt=6420 wait=168 +14:11:06 step 135560 loss=3.4284 lr=4.72e-05 tok=17.832B dt=6483 wait=168 +14:12:09 step 135580 loss=3.3190 lr=4.72e-05 tok=17.835B dt=6546 wait=168 +14:13:12 step 135600 loss=3.3763 lr=4.71e-05 tok=17.838B dt=6609 wait=169 +14:13:24 pushed ckpt_w128.pt,log_w128.txt +14:14:55 waited 91s for global r=510 +14:15:03 global r=510 load 4.287 -> 4.291 +14:16:06 step 135620 loss=3.3878 lr=4.71e-05 tok=17.840B dt=6783 wait=169 +14:17:09 step 135640 loss=3.3692 lr=4.71e-05 tok=17.843B dt=6846 wait=169 +14:18:12 step 135660 loss=3.3793 lr=4.71e-05 tok=17.845B dt=6909 wait=170 +14:19:16 step 135680 loss=3.2880 lr=4.71e-05 tok=17.848B dt=6973 wait=170 +14:20:19 step 135700 loss=3.3290 lr=4.71e-05 tok=17.851B dt=7036 wait=170 +14:21:22 step 135720 loss=3.3532 lr=4.70e-05 tok=17.853B dt=7099 wait=171 +14:22:25 step 135740 loss=3.3293 lr=4.70e-05 tok=17.856B dt=7162 wait=171 +14:23:09 pushed ckpt_w128.pt,log_w128.txt +14:24:24 waited 76s for global r=511 +14:24:32 global r=511 load 4.291 -> 4.289 +14:25:04 step 135760 loss=3.4422 lr=4.70e-05 tok=17.859B dt=7321 wait=172 +14:26:07 step 135780 loss=3.3843 lr=4.70e-05 tok=17.861B dt=7384 wait=172 +14:27:10 step 135800 loss=3.3152 lr=4.70e-05 tok=17.864B dt=7447 wait=172 +14:28:13 step 135820 loss=3.4624 lr=4.69e-05 tok=17.866B dt=7510 wait=173 +14:29:16 step 135840 loss=3.3138 lr=4.69e-05 tok=17.869B dt=7573 wait=173 +14:30:19 step 135860 loss=3.4337 lr=4.69e-05 tok=17.872B dt=7636 wait=173 +14:31:21 step 135880 loss=3.3383 lr=4.69e-05 tok=17.874B dt=7698 wait=173 +14:32:24 step 135900 loss=3.3631 lr=4.69e-05 tok=17.877B dt=7761 wait=174 +14:32:35 pushed ckpt_w128.pt,log_w128.txt +14:33:51 waited 76s for global r=512 +14:33:58 global r=512 load 4.293 -> 4.296 +14:35:03 step 135920 loss=3.3665 lr=4.69e-05 tok=17.880B dt=7920 wait=174 +14:36:06 step 135940 loss=3.1176 lr=4.68e-05 tok=17.882B dt=7983 wait=174 +14:37:09 step 135960 loss=3.3503 lr=4.68e-05 tok=17.885B dt=8046 wait=175 +14:38:12 step 135980 loss=3.3088 lr=4.68e-05 tok=17.887B dt=8109 wait=175 +14:39:15 step 136000 loss=3.3697 lr=4.68e-05 tok=17.890B dt=8172 wait=175 +14:40:18 step 136020 loss=3.3580 lr=4.68e-05 tok=17.893B dt=8235 wait=176 +14:41:20 step 136040 loss=3.3723 lr=4.67e-05 tok=17.895B dt=8297 wait=176 +14:42:06 pushed ckpt_w128.pt,log_w128.txt +14:43:21 waited 76s for global r=513 +14:43:31 global r=513 load 4.292 -> 4.292 +14:44:05 step 136060 loss=3.2713 lr=4.67e-05 tok=17.898B dt=8462 wait=176 +14:45:08 step 136080 loss=3.1887 lr=4.67e-05 tok=17.901B dt=8525 wait=177 +14:46:11 step 136100 loss=3.3153 lr=4.67e-05 tok=17.903B dt=8588 wait=177 +14:47:14 step 136120 loss=3.4434 lr=4.67e-05 tok=17.906B dt=8651 wait=177 +14:48:17 step 136140 loss=3.2817 lr=4.67e-05 tok=17.908B dt=8714 wait=178 +14:49:20 step 136160 loss=3.4172 lr=4.66e-05 tok=17.911B dt=8777 wait=178 +14:50:23 step 136180 loss=3.3564 lr=4.66e-05 tok=17.914B dt=8840 wait=179 +14:51:26 step 136200 loss=3.3648 lr=4.66e-05 tok=17.916B dt=8903 wait=179 +14:51:39 pushed ckpt_w128.pt,log_w128.txt +14:52:55 waited 76s for global r=514 +14:53:01 global r=514 load 4.281 -> 4.295 +14:54:04 step 136220 loss=3.3523 lr=4.66e-05 tok=17.919B dt=9061 wait=179 +14:55:07 step 136240 loss=3.4379 lr=4.66e-05 tok=17.921B dt=9124 wait=180 +14:56:10 step 136260 loss=3.3372 lr=4.65e-05 tok=17.924B dt=9187 wait=180 +14:57:13 step 136280 loss=3.3239 lr=4.65e-05 tok=17.927B dt=9250 wait=180 +14:58:16 step 136300 loss=3.3642 lr=4.65e-05 tok=17.929B dt=9313 wait=180 +14:59:19 step 136320 loss=3.4588 lr=4.65e-05 tok=17.932B dt=9376 wait=181 +15:00:21 step 136340 loss=3.3092 lr=4.65e-05 tok=17.935B dt=9438 wait=181 +15:01:06 pushed ckpt_w128.pt,log_w128.txt +15:02:22 waited 76s for global r=515 +15:02:30 global r=515 load 4.286 -> 4.297 +15:03:02 step 136360 loss=3.3499 lr=4.65e-05 tok=17.937B dt=9599 wait=181 +15:04:04 step 136380 loss=3.3431 lr=4.64e-05 tok=17.940B dt=9661 wait=182 +15:05:08 step 136400 loss=3.3073 lr=4.64e-05 tok=17.942B dt=9725 wait=182 +15:06:11 step 136420 loss=3.3951 lr=4.64e-05 tok=17.945B dt=9788 wait=182 +15:07:13 step 136440 loss=3.2801 lr=4.64e-05 tok=17.948B dt=9850 wait=183 +15:08:16 step 136460 loss=3.4214 lr=4.64e-05 tok=17.950B dt=9913 wait=183 +15:09:19 step 136480 loss=3.3690 lr=4.64e-05 tok=17.953B dt=9976 wait=183 +15:10:22 step 136500 loss=3.4046 lr=4.63e-05 tok=17.956B dt=10039 wait=184 +15:10:36 pushed ckpt_w128.pt,log_w128.txt +15:11:51 waited 76s for global r=516 +15:11:58 global r=516 load 4.309 -> 4.298 +15:13:02 step 136520 loss=3.3992 lr=4.63e-05 tok=17.958B dt=10199 wait=184 +15:14:05 step 136540 loss=3.3825 lr=4.63e-05 tok=17.961B dt=10262 wait=184 +15:15:08 step 136560 loss=3.2994 lr=4.63e-05 tok=17.963B dt=10325 wait=185 +15:16:11 step 136580 loss=3.2952 lr=4.63e-05 tok=17.966B dt=10388 wait=185 +15:17:14 step 136600 loss=3.3355 lr=4.62e-05 tok=17.969B dt=10451 wait=185 +15:18:16 step 136620 loss=3.4606 lr=4.62e-05 tok=17.971B dt=10513 wait=186 +15:19:19 step 136640 loss=3.3192 lr=4.62e-05 tok=17.974B dt=10576 wait=186 +15:20:03 pushed ckpt_w128.pt,log_w128.txt +15:21:18 waited 76s for global r=517 +15:21:25 global r=517 load 4.305 -> 4.295 +15:21:57 step 136660 loss=3.3727 lr=4.62e-05 tok=17.977B dt=10734 wait=186 +15:23:00 step 136680 loss=3.3118 lr=4.62e-05 tok=17.979B dt=10797 wait=187 +15:24:02 step 136700 loss=3.3165 lr=4.62e-05 tok=17.982B dt=10859 wait=187 +15:25:06 step 136720 loss=3.4830 lr=4.61e-05 tok=17.984B dt=10923 wait=187 +15:26:08 step 136740 loss=3.3280 lr=4.61e-05 tok=17.987B dt=10985 wait=188 +15:27:11 step 136760 loss=3.3870 lr=4.61e-05 tok=17.990B dt=11048 wait=188 +15:28:14 step 136780 loss=3.3910 lr=4.61e-05 tok=17.992B dt=11111 wait=188 +15:29:17 step 136800 loss=3.3976 lr=4.61e-05 tok=17.995B dt=11174 wait=189 +15:29:29 pushed ckpt_w128.pt,log_w128.txt +15:31:00 waited 91s for global r=518 +15:31:07 global r=518 load 4.275 -> 4.283 +15:32:11 step 136820 loss=3.3618 lr=4.60e-05 tok=17.997B dt=11348 wait=189 +15:33:16 step 136840 loss=2.8572 lr=4.60e-05 tok=18.000B dt=11413 wait=189 +15:34:19 step 136860 loss=3.3701 lr=4.60e-05 tok=18.003B dt=11476 wait=190 +15:35:22 step 136880 loss=3.3386 lr=4.60e-05 tok=18.005B dt=11539 wait=190 +15:36:24 step 136900 loss=3.3170 lr=4.60e-05 tok=18.008B dt=11601 wait=190 +15:37:27 step 136920 loss=3.4355 lr=4.60e-05 tok=18.011B dt=11664 wait=191 +15:38:31 step 136940 loss=3.2861 lr=4.59e-05 tok=18.013B dt=11728 wait=191 +15:39:14 pushed ckpt_w128.pt,log_w128.txt +15:40:30 waited 76s for global r=519 +15:40:36 global r=519 load 4.296 -> 4.288 +15:41:09 step 136960 loss=3.4217 lr=4.59e-05 tok=18.016B dt=11886 wait=191 +15:42:12 step 136980 loss=3.3211 lr=4.59e-05 tok=18.018B dt=11949 wait=192 +15:43:14 step 137000 loss=3.3416 lr=4.59e-05 tok=18.021B dt=12011 wait=192 +15:44:17 step 137020 loss=3.4218 lr=4.59e-05 tok=18.024B dt=12074 wait=192 +15:45:20 step 137040 loss=3.2562 lr=4.59e-05 tok=18.026B dt=12137 wait=193 +15:46:22 step 137060 loss=3.4829 lr=4.58e-05 tok=18.029B dt=12199 wait=193 +15:47:25 step 137080 loss=3.4107 lr=4.58e-05 tok=18.032B dt=12262 wait=193 +15:48:28 step 137100 loss=3.4271 lr=4.58e-05 tok=18.034B dt=12325 wait=194 +15:48:40 pushed ckpt_w128.pt,log_w128.txt +15:49:56 waited 76s for global r=520 +15:50:03 global r=520 load 4.293 -> 4.286 +15:51:07 step 137120 loss=3.2933 lr=4.58e-05 tok=18.037B dt=12484 wait=194 +15:52:09 step 137140 loss=3.3872 lr=4.58e-05 tok=18.039B dt=12546 wait=194 +15:53:12 step 137160 loss=3.4286 lr=4.57e-05 tok=18.042B dt=12609 wait=195 +15:54:15 step 137180 loss=3.3005 lr=4.57e-05 tok=18.045B dt=12672 wait=195 +15:55:17 step 137200 loss=3.3537 lr=4.57e-05 tok=18.047B dt=12734 wait=195 +15:56:20 step 137220 loss=3.3704 lr=4.57e-05 tok=18.050B dt=12797 wait=196 +15:57:23 step 137240 loss=3.2653 lr=4.57e-05 tok=18.053B dt=12860 wait=196 +15:58:11 pushed ckpt_w128.pt,log_w128.txt +16:01:45 step 137260 loss=3.4321 lr=4.57e-05 tok=18.055B dt=13122 wait=196 +16:02:47 step 137280 loss=3.2984 lr=4.56e-05 tok=18.058B dt=13184 wait=197 +16:03:50 step 137300 loss=3.4049 lr=4.56e-05 tok=18.060B dt=13247 wait=197 +16:04:52 step 137320 loss=3.2813 lr=4.56e-05 tok=18.063B dt=13309 wait=197 +16:05:55 step 137340 loss=3.2790 lr=4.56e-05 tok=18.066B dt=13372 wait=197 +16:06:58 step 137360 loss=3.3628 lr=4.56e-05 tok=18.068B dt=13435 wait=198 +16:08:01 step 137380 loss=3.2921 lr=4.55e-05 tok=18.071B dt=13498 wait=198 +16:09:03 step 137400 loss=3.5030 lr=4.55e-05 tok=18.074B dt=13560 wait=198 +16:09:18 pushed ckpt_w128.pt,log_w128.txt +16:13:23 step 137420 loss=3.3100 lr=4.55e-05 tok=18.076B dt=13820 wait=199 +16:14:26 step 137440 loss=3.4077 lr=4.55e-05 tok=18.079B dt=13883 wait=199 +16:15:28 step 137460 loss=3.3981 lr=4.55e-05 tok=18.081B dt=13945 wait=199 +16:16:32 step 137480 loss=3.3965 lr=4.55e-05 tok=18.084B dt=14009 wait=200 +16:17:35 step 137500 loss=3.3339 lr=4.54e-05 tok=18.087B dt=14072 wait=200 +16:18:37 step 137520 loss=3.4099 lr=4.54e-05 tok=18.089B dt=14134 wait=201 +16:19:39 step 137540 loss=3.3955 lr=4.54e-05 tok=18.092B dt=14196 wait=201 +16:20:24 pushed ckpt_w128.pt,log_w128.txt +16:23:57 step 137560 loss=3.3400 lr=4.54e-05 tok=18.094B dt=14454 wait=201 +16:24:59 step 137580 loss=3.3812 lr=4.54e-05 tok=18.097B dt=14516 wait=202 +16:26:02 step 137600 loss=3.4140 lr=4.54e-05 tok=18.100B dt=14579 wait=202 +16:27:05 step 137620 loss=3.3638 lr=4.53e-05 tok=18.102B dt=14642 wait=202 +16:28:08 step 137640 loss=3.3902 lr=4.53e-05 tok=18.105B dt=14705 wait=202 +16:29:11 step 137660 loss=3.3546 lr=4.53e-05 tok=18.108B dt=14768 wait=203 +16:30:13 step 137680 loss=3.4089 lr=4.53e-05 tok=18.110B dt=14830 wait=203 +16:31:16 step 137700 loss=3.3952 lr=4.53e-05 tok=18.113B dt=14893 wait=203 +16:31:30 pushed ckpt_w128.pt,log_w128.txt +16:35:35 step 137720 loss=3.3689 lr=4.52e-05 tok=18.115B dt=15152 wait=204 +16:36:38 step 137740 loss=3.4219 lr=4.52e-05 tok=18.118B dt=15215 wait=204 +16:37:41 step 137760 loss=3.3833 lr=4.52e-05 tok=18.121B dt=15278 wait=204 +16:38:20 global r=521 load 4.259 -> 4.280 +16:39:39 step 137720 loss=3.3037 lr=4.52e-05 tok=18.125B dt=15396 wait=205 +16:40:45 step 137740 loss=3.3461 lr=4.52e-05 tok=18.127B dt=15462 wait=205 +16:41:48 step 137760 loss=3.3339 lr=4.52e-05 tok=18.130B dt=15525 wait=206 +16:42:59 step 137780 loss=3.3325 lr=4.52e-05 tok=18.133B dt=15596 wait=206 +16:44:01 step 137800 loss=3.3269 lr=4.52e-05 tok=18.135B dt=15658 wait=206 +16:45:04 step 137820 loss=3.3967 lr=4.52e-05 tok=18.138B dt=15721 wait=206 +16:46:06 step 137840 loss=3.4313 lr=4.51e-05 tok=18.140B dt=15783 wait=207 +16:46:52 pushed ckpt_w128.pt,log_w128.txt +16:47:53 waited 60s for global r=522 +16:47:58 global r=522 load 4.275 -> 4.274 +16:48:31 step 137860 loss=3.4402 lr=4.51e-05 tok=18.143B dt=15928 wait=207 +16:49:49 step 137880 loss=3.2901 lr=4.51e-05 tok=18.146B dt=16006 wait=207 +16:50:57 step 137900 loss=3.2805 lr=4.51e-05 tok=18.148B dt=16074 wait=208 +16:52:00 step 137920 loss=3.3987 lr=4.51e-05 tok=18.151B dt=16137 wait=208 +16:53:02 step 137940 loss=3.4045 lr=4.51e-05 tok=18.153B dt=16199 wait=208 +16:54:05 step 137960 loss=3.3972 lr=4.50e-05 tok=18.156B dt=16262 wait=209 +16:55:08 step 137980 loss=3.3172 lr=4.50e-05 tok=18.159B dt=16325 wait=209 +16:56:11 step 138000 loss=3.3677 lr=4.50e-05 tok=18.161B dt=16388 wait=209 +16:56:23 pushed ckpt_w128.pt,log_w128.txt +16:57:09 waited 45s for global r=523 +16:57:17 global r=523 load 4.306 -> 4.289 +16:58:20 step 138020 loss=3.3640 lr=4.50e-05 tok=18.164B dt=16517 wait=210 +16:59:43 step 138040 loss=3.4044 lr=4.50e-05 tok=18.167B dt=16600 wait=210 +17:00:46 step 138060 loss=3.3412 lr=4.49e-05 tok=18.169B dt=16663 wait=210 +17:01:50 step 138080 loss=3.3345 lr=4.49e-05 tok=18.172B dt=16727 wait=211 +17:02:53 step 138100 loss=3.3934 lr=4.49e-05 tok=18.174B dt=16790 wait=211 +17:03:56 step 138120 loss=3.4283 lr=4.49e-05 tok=18.177B dt=16853 wait=211 +17:04:59 step 138140 loss=3.3088 lr=4.49e-05 tok=18.180B dt=16916 wait=211 +17:05:43 pushed ckpt_w128.pt,log_w128.txt +17:06:28 waited 46s for global r=524 +17:06:34 global r=524 load 4.282 -> 4.284 +17:07:06 step 138160 loss=3.3928 lr=4.49e-05 tok=18.182B dt=17043 wait=212 +17:08:15 step 138180 loss=3.3116 lr=4.48e-05 tok=18.185B dt=17112 wait=212 +17:09:30 step 138200 loss=3.3545 lr=4.48e-05 tok=18.188B dt=17187 wait=212 +17:10:33 step 138220 loss=3.3701 lr=4.48e-05 tok=18.190B dt=17250 wait=213 +17:11:36 step 138240 loss=3.3504 lr=4.48e-05 tok=18.193B dt=17313 wait=213 +17:12:39 step 138260 loss=3.4721 lr=4.48e-05 tok=18.195B dt=17376 wait=213 +17:13:42 step 138280 loss=3.3339 lr=4.48e-05 tok=18.198B dt=17439 wait=214 +17:14:44 step 138300 loss=3.3874 lr=4.47e-05 tok=18.201B dt=17501 wait=214 +17:14:58 pushed ckpt_w128.pt,log_w128.txt +17:15:43 waited 45s for global r=525 +17:15:49 global r=525 load 4.278 -> 4.278 +17:16:53 step 138320 loss=3.3505 lr=4.47e-05 tok=18.203B dt=17630 wait=214 +17:18:04 step 138340 loss=3.4318 lr=4.47e-05 tok=18.206B dt=17701 wait=215 +17:19:18 step 138360 loss=3.3687 lr=4.47e-05 tok=18.209B dt=17775 wait=215 +17:20:21 step 138380 loss=3.3339 lr=4.47e-05 tok=18.211B dt=17838 wait=215 +17:21:24 step 138400 loss=3.2916 lr=4.46e-05 tok=18.214B dt=17901 wait=216 +17:22:27 step 138420 loss=3.4030 lr=4.46e-05 tok=18.216B dt=17964 wait=216 +17:23:30 step 138440 loss=3.3046 lr=4.46e-05 tok=18.219B dt=18027 wait=216 +17:24:13 pushed ckpt_w128.pt,log_w128.txt +17:24:44 waited 30s for global r=526 +17:24:51 global r=526 load 4.283 -> 4.291 +17:25:31 step 138460 loss=3.4310 lr=4.46e-05 tok=18.222B dt=18148 wait=217 +17:26:46 step 138480 loss=3.3379 lr=4.46e-05 tok=18.224B dt=18223 wait=217 +17:27:49 step 138500 loss=3.4120 lr=4.46e-05 tok=18.227B dt=18286 wait=217 +17:28:52 step 138520 loss=3.3674 lr=4.45e-05 tok=18.229B dt=18349 wait=218 +17:29:55 step 138540 loss=3.3374 lr=4.45e-05 tok=18.232B dt=18412 wait=218 +17:30:58 step 138560 loss=3.4206 lr=4.45e-05 tok=18.235B dt=18475 wait=218 +17:32:01 step 138580 loss=3.4370 lr=4.45e-05 tok=18.237B dt=18538 wait=219 +17:33:04 step 138600 loss=3.3570 lr=4.45e-05 tok=18.240B dt=18601 wait=219 +17:33:21 pushed ckpt_w128.pt,log_w128.txt +17:34:06 waited 45s for global r=527 +17:34:15 global r=527 load 4.285 -> 4.298 +17:35:33 step 138620 loss=3.2998 lr=4.45e-05 tok=18.243B dt=18750 wait=219 +17:36:41 step 138640 loss=3.3621 lr=4.44e-05 tok=18.245B dt=18818 wait=220 +17:37:44 step 138660 loss=3.3536 lr=4.44e-05 tok=18.248B dt=18881 wait=220 +17:38:47 step 138680 loss=3.3091 lr=4.44e-05 tok=18.250B dt=18944 wait=220 +17:39:50 step 138700 loss=3.3709 lr=4.44e-05 tok=18.253B dt=19007 wait=221 +17:40:53 step 138720 loss=3.4511 lr=4.44e-05 tok=18.256B dt=19070 wait=221 +17:41:56 step 138740 loss=3.3316 lr=4.44e-05 tok=18.258B dt=19133 wait=221 +17:42:39 pushed ckpt_w128.pt,log_w128.txt +17:43:24 waited 45s for global r=528 +17:43:33 global r=528 load 4.323 -> 4.296 +17:44:05 step 138760 loss=3.4550 lr=4.43e-05 tok=18.261B dt=19262 wait=222 +17:45:26 step 138780 loss=3.3676 lr=4.43e-05 tok=18.264B dt=19343 wait=222 +17:46:30 step 138800 loss=3.3136 lr=4.43e-05 tok=18.266B dt=19407 wait=222 +17:47:33 step 138820 loss=3.4896 lr=4.43e-05 tok=18.269B dt=19470 wait=223 +17:48:36 step 138840 loss=3.3289 lr=4.43e-05 tok=18.271B dt=19533 wait=223 +17:49:39 step 138860 loss=3.4358 lr=4.42e-05 tok=18.274B dt=19597 wait=223 +17:50:42 step 138880 loss=3.3856 lr=4.42e-05 tok=18.277B dt=19659 wait=224 +17:51:45 step 138900 loss=3.4069 lr=4.42e-05 tok=18.279B dt=19722 wait=224 +17:52:00 pushed ckpt_w128.pt,log_w128.txt +17:52:46 waited 45s for global r=529 +17:52:51 global r=529 load 4.296 -> 4.299 +17:54:01 step 138920 loss=3.3255 lr=4.42e-05 tok=18.282B dt=19858 wait=224 +17:55:18 step 138940 loss=3.3951 lr=4.42e-05 tok=18.285B dt=19935 wait=225 +17:56:21 step 138960 loss=3.3965 lr=4.42e-05 tok=18.287B dt=19998 wait=225 +17:57:23 step 138980 loss=3.3929 lr=4.41e-05 tok=18.290B dt=20060 wait=225 +17:58:26 step 139000 loss=3.3987 lr=4.41e-05 tok=18.292B dt=20123 wait=226 +17:59:29 step 139020 loss=3.4176 lr=4.41e-05 tok=18.295B dt=20186 wait=226 +18:00:32 step 139040 loss=3.2752 lr=4.41e-05 tok=18.298B dt=20249 wait=226 +18:01:15 pushed ckpt_w128.pt,log_w128.txt +18:02:01 waited 46s for global r=530 +18:02:07 global r=530 load 4.307 -> 4.301 +18:02:39 step 139060 loss=3.4529 lr=4.41e-05 tok=18.300B dt=20376 wait=227 +18:03:50 step 139080 loss=3.3375 lr=4.41e-05 tok=18.303B dt=20447 wait=227 +18:05:04 step 139100 loss=3.3233 lr=4.40e-05 tok=18.306B dt=20522 wait=227 +18:06:09 step 139120 loss=3.1926 lr=4.40e-05 tok=18.308B dt=20586 wait=227 +18:07:12 step 139140 loss=3.3187 lr=4.40e-05 tok=18.311B dt=20649 wait=228 +18:08:15 step 139160 loss=3.3272 lr=4.40e-05 tok=18.313B dt=20712 wait=228 +18:09:18 step 139180 loss=3.4275 lr=4.40e-05 tok=18.316B dt=20775 wait=228 +18:10:22 step 139200 loss=3.3454 lr=4.39e-05 tok=18.319B dt=20839 wait=229 +18:10:35 pushed ckpt_w128.pt,log_w128.txt +18:11:21 waited 46s for global r=531 +18:11:28 global r=531 load 4.307 -> 4.297 +18:12:31 step 139220 loss=3.3912 lr=4.39e-05 tok=18.321B dt=20968 wait=229 +18:13:47 step 139240 loss=3.4116 lr=4.39e-05 tok=18.324B dt=21044 wait=229 +18:14:56 step 139260 loss=3.3699 lr=4.39e-05 tok=18.326B dt=21113 wait=230 +18:15:58 step 139280 loss=3.1941 lr=4.39e-05 tok=18.329B dt=21175 wait=230 +18:17:01 step 139300 loss=3.1925 lr=4.39e-05 tok=18.332B dt=21238 wait=230 +18:18:05 step 139320 loss=3.2891 lr=4.38e-05 tok=18.334B dt=21302 wait=231 +18:19:07 step 139340 loss=3.3390 lr=4.38e-05 tok=18.337B dt=21364 wait=231 +18:19:52 pushed ckpt_w128.pt,log_w128.txt +18:22:53 waited 181s for global r=532 +18:23:02 global r=532 load 4.295 -> 4.315 +18:23:35 step 139360 loss=3.2761 lr=4.38e-05 tok=18.340B dt=21632 wait=231 +18:24:46 step 139380 loss=3.0227 lr=4.38e-05 tok=18.342B dt=21703 wait=232 +18:26:00 step 139400 loss=3.2866 lr=4.38e-05 tok=18.345B dt=21777 wait=232 +18:27:03 step 139420 loss=3.4202 lr=4.38e-05 tok=18.347B dt=21840 wait=232 +18:28:06 step 139440 loss=3.2430 lr=4.37e-05 tok=18.350B dt=21903 wait=233 +18:29:09 step 139460 loss=3.2944 lr=4.37e-05 tok=18.353B dt=21966 wait=233 +18:30:12 step 139480 loss=3.3276 lr=4.37e-05 tok=18.355B dt=22029 wait=233 +18:31:15 step 139500 loss=3.3148 lr=4.37e-05 tok=18.358B dt=22092 wait=234 +18:31:27 pushed ckpt_w128.pt,log_w128.txt +18:35:09 global r=533 load 4.305 -> 4.303 +18:36:18 step 139520 loss=3.4332 lr=4.37e-05 tok=18.362B dt=22395 wait=234 +18:37:33 step 139540 loss=3.4352 lr=4.37e-05 tok=18.364B dt=22470 wait=235 +18:38:36 step 139560 loss=3.3577 lr=4.36e-05 tok=18.367B dt=22533 wait=235 +18:39:39 step 139580 loss=3.2505 lr=4.36e-05 tok=18.370B dt=22596 wait=235 +18:40:42 step 139600 loss=3.3604 lr=4.36e-05 tok=18.372B dt=22659 wait=236 +18:41:46 step 139620 loss=3.3809 lr=4.36e-05 tok=18.375B dt=22723 wait=236 +18:42:48 step 139640 loss=3.3190 lr=4.36e-05 tok=18.378B dt=22785 wait=236 +18:43:33 pushed ckpt_w128.pt,log_w128.txt +18:46:35 waited 182s for global r=534 +18:46:41 global r=534 load 4.299 -> 4.295 +18:47:14 step 139660 loss=3.4332 lr=4.35e-05 tok=18.380B dt=23051 wait=236 +18:48:33 step 139680 loss=3.3350 lr=4.35e-05 tok=18.383B dt=23130 wait=237 +18:49:41 step 139700 loss=3.3119 lr=4.35e-05 tok=18.385B dt=23198 wait=237 +18:50:44 step 139720 loss=3.4545 lr=4.35e-05 tok=18.388B dt=23261 wait=237 +18:51:47 step 139740 loss=3.3730 lr=4.35e-05 tok=18.391B dt=23324 wait=238 +18:52:49 step 139760 loss=3.4045 lr=4.35e-05 tok=18.393B dt=23386 wait=238 +18:53:51 step 139780 loss=3.3528 lr=4.34e-05 tok=18.396B dt=23448 wait=238 +18:54:55 step 139800 loss=3.3459 lr=4.34e-05 tok=18.399B dt=23512 wait=239 +18:55:12 pushed ckpt_w128.pt,log_w128.txt +18:58:19 waited 186s for global r=535 +18:58:25 global r=535 load 4.308 -> 4.308 +18:59:37 step 139820 loss=3.3949 lr=4.34e-05 tok=18.401B dt=23794 wait=239 +19:00:52 step 139840 loss=3.3737 lr=4.34e-05 tok=18.404B dt=23869 wait=239 +19:01:54 step 139860 loss=3.3774 lr=4.34e-05 tok=18.406B dt=23931 wait=240 +19:02:57 step 139880 loss=3.3940 lr=4.34e-05 tok=18.409B dt=23994 wait=240 +19:04:00 step 139900 loss=3.3454 lr=4.33e-05 tok=18.412B dt=24057 wait=240 +19:05:02 step 139920 loss=3.4499 lr=4.33e-05 tok=18.414B dt=24119 wait=241 +19:06:05 step 139940 loss=3.2608 lr=4.33e-05 tok=18.417B dt=24182 wait=241 +19:06:49 pushed ckpt_w128.pt,log_w128.txt +19:09:51 waited 182s for global r=536 +19:09:57 global r=536 load 4.316 -> 4.310 +19:10:30 step 139960 loss=3.3842 lr=4.33e-05 tok=18.420B dt=24447 wait=241 +19:11:51 step 139980 loss=3.3168 lr=4.33e-05 tok=18.422B dt=24528 wait=242 +19:12:54 step 140000 loss=3.3739 lr=4.33e-05 tok=18.425B dt=24591 wait=242 +19:13:57 step 140020 loss=3.4583 lr=4.32e-05 tok=18.427B dt=24654 wait=242 +19:15:00 step 140040 loss=3.2296 lr=4.32e-05 tok=18.430B dt=24717 wait=243 +19:16:03 step 140060 loss=3.4170 lr=4.32e-05 tok=18.433B dt=24780 wait=243 +19:17:06 step 140080 loss=3.2821 lr=4.32e-05 tok=18.435B dt=24843 wait=243 +19:18:09 step 140100 loss=3.3447 lr=4.32e-05 tok=18.438B dt=24906 wait=244 +19:18:25 pushed ckpt_w128.pt,log_w128.txt +19:22:29 step 140120 loss=3.3727 lr=4.32e-05 tok=18.441B dt=25166 wait=244 +19:23:32 step 140140 loss=3.3999 lr=4.31e-05 tok=18.443B dt=25229 wait=244 +19:24:35 step 140160 loss=3.2800 lr=4.31e-05 tok=18.446B dt=25292 wait=244 +19:25:37 step 140180 loss=3.3236 lr=4.31e-05 tok=18.448B dt=25355 wait=245 +19:26:40 step 140200 loss=3.2547 lr=4.31e-05 tok=18.451B dt=25417 wait=245 +19:27:43 step 140220 loss=3.4561 lr=4.31e-05 tok=18.454B dt=25480 wait=245 +19:28:45 step 140240 loss=3.3860 lr=4.30e-05 tok=18.456B dt=25542 wait=246 +19:29:30 pushed ckpt_w128.pt,log_w128.txt +19:33:04 step 140260 loss=3.4314 lr=4.30e-05 tok=18.459B dt=25801 wait=246 +19:34:06 step 140280 loss=3.4384 lr=4.30e-05 tok=18.461B dt=25863 wait=246 +19:35:09 step 140300 loss=3.3583 lr=4.30e-05 tok=18.464B dt=25926 wait=247 +19:36:11 step 140320 loss=3.3807 lr=4.30e-05 tok=18.467B dt=25988 wait=247 +19:37:15 step 140340 loss=3.4261 lr=4.30e-05 tok=18.469B dt=26052 wait=247 +19:38:17 step 140360 loss=3.3904 lr=4.29e-05 tok=18.472B dt=26114 wait=248 +19:39:20 step 140380 loss=3.3712 lr=4.29e-05 tok=18.475B dt=26177 wait=248 +19:40:23 step 140400 loss=3.4137 lr=4.29e-05 tok=18.477B dt=26240 wait=248 +19:40:38 pushed ckpt_w128.pt,log_w128.txt +19:43:25 waited 166s for global r=537 +19:43:33 global r=537 load 4.300 -> 4.316 +19:44:05 step 140260 loss=3.4143 lr=4.30e-05 tok=18.479B dt=26462 wait=248 +19:45:16 step 140280 loss=3.3343 lr=4.30e-05 tok=18.481B dt=26533 wait=249 +19:46:19 step 140300 loss=3.3667 lr=4.30e-05 tok=18.484B dt=26596 wait=249 +19:47:22 step 140320 loss=3.4614 lr=4.30e-05 tok=18.486B dt=26659 wait=249 +19:48:25 step 140340 loss=3.3735 lr=4.30e-05 tok=18.489B dt=26722 wait=250 +19:49:27 step 140360 loss=3.3652 lr=4.29e-05 tok=18.492B dt=26784 wait=250 +19:50:31 step 140380 loss=3.3602 lr=4.29e-05 tok=18.494B dt=26848 wait=250 +19:51:33 step 140400 loss=3.3760 lr=4.29e-05 tok=18.497B dt=26910 wait=251 +19:51:46 pushed ckpt_w128.pt,log_w128.txt +19:55:51 step 140420 loss=3.4734 lr=4.29e-05 tok=18.500B dt=27168 wait=251 +19:56:54 step 140440 loss=3.3299 lr=4.29e-05 tok=18.502B dt=27231 wait=251 +19:57:57 step 140460 loss=3.2763 lr=4.29e-05 tok=18.505B dt=27294 wait=252 +19:58:06 global r=538 load 4.304 -> 4.292 +19:59:16 step 140420 loss=3.2081 lr=4.29e-05 tok=18.507B dt=27373 wait=252 +20:00:33 step 140440 loss=3.3666 lr=4.29e-05 tok=18.510B dt=27450 wait=253 +20:01:35 step 140460 loss=3.4038 lr=4.29e-05 tok=18.513B dt=27512 wait=253 +20:02:38 step 140480 loss=3.3614 lr=4.28e-05 tok=18.515B dt=27575 wait=253 +20:03:41 step 140500 loss=3.3935 lr=4.28e-05 tok=18.518B dt=27638 wait=253 +20:04:43 step 140520 loss=3.4776 lr=4.28e-05 tok=18.520B dt=27700 wait=254 +20:05:46 step 140540 loss=3.3127 lr=4.28e-05 tok=18.523B dt=27763 wait=254 +20:06:36 pushed ckpt_w128.pt,log_w128.txt +20:07:21 waited 45s for global r=539 +20:07:29 global r=539 load 4.286 -> 4.291 +20:08:02 step 140560 loss=3.3599 lr=4.28e-05 tok=18.526B dt=27899 wait=254 +20:09:12 step 140580 loss=3.3749 lr=4.28e-05 tok=18.528B dt=27969 wait=255 +20:10:13 mix source 1 lap 1 +20:10:27 step 140600 loss=3.3031 lr=4.27e-05 tok=18.531B dt=28044 wait=255 +20:11:30 step 140620 loss=3.3771 lr=4.27e-05 tok=18.534B dt=28107 wait=255 +20:12:33 step 140640 loss=3.2936 lr=4.27e-05 tok=18.536B dt=28170 wait=256 +20:13:35 step 140660 loss=3.4050 lr=4.27e-05 tok=18.539B dt=28232 wait=256 +20:14:38 step 140680 loss=3.2818 lr=4.27e-05 tok=18.541B dt=28295 wait=256 +20:15:42 step 140700 loss=3.3328 lr=4.27e-05 tok=18.544B dt=28359 wait=257 +20:16:02 pushed ckpt_w128.pt,log_w128.txt +20:16:33 waited 30s for global r=540 +20:16:50 global r=540 load 4.297 -> 4.293 +20:17:54 step 140720 loss=3.3310 lr=4.26e-05 tok=18.547B dt=28491 wait=257 +20:19:15 step 140740 loss=3.3949 lr=4.26e-05 tok=18.549B dt=28572 wait=257 +20:20:20 step 140760 loss=3.3796 lr=4.26e-05 tok=18.552B dt=28637 wait=258 +20:21:22 step 140780 loss=3.3714 lr=4.26e-05 tok=18.555B dt=28699 wait=258 +20:22:25 step 140800 loss=3.3284 lr=4.26e-05 tok=18.557B dt=28762 wait=258 +20:23:28 step 140820 loss=3.4138 lr=4.26e-05 tok=18.560B dt=28825 wait=259 +20:24:31 step 140840 loss=3.3231 lr=4.25e-05 tok=18.562B dt=28888 wait=259 +20:25:18 pushed ckpt_w128.pt,log_w128.txt +20:26:04 waited 45s for global r=541 +20:26:12 global r=541 load 4.295 -> 4.297 +20:26:45 step 140860 loss=3.4938 lr=4.25e-05 tok=18.565B dt=29022 wait=259 +20:27:54 step 140880 loss=3.3033 lr=4.25e-05 tok=18.568B dt=29091 wait=260 +20:29:10 step 140900 loss=3.3738 lr=4.25e-05 tok=18.570B dt=29167 wait=260 +20:30:13 step 140920 loss=3.4601 lr=4.25e-05 tok=18.573B dt=29230 wait=260 +20:31:15 step 140940 loss=3.3834 lr=4.24e-05 tok=18.576B dt=29292 wait=261 +20:32:17 step 140960 loss=3.4314 lr=4.24e-05 tok=18.578B dt=29354 wait=261 +20:33:19 step 140980 loss=3.3704 lr=4.24e-05 tok=18.581B dt=29416 wait=261 +20:34:22 step 141000 loss=3.3831 lr=4.24e-05 tok=18.583B dt=29479 wait=262 +20:34:45 pushed ckpt_w128.pt,log_w128.txt +20:35:31 waited 45s for global r=542 +20:35:39 global r=542 load 4.282 -> 4.283 +20:36:44 step 141020 loss=3.3576 lr=4.24e-05 tok=18.586B dt=29621 wait=262 +20:38:00 step 141040 loss=3.4198 lr=4.24e-05 tok=18.589B dt=29697 wait=262 +20:39:09 step 141060 loss=3.4031 lr=4.23e-05 tok=18.591B dt=29766 wait=263 +20:40:12 step 141080 loss=3.3483 lr=4.23e-05 tok=18.594B dt=29829 wait=263 +20:41:14 step 141100 loss=3.3557 lr=4.23e-05 tok=18.596B dt=29891 wait=263 +20:42:16 step 141120 loss=3.5019 lr=4.23e-05 tok=18.599B dt=29953 wait=264 +20:43:19 step 141140 loss=3.3752 lr=4.23e-05 tok=18.602B dt=30016 wait=264 +20:44:09 pushed ckpt_w128.pt,log_w128.txt +20:44:55 waited 46s for global r=543 +20:45:06 global r=543 load 4.281 -> 4.295 +20:45:38 step 141160 loss=3.5122 lr=4.23e-05 tok=18.604B dt=30156 wait=264 +20:46:45 step 141180 loss=3.3655 lr=4.22e-05 tok=18.607B dt=30222 wait=265 +20:48:04 step 141200 loss=3.3390 lr=4.22e-05 tok=18.610B dt=30301 wait=265 +20:49:07 step 141220 loss=3.4752 lr=4.22e-05 tok=18.612B dt=30364 wait=265 +20:50:10 step 141240 loss=3.2667 lr=4.22e-05 tok=18.615B dt=30427 wait=266 +20:51:13 step 141260 loss=3.4388 lr=4.22e-05 tok=18.617B dt=30490 wait=266 +20:52:16 step 141280 loss=3.2875 lr=4.22e-05 tok=18.620B dt=30553 wait=266 +20:53:19 step 141300 loss=3.3190 lr=4.21e-05 tok=18.623B dt=30616 wait=266 +20:53:39 pushed ckpt_w128.pt,log_w128.txt +20:54:25 waited 46s for global r=544 +20:54:36 global r=544 load 4.284 -> 4.291 +20:55:40 step 141320 loss=3.2922 lr=4.21e-05 tok=18.625B dt=30757 wait=267 +20:56:51 step 141340 loss=3.3758 lr=4.21e-05 tok=18.628B dt=30828 wait=267 +20:58:05 step 141360 loss=3.3740 lr=4.21e-05 tok=18.631B dt=30902 wait=268 +20:59:08 step 141380 loss=3.3184 lr=4.21e-05 tok=18.633B dt=30965 wait=268 +21:00:11 step 141400 loss=3.3340 lr=4.21e-05 tok=18.636B dt=31028 wait=268 +21:01:14 step 141420 loss=3.4759 lr=4.20e-05 tok=18.638B dt=31091 wait=268 +21:02:17 step 141440 loss=3.3036 lr=4.20e-05 tok=18.641B dt=31154 wait=269 +21:03:15 pushed ckpt_w128.pt,log_w128.txt +21:04:01 waited 46s for global r=545 +21:04:14 global r=545 load 4.291 -> 4.292 +21:04:47 step 141460 loss=3.4652 lr=4.20e-05 tok=18.644B dt=31304 wait=269 +21:05:55 step 141480 loss=3.3368 lr=4.20e-05 tok=18.646B dt=31372 wait=269 +21:07:12 step 141500 loss=3.3694 lr=4.20e-05 tok=18.649B dt=31449 wait=270 +21:08:15 step 141520 loss=3.4573 lr=4.20e-05 tok=18.652B dt=31512 wait=270 +21:09:17 step 141540 loss=3.3358 lr=4.19e-05 tok=18.654B dt=31574 wait=270 +21:10:20 step 141560 loss=3.4207 lr=4.19e-05 tok=18.657B dt=31637 wait=271 +21:11:23 step 141580 loss=3.3018 lr=4.19e-05 tok=18.659B dt=31700 wait=271 +21:12:25 step 141600 loss=3.4551 lr=4.19e-05 tok=18.662B dt=31762 wait=271 +21:12:42 pushed ckpt_w128.pt,log_w128.txt +21:13:28 waited 45s for global r=546 +21:13:37 global r=546 load 4.298 -> 4.295 +21:14:41 step 141620 loss=3.3118 lr=4.19e-05 tok=18.665B dt=31898 wait=272 +21:15:52 step 141640 loss=3.3788 lr=4.19e-05 tok=18.667B dt=31969 wait=272 +21:17:07 step 141660 loss=3.3258 lr=4.18e-05 tok=18.670B dt=32044 wait=272 +21:18:09 step 141680 loss=3.3045 lr=4.18e-05 tok=18.673B dt=32106 wait=273 +21:19:12 step 141700 loss=3.3309 lr=4.18e-05 tok=18.675B dt=32169 wait=273 +21:20:14 step 141720 loss=3.4040 lr=4.18e-05 tok=18.678B dt=32232 wait=273 +21:21:17 step 141740 loss=3.3789 lr=4.18e-05 tok=18.680B dt=32294 wait=274 +21:22:12 pushed ckpt_w128.pt,log_w128.txt +21:22:57 waited 46s for global r=547 +21:23:08 global r=547 load 4.291 -> 4.288 +21:23:41 step 141760 loss=3.4202 lr=4.18e-05 tok=18.683B dt=32438 wait=274 +21:24:43 step 141780 loss=3.3369 lr=4.17e-05 tok=18.686B dt=32500 wait=274 +21:26:06 step 141800 loss=3.2957 lr=4.17e-05 tok=18.688B dt=32583 wait=275 +21:27:09 step 141820 loss=3.4127 lr=4.17e-05 tok=18.691B dt=32646 wait=275 +21:28:12 step 141840 loss=3.2779 lr=4.17e-05 tok=18.693B dt=32709 wait=275 +21:29:15 step 141860 loss=3.4366 lr=4.17e-05 tok=18.696B dt=32772 wait=276 +21:30:18 step 141880 loss=3.3167 lr=4.17e-05 tok=18.699B dt=32835 wait=276 +21:31:21 step 141900 loss=3.3951 lr=4.16e-05 tok=18.701B dt=32898 wait=276 +21:31:39 pushed ckpt_w128.pt,log_w128.txt +21:32:55 waited 76s for global r=548 +21:33:15 global r=548 load 4.308 -> 4.289 +21:34:18 step 141920 loss=3.3455 lr=4.16e-05 tok=18.704B dt=33075 wait=277 +21:35:41 step 141940 loss=3.4169 lr=4.16e-05 tok=18.707B dt=33158 wait=277 +21:36:43 step 141960 loss=3.4145 lr=4.16e-05 tok=18.709B dt=33220 wait=277 +21:37:47 step 141980 loss=3.3205 lr=4.16e-05 tok=18.712B dt=33284 wait=278 +21:38:49 step 142000 loss=3.3684 lr=4.15e-05 tok=18.714B dt=33347 wait=278 +21:39:52 step 142020 loss=3.4244 lr=4.15e-05 tok=18.717B dt=33409 wait=278 +21:40:55 step 142040 loss=3.2740 lr=4.15e-05 tok=18.720B dt=33472 wait=279 +21:41:46 pushed ckpt_w128.pt,log_w128.txt +21:42:32 waited 46s for global r=549 +21:42:41 global r=549 load 4.301 -> 4.295 +21:43:14 step 142060 loss=3.4138 lr=4.15e-05 tok=18.722B dt=33611 wait=279 +21:44:25 step 142080 loss=3.3547 lr=4.15e-05 tok=18.725B dt=33682 wait=279 +21:45:40 step 142100 loss=3.3191 lr=4.15e-05 tok=18.728B dt=33757 wait=280 +21:46:43 step 142120 loss=3.4001 lr=4.14e-05 tok=18.730B dt=33820 wait=280 +21:47:46 step 142140 loss=3.2925 lr=4.14e-05 tok=18.733B dt=33883 wait=280 +21:48:49 step 142160 loss=3.4064 lr=4.14e-05 tok=18.735B dt=33946 wait=280 +21:49:52 step 142180 loss=3.2903 lr=4.14e-05 tok=18.738B dt=34009 wait=281 +21:50:55 step 142200 loss=3.4406 lr=4.14e-05 tok=18.741B dt=34072 wait=281 +21:51:15 pushed ckpt_w128.pt,log_w128.txt +21:51:46 waited 30s for global r=550 +21:52:03 global r=550 load 4.292 -> 4.295 +21:53:07 step 142220 loss=3.3617 lr=4.14e-05 tok=18.743B dt=34204 wait=281 +21:54:25 step 142240 loss=3.3646 lr=4.13e-05 tok=18.746B dt=34282 wait=282 +21:55:33 step 142260 loss=3.4054 lr=4.13e-05 tok=18.749B dt=34350 wait=282 +21:56:35 step 142280 loss=3.3720 lr=4.13e-05 tok=18.751B dt=34412 wait=282 +21:57:38 step 142300 loss=3.3041 lr=4.13e-05 tok=18.754B dt=34475 wait=283 +21:58:40 step 142320 loss=3.4685 lr=4.13e-05 tok=18.756B dt=34537 wait=283 +21:59:43 step 142340 loss=3.2837 lr=4.13e-05 tok=18.759B dt=34600 wait=283 +22:00:31 pushed ckpt_w128.pt,log_w128.txt +22:01:16 waited 46s for global r=551 +22:01:28 global r=551 load 4.273 -> 4.284 +22:02:01 step 142360 loss=3.4526 lr=4.12e-05 tok=18.762B dt=34738 wait=284 +22:03:09 step 142380 loss=3.3289 lr=4.12e-05 tok=18.764B dt=34806 wait=284 +22:04:26 step 142400 loss=3.1813 lr=4.12e-05 tok=18.767B dt=34883 wait=284 +22:05:30 step 142420 loss=3.2938 lr=4.12e-05 tok=18.770B dt=34947 wait=285 +22:06:33 step 142440 loss=3.3376 lr=4.12e-05 tok=18.772B dt=35010 wait=285 +22:07:35 step 142460 loss=3.4338 lr=4.12e-05 tok=18.775B dt=35072 wait=285 +22:08:38 step 142480 loss=3.3427 lr=4.11e-05 tok=18.777B dt=35135 wait=286 +22:09:41 step 142500 loss=3.3192 lr=4.11e-05 tok=18.780B dt=35198 wait=286 +22:09:53 pushed ckpt_w128.pt,log_w128.txt +22:12:55 waited 182s for global r=552 +22:13:03 global r=552 load 4.313 -> 4.296 +22:14:07 step 142520 loss=3.3073 lr=4.11e-05 tok=18.783B dt=35464 wait=286 +22:15:23 step 142540 loss=3.3863 lr=4.11e-05 tok=18.785B dt=35540 wait=287 +22:16:32 step 142560 loss=3.3092 lr=4.11e-05 tok=18.788B dt=35609 wait=287 +22:17:34 step 142580 loss=3.3560 lr=4.11e-05 tok=18.790B dt=35671 wait=287 +22:18:37 step 142600 loss=3.3987 lr=4.10e-05 tok=18.793B dt=35734 wait=288 +22:19:40 step 142620 loss=3.4242 lr=4.10e-05 tok=18.796B dt=35797 wait=288 +22:20:43 step 142640 loss=3.3063 lr=4.10e-05 tok=18.798B dt=35860 wait=288 +22:21:28 pushed ckpt_w128.pt,log_w128.txt +22:22:59 waited 91s for global r=553 +22:23:06 global r=553 load 4.299 -> 4.276 +22:23:39 step 142660 loss=3.4457 lr=4.10e-05 tok=18.801B dt=36036 wait=289 +22:24:55 step 142680 loss=3.3768 lr=4.10e-05 tok=18.804B dt=36112 wait=289 +22:26:02 step 142700 loss=3.3618 lr=4.10e-05 tok=18.806B dt=36180 wait=289 +22:27:05 step 142720 loss=3.4007 lr=4.09e-05 tok=18.809B dt=36242 wait=290 +22:28:07 step 142740 loss=3.3576 lr=4.09e-05 tok=18.811B dt=36305 wait=290 +22:29:10 step 142760 loss=3.4104 lr=4.09e-05 tok=18.814B dt=36367 wait=290 +22:30:12 step 142780 loss=3.3239 lr=4.09e-05 tok=18.817B dt=36429 wait=291 +22:31:15 step 142800 loss=3.3953 lr=4.09e-05 tok=18.819B dt=36492 wait=291 diff --git a/log_w129.txt b/log_w129.txt new file mode 100644 index 0000000000000000000000000000000000000000..aa3694e95a723d72ada2b2dd9253ecef05b881dc --- /dev/null +++ b/log_w129.txt @@ -0,0 +1,334 @@ +launch +12:58:48 name=w129 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 76/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:58:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:58:54 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:59:04 params 1.08B +12:59:05 resumed step=134400 seen=17.70B round=502 +12:59:05 stream salt=134400 +13:00:53 blocks compiled +13:00:53 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:03:26 step 134401 loss=3.0659 lr=4.82e-05 tok=17.697B dt=153 wait=112 +13:03:50 step 134402 loss=3.2210 lr=4.82e-05 tok=17.698B dt=177 wait=112 +13:04:53 step 134420 loss=3.4344 lr=4.82e-05 tok=17.700B dt=239 wait=113 +13:05:57 step 134440 loss=3.3886 lr=4.82e-05 tok=17.703B dt=303 wait=113 +13:06:14 global r=503 load 4.279 -> 4.283 +13:06:46 step 134560 loss=3.4339 lr=4.81e-05 tok=17.704B dt=352 wait=113 +13:07:48 step 134580 loss=3.3960 lr=4.81e-05 tok=17.707B dt=415 wait=114 +13:08:51 step 134600 loss=3.4106 lr=4.80e-05 tok=17.709B dt=477 wait=114 +13:09:53 step 134620 loss=3.5101 lr=4.80e-05 tok=17.712B dt=540 wait=114 +13:10:56 step 134640 loss=3.3278 lr=4.80e-05 tok=17.714B dt=603 wait=115 +13:11:59 step 134660 loss=3.4747 lr=4.80e-05 tok=17.717B dt=666 wait=115 +13:13:02 step 134680 loss=3.3899 lr=4.80e-05 tok=17.720B dt=729 wait=115 +13:14:05 step 134700 loss=3.4350 lr=4.80e-05 tok=17.722B dt=792 wait=116 +13:14:18 pushed ckpt_w129.pt,log_w129.txt +13:15:18 waited 60s for global r=504 +13:15:27 global r=504 load 4.283 -> 4.279 +13:16:31 step 134720 loss=3.3631 lr=4.79e-05 tok=17.725B dt=938 wait=116 +13:17:34 step 134740 loss=3.3830 lr=4.79e-05 tok=17.727B dt=1001 wait=116 +13:18:37 step 134760 loss=3.3829 lr=4.79e-05 tok=17.730B dt=1063 wait=117 +13:19:40 step 134780 loss=3.3694 lr=4.79e-05 tok=17.733B dt=1126 wait=117 +13:20:42 step 134800 loss=3.3533 lr=4.79e-05 tok=17.735B dt=1189 wait=117 +13:21:45 step 134820 loss=3.4098 lr=4.78e-05 tok=17.738B dt=1251 wait=118 +13:22:48 step 134840 loss=3.3319 lr=4.78e-05 tok=17.741B dt=1314 wait=118 +13:23:31 pushed ckpt_w129.pt,log_w129.txt +13:24:46 waited 76s for global r=505 +13:24:56 global r=505 load 4.294 -> 4.282 +13:25:29 step 134860 loss=3.4775 lr=4.78e-05 tok=17.743B dt=1475 wait=118 +13:26:32 step 134880 loss=3.3654 lr=4.78e-05 tok=17.746B dt=1538 wait=119 +13:27:34 step 134900 loss=3.3300 lr=4.78e-05 tok=17.748B dt=1600 wait=119 +13:28:36 step 134920 loss=3.4484 lr=4.78e-05 tok=17.751B dt=1663 wait=119 +13:29:39 step 134940 loss=3.2934 lr=4.77e-05 tok=17.754B dt=1726 wait=120 +13:30:42 step 134960 loss=3.3982 lr=4.77e-05 tok=17.756B dt=1789 wait=120 +13:31:44 step 134980 loss=3.3023 lr=4.77e-05 tok=17.759B dt=1851 wait=120 +13:32:47 step 135000 loss=3.3006 lr=4.77e-05 tok=17.762B dt=1914 wait=121 +13:33:01 pushed ckpt_w129.pt,log_w129.txt +13:34:32 waited 91s for global r=506 +13:34:40 global r=506 load 4.281 -> 4.282 +13:35:45 step 135020 loss=3.2324 lr=4.77e-05 tok=17.764B dt=2091 wait=121 +13:36:48 step 135040 loss=3.4115 lr=4.76e-05 tok=17.767B dt=2154 wait=121 +13:37:51 step 135060 loss=3.4448 lr=4.76e-05 tok=17.769B dt=2218 wait=122 +13:38:54 step 135080 loss=3.4014 lr=4.76e-05 tok=17.772B dt=2281 wait=122 +13:39:57 step 135100 loss=3.3707 lr=4.76e-05 tok=17.775B dt=2343 wait=122 +13:40:59 step 135120 loss=3.4579 lr=4.76e-05 tok=17.777B dt=2406 wait=122 +13:42:02 step 135140 loss=3.3118 lr=4.76e-05 tok=17.780B dt=2469 wait=123 +13:42:49 pushed ckpt_w129.pt,log_w129.txt +13:45:55 waited 186s for global r=507 +13:46:03 global r=507 load 4.265 -> 4.282 +13:46:36 step 135160 loss=3.5314 lr=4.75e-05 tok=17.783B dt=2743 wait=123 +13:47:39 step 135180 loss=3.4349 lr=4.75e-05 tok=17.785B dt=2805 wait=123 +13:48:41 step 135200 loss=3.3066 lr=4.75e-05 tok=17.788B dt=2868 wait=124 +13:49:44 step 135220 loss=3.4039 lr=4.75e-05 tok=17.790B dt=2931 wait=124 +13:50:48 step 135240 loss=3.3180 lr=4.75e-05 tok=17.793B dt=2994 wait=124 +13:51:50 step 135260 loss=3.4461 lr=4.74e-05 tok=17.796B dt=3057 wait=125 +13:52:52 step 135280 loss=3.3431 lr=4.74e-05 tok=17.798B dt=3119 wait=125 +13:53:55 step 135300 loss=3.3777 lr=4.74e-05 tok=17.801B dt=3182 wait=125 +13:54:07 pushed ckpt_w129.pt,log_w129.txt +13:55:53 waited 106s for global r=508 +13:55:59 global r=508 load 4.274 -> 4.283 +13:57:04 step 135320 loss=3.3398 lr=4.74e-05 tok=17.804B dt=3371 wait=126 +13:58:06 step 135340 loss=3.4081 lr=4.74e-05 tok=17.806B dt=3433 wait=126 +13:59:10 step 135360 loss=3.3654 lr=4.74e-05 tok=17.809B dt=3496 wait=126 +14:00:12 step 135380 loss=3.4272 lr=4.73e-05 tok=17.811B dt=3559 wait=127 +14:01:15 step 135400 loss=3.3551 lr=4.73e-05 tok=17.814B dt=3622 wait=127 +14:02:18 step 135420 loss=3.4166 lr=4.73e-05 tok=17.817B dt=3684 wait=127 +14:03:20 step 135440 loss=3.3165 lr=4.73e-05 tok=17.819B dt=3746 wait=128 +14:04:03 pushed ckpt_w129.pt,log_w129.txt +14:05:18 waited 76s for global r=509 +14:05:25 global r=509 load 4.311 -> 4.282 +14:05:58 step 135460 loss=3.4710 lr=4.73e-05 tok=17.822B dt=3904 wait=128 +14:07:00 step 135480 loss=3.3545 lr=4.73e-05 tok=17.824B dt=3967 wait=128 +14:08:05 step 135500 loss=3.3386 lr=4.72e-05 tok=17.827B dt=4032 wait=129 +14:09:08 step 135520 loss=3.4847 lr=4.72e-05 tok=17.830B dt=4094 wait=129 +14:10:10 step 135540 loss=3.3297 lr=4.72e-05 tok=17.832B dt=4157 wait=129 +14:11:13 step 135560 loss=3.4830 lr=4.72e-05 tok=17.835B dt=4220 wait=129 +14:12:16 step 135580 loss=3.3819 lr=4.72e-05 tok=17.838B dt=4282 wait=130 +14:13:18 step 135600 loss=3.4240 lr=4.71e-05 tok=17.840B dt=4345 wait=130 +14:13:32 pushed ckpt_w129.pt,log_w129.txt +14:14:47 waited 76s for global r=510 +14:14:56 global r=510 load 4.287 -> 4.291 +14:16:00 step 135620 loss=3.4032 lr=4.71e-05 tok=17.843B dt=4507 wait=130 +14:17:03 step 135640 loss=3.3885 lr=4.71e-05 tok=17.845B dt=4569 wait=131 +14:18:06 step 135660 loss=3.4077 lr=4.71e-05 tok=17.848B dt=4633 wait=131 +14:19:09 step 135680 loss=3.4084 lr=4.71e-05 tok=17.851B dt=4695 wait=131 +14:20:12 step 135700 loss=3.3439 lr=4.71e-05 tok=17.853B dt=4759 wait=132 +14:21:15 step 135720 loss=3.4075 lr=4.70e-05 tok=17.856B dt=4822 wait=132 +14:22:17 step 135740 loss=3.3073 lr=4.70e-05 tok=17.859B dt=4884 wait=132 +14:23:00 pushed ckpt_w129.pt,log_w129.txt +14:24:16 waited 76s for global r=511 +14:24:22 global r=511 load 4.275 -> 4.289 +14:24:55 step 135760 loss=3.3484 lr=4.70e-05 tok=17.861B dt=5042 wait=133 +14:25:58 step 135780 loss=3.2092 lr=4.70e-05 tok=17.864B dt=5105 wait=133 +14:27:02 step 135800 loss=3.3215 lr=4.70e-05 tok=17.866B dt=5168 wait=133 +14:28:04 step 135820 loss=3.4301 lr=4.69e-05 tok=17.869B dt=5231 wait=134 +14:29:07 step 135840 loss=3.3593 lr=4.69e-05 tok=17.872B dt=5294 wait=134 +14:30:09 step 135860 loss=3.4383 lr=4.69e-05 tok=17.874B dt=5356 wait=134 +14:31:12 step 135880 loss=3.3852 lr=4.69e-05 tok=17.877B dt=5419 wait=135 +14:32:15 step 135900 loss=3.4247 lr=4.69e-05 tok=17.880B dt=5482 wait=135 +14:32:29 pushed ckpt_w129.pt,log_w129.txt +14:33:59 waited 91s for global r=512 +14:34:05 global r=512 load 4.313 -> 4.296 +14:35:09 step 135920 loss=3.3011 lr=4.69e-05 tok=17.882B dt=5655 wait=135 +14:36:11 step 135940 loss=3.4007 lr=4.68e-05 tok=17.885B dt=5718 wait=136 +14:37:14 step 135960 loss=3.3708 lr=4.68e-05 tok=17.887B dt=5781 wait=136 +14:38:17 step 135980 loss=3.3222 lr=4.68e-05 tok=17.890B dt=5844 wait=136 +14:39:20 step 136000 loss=3.3639 lr=4.68e-05 tok=17.893B dt=5906 wait=137 +14:40:23 step 136020 loss=3.4021 lr=4.68e-05 tok=17.895B dt=5969 wait=137 +14:41:25 step 136040 loss=3.3603 lr=4.67e-05 tok=17.898B dt=6032 wait=137 +14:42:09 pushed ckpt_w129.pt,log_w129.txt +14:43:24 waited 76s for global r=513 +14:43:34 global r=513 load 4.306 -> 4.292 +14:44:06 step 136060 loss=3.4595 lr=4.67e-05 tok=17.901B dt=6193 wait=138 +14:45:09 step 136080 loss=3.3422 lr=4.67e-05 tok=17.903B dt=6255 wait=138 +14:46:12 step 136100 loss=3.3419 lr=4.67e-05 tok=17.906B dt=6319 wait=138 +14:47:15 step 136120 loss=3.4680 lr=4.67e-05 tok=17.908B dt=6381 wait=138 +14:48:18 step 136140 loss=3.2447 lr=4.67e-05 tok=17.911B dt=6444 wait=139 +14:49:20 step 136160 loss=3.3867 lr=4.66e-05 tok=17.914B dt=6507 wait=139 +14:50:23 step 136180 loss=3.3419 lr=4.66e-05 tok=17.916B dt=6569 wait=139 +14:51:25 step 136200 loss=3.4110 lr=4.66e-05 tok=17.919B dt=6632 wait=140 +14:51:39 pushed ckpt_w129.pt,log_w129.txt +14:52:55 waited 76s for global r=514 +14:53:02 global r=514 load 4.304 -> 4.295 +14:54:06 step 136220 loss=3.3152 lr=4.66e-05 tok=17.921B dt=6793 wait=140 +14:55:10 step 136240 loss=3.3572 lr=4.66e-05 tok=17.924B dt=6856 wait=140 +14:56:12 step 136260 loss=3.3222 lr=4.65e-05 tok=17.927B dt=6919 wait=141 +14:57:15 step 136280 loss=3.3574 lr=4.65e-05 tok=17.929B dt=6982 wait=141 +14:58:18 step 136300 loss=3.3460 lr=4.65e-05 tok=17.932B dt=7045 wait=141 +14:59:21 step 136320 loss=3.4058 lr=4.65e-05 tok=17.935B dt=7108 wait=142 +15:00:24 step 136340 loss=3.2499 lr=4.65e-05 tok=17.937B dt=7170 wait=142 +15:01:08 pushed ckpt_w129.pt,log_w129.txt +15:02:23 waited 76s for global r=515 +15:02:29 global r=515 load 4.311 -> 4.297 +15:03:02 step 136360 loss=3.4596 lr=4.65e-05 tok=17.940B dt=7328 wait=142 +15:04:04 step 136380 loss=3.3546 lr=4.64e-05 tok=17.942B dt=7391 wait=143 +15:05:07 step 136400 loss=3.3266 lr=4.64e-05 tok=17.945B dt=7454 wait=143 +15:06:10 step 136420 loss=3.4393 lr=4.64e-05 tok=17.948B dt=7517 wait=143 +15:07:13 step 136440 loss=3.3822 lr=4.64e-05 tok=17.950B dt=7579 wait=143 +15:08:16 step 136460 loss=3.4624 lr=4.64e-05 tok=17.953B dt=7642 wait=144 +15:09:19 step 136480 loss=3.4057 lr=4.64e-05 tok=17.956B dt=7705 wait=144 +15:10:21 step 136500 loss=3.4085 lr=4.63e-05 tok=17.958B dt=7768 wait=144 +15:10:33 pushed ckpt_w129.pt,log_w129.txt +15:11:49 waited 76s for global r=516 +15:11:55 global r=516 load 4.307 -> 4.298 +15:12:58 step 136520 loss=3.2830 lr=4.63e-05 tok=17.961B dt=7925 wait=145 +15:14:01 step 136540 loss=3.3975 lr=4.63e-05 tok=17.963B dt=7988 wait=145 +15:15:04 step 136560 loss=3.3366 lr=4.63e-05 tok=17.966B dt=8051 wait=145 +15:16:07 step 136580 loss=3.3423 lr=4.63e-05 tok=17.969B dt=8113 wait=146 +15:17:09 step 136600 loss=3.3122 lr=4.62e-05 tok=17.971B dt=8176 wait=146 +15:18:12 step 136620 loss=3.4318 lr=4.62e-05 tok=17.974B dt=8239 wait=146 +15:19:15 step 136640 loss=3.3675 lr=4.62e-05 tok=17.977B dt=8301 wait=147 +15:19:58 pushed ckpt_w129.pt,log_w129.txt +15:21:29 waited 91s for global r=517 +15:21:39 global r=517 load 4.299 -> 4.295 +15:22:12 step 136660 loss=3.3390 lr=4.62e-05 tok=17.979B dt=8478 wait=147 +15:23:15 step 136680 loss=3.3083 lr=4.62e-05 tok=17.982B dt=8541 wait=147 +15:24:18 step 136700 loss=3.2720 lr=4.62e-05 tok=17.984B dt=8604 wait=148 +15:25:20 step 136720 loss=3.3487 lr=4.61e-05 tok=17.987B dt=8667 wait=148 +15:26:24 step 136740 loss=3.2127 lr=4.61e-05 tok=17.990B dt=8730 wait=148 +15:27:27 step 136760 loss=3.4054 lr=4.61e-05 tok=17.992B dt=8794 wait=148 +15:28:30 step 136780 loss=3.3600 lr=4.61e-05 tok=17.995B dt=8856 wait=149 +15:29:33 step 136800 loss=3.3632 lr=4.61e-05 tok=17.997B dt=8919 wait=149 +15:29:44 pushed ckpt_w129.pt,log_w129.txt +15:31:00 waited 76s for global r=518 +15:31:13 global r=518 load 4.296 -> 4.283 +15:32:17 step 136820 loss=3.1579 lr=4.60e-05 tok=18.000B dt=9084 wait=149 +15:33:20 step 136840 loss=3.3810 lr=4.60e-05 tok=18.003B dt=9147 wait=150 +15:34:23 step 136860 loss=3.3796 lr=4.60e-05 tok=18.005B dt=9210 wait=150 +15:35:26 step 136880 loss=3.2485 lr=4.60e-05 tok=18.008B dt=9273 wait=150 +15:36:29 step 136900 loss=3.4117 lr=4.60e-05 tok=18.011B dt=9335 wait=151 +15:37:32 step 136920 loss=3.4361 lr=4.60e-05 tok=18.013B dt=9398 wait=151 +15:38:34 step 136940 loss=3.3289 lr=4.59e-05 tok=18.016B dt=9461 wait=151 +15:39:19 pushed ckpt_w129.pt,log_w129.txt +15:40:35 waited 76s for global r=519 +15:40:45 global r=519 load 4.307 -> 4.288 +15:41:18 step 136960 loss=3.4104 lr=4.59e-05 tok=18.018B dt=9624 wait=152 +15:42:21 step 136980 loss=3.3086 lr=4.59e-05 tok=18.021B dt=9688 wait=152 +15:43:23 step 137000 loss=3.3036 lr=4.59e-05 tok=18.024B dt=9750 wait=152 +15:44:26 step 137020 loss=3.3412 lr=4.59e-05 tok=18.026B dt=9813 wait=153 +15:45:29 step 137040 loss=3.3570 lr=4.59e-05 tok=18.029B dt=9876 wait=153 +15:46:32 step 137060 loss=3.4010 lr=4.58e-05 tok=18.032B dt=9939 wait=153 +15:47:35 step 137080 loss=3.3988 lr=4.58e-05 tok=18.034B dt=10001 wait=154 +15:48:37 step 137100 loss=3.3935 lr=4.58e-05 tok=18.037B dt=10064 wait=154 +15:48:51 pushed ckpt_w129.pt,log_w129.txt +15:49:51 waited 61s for global r=520 +15:50:03 global r=520 load 4.295 -> 4.286 +15:51:07 step 137120 loss=3.3986 lr=4.58e-05 tok=18.039B dt=10213 wait=154 +15:52:10 step 137140 loss=3.4103 lr=4.58e-05 tok=18.042B dt=10276 wait=155 +15:53:13 step 137160 loss=3.4292 lr=4.57e-05 tok=18.045B dt=10339 wait=155 +15:54:15 step 137180 loss=3.3873 lr=4.57e-05 tok=18.047B dt=10402 wait=155 +15:55:19 step 137200 loss=3.4098 lr=4.57e-05 tok=18.050B dt=10465 wait=156 +15:56:22 step 137220 loss=3.4583 lr=4.57e-05 tok=18.053B dt=10528 wait=156 +15:57:25 step 137240 loss=3.3805 lr=4.57e-05 tok=18.055B dt=10591 wait=156 +15:58:11 pushed ckpt_w129.pt,log_w129.txt +16:01:45 step 137260 loss=3.4772 lr=4.57e-05 tok=18.058B dt=10852 wait=157 +16:02:48 step 137280 loss=3.3555 lr=4.56e-05 tok=18.060B dt=10915 wait=157 +16:03:51 step 137300 loss=3.4077 lr=4.56e-05 tok=18.063B dt=10977 wait=157 +16:04:54 step 137320 loss=3.2517 lr=4.56e-05 tok=18.066B dt=11040 wait=157 +16:05:57 step 137340 loss=3.3732 lr=4.56e-05 tok=18.068B dt=11103 wait=158 +16:06:59 step 137360 loss=3.3365 lr=4.56e-05 tok=18.071B dt=11166 wait=158 +16:08:02 step 137380 loss=3.3355 lr=4.55e-05 tok=18.074B dt=11228 wait=158 +16:09:05 step 137400 loss=3.4962 lr=4.55e-05 tok=18.076B dt=11291 wait=159 +16:09:17 pushed ckpt_w129.pt,log_w129.txt +16:13:21 step 137420 loss=3.4155 lr=4.55e-05 tok=18.079B dt=11548 wait=159 +16:14:25 step 137440 loss=3.4002 lr=4.55e-05 tok=18.081B dt=11612 wait=159 +16:15:28 step 137460 loss=3.3872 lr=4.55e-05 tok=18.084B dt=11674 wait=160 +16:16:31 step 137480 loss=3.4260 lr=4.55e-05 tok=18.087B dt=11737 wait=160 +16:17:34 step 137500 loss=3.3985 lr=4.54e-05 tok=18.089B dt=11800 wait=160 +16:18:37 step 137520 loss=3.3707 lr=4.54e-05 tok=18.092B dt=11863 wait=161 +16:19:39 step 137540 loss=3.3294 lr=4.54e-05 tok=18.094B dt=11926 wait=161 +16:20:24 pushed ckpt_w129.pt,log_w129.txt +16:23:58 step 137560 loss=3.3758 lr=4.54e-05 tok=18.097B dt=12184 wait=161 +16:25:00 step 137580 loss=3.3942 lr=4.54e-05 tok=18.100B dt=12247 wait=162 +16:26:03 step 137600 loss=3.4117 lr=4.54e-05 tok=18.102B dt=12310 wait=162 +16:27:06 step 137620 loss=3.3983 lr=4.53e-05 tok=18.105B dt=12372 wait=162 +16:28:08 step 137640 loss=3.3885 lr=4.53e-05 tok=18.108B dt=12434 wait=163 +16:29:11 step 137660 loss=3.4731 lr=4.53e-05 tok=18.110B dt=12497 wait=163 +16:30:14 step 137680 loss=3.4472 lr=4.53e-05 tok=18.113B dt=12561 wait=163 +16:31:17 step 137700 loss=3.2803 lr=4.53e-05 tok=18.115B dt=12623 wait=164 +16:31:30 pushed ckpt_w129.pt,log_w129.txt +16:35:35 step 137720 loss=3.3753 lr=4.52e-05 tok=18.118B dt=12882 wait=164 +16:36:38 step 137740 loss=3.4140 lr=4.52e-05 tok=18.121B dt=12944 wait=164 +16:37:40 step 137760 loss=3.3939 lr=4.52e-05 tok=18.123B dt=13007 wait=165 +16:38:43 step 137780 loss=3.3426 lr=4.52e-05 tok=18.126B dt=13069 wait=165 +16:38:52 global r=521 load 4.285 -> 4.280 +16:39:55 step 137720 loss=3.3395 lr=4.52e-05 tok=18.129B dt=13141 wait=165 +16:40:58 step 137740 loss=3.4317 lr=4.52e-05 tok=18.131B dt=13204 wait=166 +16:42:01 step 137760 loss=3.3760 lr=4.52e-05 tok=18.134B dt=13268 wait=166 +16:43:03 step 137780 loss=3.3152 lr=4.52e-05 tok=18.136B dt=13330 wait=166 +16:44:06 step 137800 loss=3.3736 lr=4.52e-05 tok=18.139B dt=13392 wait=167 +16:45:09 step 137820 loss=3.4763 lr=4.52e-05 tok=18.142B dt=13456 wait=167 +16:46:11 step 137840 loss=3.3515 lr=4.51e-05 tok=18.144B dt=13518 wait=167 +16:46:58 pushed ckpt_w129.pt,log_w129.txt +16:47:43 waited 46s for global r=522 +16:47:50 global r=522 load 4.272 -> 4.274 +16:48:23 step 137860 loss=3.4682 lr=4.51e-05 tok=18.147B dt=13650 wait=167 +16:49:26 step 137880 loss=3.2960 lr=4.51e-05 tok=18.150B dt=13712 wait=168 +16:50:28 step 137900 loss=3.3548 lr=4.51e-05 tok=18.152B dt=13775 wait=168 +16:51:31 step 137920 loss=3.4574 lr=4.51e-05 tok=18.155B dt=13837 wait=168 +16:52:33 step 137940 loss=3.3566 lr=4.51e-05 tok=18.157B dt=13900 wait=169 +16:53:36 step 137960 loss=3.4344 lr=4.50e-05 tok=18.160B dt=13962 wait=169 +16:54:39 step 137980 loss=3.3715 lr=4.50e-05 tok=18.163B dt=14025 wait=169 +16:55:41 step 138000 loss=3.3738 lr=4.50e-05 tok=18.165B dt=14088 wait=170 +16:55:55 pushed ckpt_w129.pt,log_w129.txt +16:57:11 waited 76s for global r=523 +16:57:18 global r=523 load 4.296 -> 4.289 +16:58:21 step 138020 loss=3.4213 lr=4.50e-05 tok=18.168B dt=14248 wait=170 +16:59:24 step 138040 loss=3.4432 lr=4.50e-05 tok=18.171B dt=14311 wait=170 +17:00:27 step 138060 loss=3.3506 lr=4.49e-05 tok=18.173B dt=14373 wait=171 +17:01:29 step 138080 loss=3.3876 lr=4.49e-05 tok=18.176B dt=14436 wait=171 +17:02:32 step 138100 loss=3.3572 lr=4.49e-05 tok=18.178B dt=14498 wait=171 +17:03:35 step 138120 loss=3.4061 lr=4.49e-05 tok=18.181B dt=14562 wait=172 +17:04:38 step 138140 loss=3.3648 lr=4.49e-05 tok=18.184B dt=14624 wait=172 +17:05:21 pushed ckpt_w129.pt,log_w129.txt +17:06:22 waited 61s for global r=524 +17:06:29 global r=524 load 4.284 -> 4.284 +17:07:02 step 138160 loss=3.4496 lr=4.49e-05 tok=18.186B dt=14769 wait=172 +17:08:04 step 138180 loss=3.3512 lr=4.48e-05 tok=18.189B dt=14831 wait=173 +17:09:08 step 138200 loss=3.4011 lr=4.48e-05 tok=18.191B dt=14894 wait=173 +17:10:10 step 138220 loss=3.4589 lr=4.48e-05 tok=18.194B dt=14957 wait=173 +17:11:13 step 138240 loss=3.3650 lr=4.48e-05 tok=18.197B dt=15019 wait=173 +17:12:16 step 138260 loss=3.3969 lr=4.48e-05 tok=18.199B dt=15082 wait=174 +17:13:18 step 138280 loss=3.4138 lr=4.48e-05 tok=18.202B dt=15145 wait=174 +17:14:22 step 138300 loss=3.2969 lr=4.47e-05 tok=18.205B dt=15208 wait=174 +17:14:34 pushed ckpt_w129.pt,log_w129.txt +17:15:34 waited 61s for global r=525 +17:15:42 global r=525 load 4.289 -> 4.278 +17:16:46 step 138320 loss=3.3482 lr=4.47e-05 tok=18.207B dt=15352 wait=175 +17:17:48 step 138340 loss=3.3755 lr=4.47e-05 tok=18.210B dt=15415 wait=175 +17:18:51 step 138360 loss=3.4101 lr=4.47e-05 tok=18.212B dt=15478 wait=175 +17:19:54 step 138380 loss=3.4031 lr=4.47e-05 tok=18.215B dt=15541 wait=176 +17:20:56 step 138400 loss=3.3965 lr=4.46e-05 tok=18.218B dt=15603 wait=176 +17:21:59 step 138420 loss=3.5537 lr=4.46e-05 tok=18.220B dt=15666 wait=176 +17:23:02 step 138440 loss=3.3395 lr=4.46e-05 tok=18.223B dt=15729 wait=177 +17:23:47 pushed ckpt_w129.pt,log_w129.txt +17:24:47 waited 60s for global r=526 +17:24:55 global r=526 load 4.276 -> 4.291 +17:25:28 step 138460 loss=3.4000 lr=4.46e-05 tok=18.226B dt=15875 wait=177 +17:26:31 step 138480 loss=3.4098 lr=4.46e-05 tok=18.228B dt=15938 wait=178 +17:27:34 step 138500 loss=3.3325 lr=4.46e-05 tok=18.231B dt=16000 wait=178 +17:28:37 step 138520 loss=3.4541 lr=4.45e-05 tok=18.233B dt=16063 wait=178 +17:29:39 step 138540 loss=3.3569 lr=4.45e-05 tok=18.236B dt=16126 wait=179 +17:30:42 step 138560 loss=3.3228 lr=4.45e-05 tok=18.239B dt=16189 wait=179 +17:31:45 step 138580 loss=3.4347 lr=4.45e-05 tok=18.241B dt=16251 wait=179 +17:32:48 step 138600 loss=3.4263 lr=4.45e-05 tok=18.244B dt=16314 wait=180 +17:33:01 pushed ckpt_w129.pt,log_w129.txt +17:34:17 waited 76s for global r=527 +17:34:31 global r=527 load 4.320 -> 4.298 +17:35:35 step 138620 loss=3.3340 lr=4.45e-05 tok=18.247B dt=16482 wait=180 +17:36:38 step 138640 loss=3.3994 lr=4.44e-05 tok=18.249B dt=16545 wait=180 +17:37:41 step 138660 loss=3.3442 lr=4.44e-05 tok=18.252B dt=16608 wait=181 +17:38:45 step 138680 loss=3.3222 lr=4.44e-05 tok=18.254B dt=16671 wait=181 +17:39:48 step 138700 loss=3.3606 lr=4.44e-05 tok=18.257B dt=16734 wait=181 +17:40:50 step 138720 loss=3.4859 lr=4.44e-05 tok=18.260B dt=16797 wait=182 +17:41:53 step 138740 loss=3.4072 lr=4.44e-05 tok=18.262B dt=16859 wait=182 +17:42:40 pushed ckpt_w129.pt,log_w129.txt +17:43:25 waited 45s for global r=528 +17:43:33 global r=528 load 4.284 -> 4.296 +17:44:06 step 138760 loss=3.4758 lr=4.43e-05 tok=18.265B dt=16992 wait=182 +17:45:08 step 138780 loss=3.2800 lr=4.43e-05 tok=18.268B dt=17055 wait=183 +17:46:11 step 138800 loss=3.3689 lr=4.43e-05 tok=18.270B dt=17117 wait=183 +17:47:14 step 138820 loss=3.4122 lr=4.43e-05 tok=18.273B dt=17180 wait=183 +17:48:16 step 138840 loss=3.3552 lr=4.43e-05 tok=18.275B dt=17242 wait=184 +17:49:19 step 138860 loss=3.4379 lr=4.42e-05 tok=18.278B dt=17305 wait=184 +17:50:22 step 138880 loss=3.4341 lr=4.42e-05 tok=18.281B dt=17368 wait=184 +17:51:25 step 138900 loss=3.4759 lr=4.42e-05 tok=18.283B dt=17431 wait=185 +17:51:38 pushed ckpt_w129.pt,log_w129.txt +17:52:54 waited 76s for global r=529 +17:53:00 global r=529 load 4.297 -> 4.299 +17:54:04 step 138920 loss=3.3746 lr=4.42e-05 tok=18.286B dt=17590 wait=185 +17:55:07 step 138940 loss=3.4151 lr=4.42e-05 tok=18.288B dt=17653 wait=185 +17:56:10 step 138960 loss=3.3382 lr=4.42e-05 tok=18.291B dt=17716 wait=186 +17:57:13 step 138980 loss=3.3294 lr=4.41e-05 tok=18.294B dt=17779 wait=186 +17:58:15 step 139000 loss=3.3864 lr=4.41e-05 tok=18.296B dt=17842 wait=186 +17:59:18 step 139020 loss=3.5467 lr=4.41e-05 tok=18.299B dt=17904 wait=187 +18:00:20 step 139040 loss=3.3168 lr=4.41e-05 tok=18.302B dt=17967 wait=187 +18:01:04 pushed ckpt_w129.pt,log_w129.txt +18:02:05 waited 60s for global r=530 +18:02:11 global r=530 load 4.308 -> 4.301 +18:02:44 step 139060 loss=3.4288 lr=4.41e-05 tok=18.304B dt=18111 wait=187 +18:03:48 step 139080 loss=3.1506 lr=4.41e-05 tok=18.307B dt=18175 wait=188 +18:04:52 step 139100 loss=3.3256 lr=4.40e-05 tok=18.309B dt=18238 wait=188 +18:05:55 step 139120 loss=3.4415 lr=4.40e-05 tok=18.312B dt=18302 wait=188 +18:06:58 step 139140 loss=3.3301 lr=4.40e-05 tok=18.315B dt=18364 wait=189 +18:08:01 step 139160 loss=3.4298 lr=4.40e-05 tok=18.317B dt=18428 wait=189 +18:09:04 step 139180 loss=3.3491 lr=4.40e-05 tok=18.320B dt=18490 wait=189 +18:10:06 step 139200 loss=3.4302 lr=4.39e-05 tok=18.323B dt=18553 wait=190 diff --git a/log_w13.txt b/log_w13.txt new file mode 100644 index 0000000000000000000000000000000000000000..c8093a2728daeca3b7f4b66357aa5cf4aff7c987 --- /dev/null +++ b/log_w13.txt @@ -0,0 +1,134 @@ +launch +03:58:11 name=w13 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +03:58:16 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +03:58:25 params 1.08B +03:58:27 resumed step=36600 seen=4.80B round=17 +03:58:27 stream salt=36600 +04:00:03 start: accum=4, 131072 tokens per optimiser step +04:02:22 step 36620 loss=3.7305 lr=2.79e-04 tok=4.800B dt=140 +04:03:40 step 36640 loss=3.8726 lr=2.79e-04 tok=4.802B dt=218 +04:04:56 step 36660 loss=3.7826 lr=2.79e-04 tok=4.805B dt=293 +04:06:11 step 36680 loss=3.8429 lr=2.79e-04 tok=4.808B dt=369 +04:07:27 step 36700 loss=3.6758 lr=2.79e-04 tok=4.810B dt=444 +04:08:42 step 36720 loss=3.7452 lr=2.79e-04 tok=4.813B dt=520 +04:09:58 step 36740 loss=3.7567 lr=2.79e-04 tok=4.816B dt=595 +04:11:14 step 36760 loss=3.7665 lr=2.79e-04 tok=4.818B dt=671 +04:12:29 step 36780 loss=3.6661 lr=2.79e-04 tok=4.821B dt=747 +04:13:45 step 36800 loss=3.8098 lr=2.79e-04 tok=4.823B dt=822 +04:15:00 step 36820 loss=3.7480 lr=2.79e-04 tok=4.826B dt=898 +04:16:16 step 36840 loss=3.7010 lr=2.79e-04 tok=4.829B dt=973 +04:17:31 step 36860 loss=3.7454 lr=2.79e-04 tok=4.831B dt=1049 +04:18:47 step 36880 loss=3.6891 lr=2.79e-04 tok=4.834B dt=1124 +04:20:02 step 36900 loss=3.7434 lr=2.79e-04 tok=4.837B dt=1200 +04:20:15 pushed ckpt_w13.pt,log_w13.txt +04:21:37 step 36920 loss=3.8332 lr=2.79e-04 tok=4.839B dt=1294 +04:22:52 step 36940 loss=3.8029 lr=2.78e-04 tok=4.842B dt=1370 +04:24:08 step 36960 loss=3.7526 lr=2.78e-04 tok=4.844B dt=1445 +04:25:24 step 36980 loss=3.8251 lr=2.78e-04 tok=4.847B dt=1521 +04:26:39 step 37000 loss=3.7582 lr=2.78e-04 tok=4.850B dt=1597 +04:27:55 step 37020 loss=3.7059 lr=2.78e-04 tok=4.852B dt=1672 +04:29:10 step 37040 loss=3.6509 lr=2.78e-04 tok=4.855B dt=1748 +04:30:26 step 37060 loss=3.6807 lr=2.78e-04 tok=4.858B dt=1823 +04:31:41 step 37080 loss=3.6634 lr=2.78e-04 tok=4.860B dt=1899 +04:32:57 step 37100 loss=3.7715 lr=2.78e-04 tok=4.863B dt=1974 +04:34:12 step 37120 loss=3.7071 lr=2.78e-04 tok=4.865B dt=2050 +04:35:28 step 37140 loss=3.7181 lr=2.78e-04 tok=4.868B dt=2125 +04:36:43 step 37160 loss=3.7924 lr=2.78e-04 tok=4.871B dt=2201 +04:37:59 step 37180 loss=3.6392 lr=2.78e-04 tok=4.873B dt=2276 +04:39:14 step 37200 loss=3.7688 lr=2.78e-04 tok=4.876B dt=2352 +04:39:28 pushed ckpt_w13.pt,log_w13.txt +04:40:49 step 37220 loss=3.7832 lr=2.78e-04 tok=4.878B dt=2447 +04:42:05 step 37240 loss=3.7073 lr=2.78e-04 tok=4.881B dt=2522 +04:43:20 step 37260 loss=3.6393 lr=2.78e-04 tok=4.884B dt=2597 +04:44:35 step 37280 loss=3.7033 lr=2.78e-04 tok=4.886B dt=2673 +04:45:51 step 37300 loss=3.7182 lr=2.78e-04 tok=4.889B dt=2748 +04:47:07 step 37320 loss=3.8038 lr=2.78e-04 tok=4.892B dt=2824 +04:48:22 step 37340 loss=3.8753 lr=2.78e-04 tok=4.894B dt=2900 +04:49:38 step 37360 loss=3.7943 lr=2.78e-04 tok=4.897B dt=2975 +04:50:53 step 37380 loss=3.6799 lr=2.78e-04 tok=4.899B dt=3051 +04:52:09 step 37400 loss=3.7130 lr=2.78e-04 tok=4.902B dt=3126 +04:53:25 step 37420 loss=3.7839 lr=2.78e-04 tok=4.905B dt=3202 +04:54:40 step 37440 loss=3.7705 lr=2.78e-04 tok=4.907B dt=3278 +04:55:56 step 37460 loss=3.6786 lr=2.78e-04 tok=4.910B dt=3353 +04:57:11 step 37480 loss=3.8086 lr=2.78e-04 tok=4.913B dt=3429 +04:58:27 step 37500 loss=3.6595 lr=2.78e-04 tok=4.915B dt=3504 +04:58:39 pushed ckpt_w13.pt,log_w13.txt +05:00:00 step 37520 loss=3.7682 lr=2.78e-04 tok=4.918B dt=3598 +05:01:16 step 37540 loss=3.6483 lr=2.78e-04 tok=4.920B dt=3673 +05:02:31 step 37560 loss=3.7235 lr=2.78e-04 tok=4.923B dt=3749 +05:03:47 step 37580 loss=3.8563 lr=2.78e-04 tok=4.926B dt=3824 +05:05:02 step 37600 loss=3.7141 lr=2.78e-04 tok=4.928B dt=3900 +05:06:18 step 37620 loss=3.7487 lr=2.78e-04 tok=4.931B dt=3975 +05:07:34 step 37640 loss=3.8330 lr=2.78e-04 tok=4.934B dt=4051 +05:08:49 step 37660 loss=3.7464 lr=2.78e-04 tok=4.936B dt=4127 +05:10:05 step 37680 loss=3.7052 lr=2.78e-04 tok=4.939B dt=4202 +05:11:20 step 37700 loss=3.7444 lr=2.78e-04 tok=4.941B dt=4278 +05:12:36 step 37720 loss=3.6807 lr=2.78e-04 tok=4.944B dt=4353 +05:13:51 step 37740 loss=3.7462 lr=2.78e-04 tok=4.947B dt=4429 +05:15:07 step 37760 loss=3.6973 lr=2.78e-04 tok=4.949B dt=4504 +05:16:22 step 37780 loss=3.6735 lr=2.78e-04 tok=4.952B dt=4580 +05:17:38 step 37800 loss=3.8080 lr=2.77e-04 tok=4.955B dt=4655 +05:17:50 pushed ckpt_w13.pt,log_w13.txt +05:19:11 step 37820 loss=3.7201 lr=2.77e-04 tok=4.957B dt=4749 +05:20:27 step 37840 loss=3.6883 lr=2.77e-04 tok=4.960B dt=4824 +05:21:42 step 37860 loss=3.7754 lr=2.77e-04 tok=4.962B dt=4900 +05:22:58 step 37880 loss=3.7776 lr=2.77e-04 tok=4.965B dt=4975 +05:24:14 step 37900 loss=3.6661 lr=2.77e-04 tok=4.968B dt=5051 +05:25:29 step 37920 loss=3.7910 lr=2.77e-04 tok=4.970B dt=5127 +05:26:45 step 37940 loss=3.7029 lr=2.77e-04 tok=4.973B dt=5202 +05:28:00 step 37960 loss=3.5945 lr=2.77e-04 tok=4.975B dt=5278 +05:29:16 step 37980 loss=3.7841 lr=2.77e-04 tok=4.978B dt=5353 +05:30:31 step 38000 loss=3.8091 lr=2.77e-04 tok=4.981B dt=5429 +05:31:47 step 38020 loss=3.7497 lr=2.77e-04 tok=4.983B dt=5504 +05:33:02 step 38040 loss=3.6759 lr=2.77e-04 tok=4.986B dt=5580 +05:34:18 step 38060 loss=3.6289 lr=2.77e-04 tok=4.989B dt=5655 +05:35:34 step 38080 loss=3.7056 lr=2.77e-04 tok=4.991B dt=5731 +05:36:49 step 38100 loss=3.8037 lr=2.77e-04 tok=4.994B dt=5807 +05:37:03 pushed ckpt_w13.pt,log_w13.txt +05:38:28 step 38120 loss=3.7314 lr=2.77e-04 tok=4.996B dt=5906 +05:39:44 step 38140 loss=3.7577 lr=2.77e-04 tok=4.999B dt=5981 +05:41:00 step 38160 loss=3.8679 lr=2.77e-04 tok=5.002B dt=6057 +05:42:15 step 38180 loss=3.8001 lr=2.77e-04 tok=5.004B dt=6133 +05:43:31 step 38200 loss=3.7132 lr=2.77e-04 tok=5.007B dt=6208 +05:44:46 step 38220 loss=3.7783 lr=2.77e-04 tok=5.010B dt=6284 +05:46:02 step 38240 loss=3.7170 lr=2.77e-04 tok=5.012B dt=6359 +05:47:17 step 38260 loss=3.7639 lr=2.77e-04 tok=5.015B dt=6435 +05:48:33 step 38280 loss=3.6951 lr=2.77e-04 tok=5.017B dt=6510 +05:49:48 step 38300 loss=3.7267 lr=2.77e-04 tok=5.020B dt=6586 +05:51:04 step 38320 loss=3.7599 lr=2.77e-04 tok=5.023B dt=6661 +05:52:20 step 38340 loss=3.8152 lr=2.77e-04 tok=5.025B dt=6737 +05:53:35 step 38360 loss=3.7228 lr=2.77e-04 tok=5.028B dt=6813 +05:54:51 step 38380 loss=3.7145 lr=2.77e-04 tok=5.031B dt=6888 +05:56:06 step 38400 loss=3.6736 lr=2.77e-04 tok=5.033B dt=6964 +05:56:18 pushed ckpt_w13.pt,log_w13.txt +05:57:38 step 38420 loss=3.7869 lr=2.77e-04 tok=5.036B dt=7055 +05:58:53 step 38440 loss=3.7282 lr=2.77e-04 tok=5.038B dt=7131 +06:00:09 step 38460 loss=3.7672 lr=2.77e-04 tok=5.041B dt=7206 +06:01:24 step 38480 loss=3.6978 lr=2.77e-04 tok=5.044B dt=7282 +06:02:40 step 38500 loss=3.7634 lr=2.77e-04 tok=5.046B dt=7357 +06:03:55 step 38520 loss=3.7354 lr=2.77e-04 tok=5.049B dt=7433 +06:05:11 step 38540 loss=3.6806 lr=2.77e-04 tok=5.052B dt=7508 +06:06:26 step 38560 loss=3.7402 lr=2.77e-04 tok=5.054B dt=7584 +06:07:42 step 38580 loss=3.8328 lr=2.77e-04 tok=5.057B dt=7660 +06:08:58 step 38600 loss=3.7463 lr=2.77e-04 tok=5.059B dt=7735 +06:10:13 step 38620 loss=3.8170 lr=2.77e-04 tok=5.062B dt=7811 +06:11:29 step 38640 loss=3.8032 lr=2.76e-04 tok=5.065B dt=7886 +06:12:45 step 38660 loss=3.7277 lr=2.76e-04 tok=5.067B dt=7962 +06:14:00 step 38680 loss=3.6548 lr=2.76e-04 tok=5.070B dt=8037 +06:15:16 step 38700 loss=3.7556 lr=2.76e-04 tok=5.072B dt=8113 +06:15:30 pushed ckpt_w13.pt,log_w13.txt +06:16:53 step 38720 loss=3.7836 lr=2.76e-04 tok=5.075B dt=8210 +06:18:08 step 38740 loss=3.6945 lr=2.76e-04 tok=5.078B dt=8286 +06:19:24 step 38760 loss=3.7275 lr=2.76e-04 tok=5.080B dt=8361 +06:20:39 step 38780 loss=3.7360 lr=2.76e-04 tok=5.083B dt=8437 +06:21:55 step 38800 loss=3.7074 lr=2.76e-04 tok=5.086B dt=8512 +06:23:10 step 38820 loss=3.8122 lr=2.76e-04 tok=5.088B dt=8588 +06:24:26 step 38840 loss=3.7427 lr=2.76e-04 tok=5.091B dt=8663 +06:25:42 step 38860 loss=3.8336 lr=2.76e-04 tok=5.093B dt=8739 +06:26:57 step 38880 loss=3.7533 lr=2.76e-04 tok=5.096B dt=8815 +06:28:13 step 38900 loss=3.6944 lr=2.76e-04 tok=5.099B dt=8890 +06:29:28 step 38920 loss=3.7743 lr=2.76e-04 tok=5.101B dt=8966 +06:30:44 step 38940 loss=3.7390 lr=2.76e-04 tok=5.104B dt=9041 +06:31:59 step 38960 loss=3.7245 lr=2.76e-04 tok=5.107B dt=9117 +06:33:15 step 38980 loss=3.7625 lr=2.76e-04 tok=5.109B dt=9192 +06:34:30 step 39000 loss=3.6521 lr=2.76e-04 tok=5.112B dt=9268 diff --git a/log_w130.txt b/log_w130.txt new file mode 100644 index 0000000000000000000000000000000000000000..d22556e963b445637adf7611bfdeaae56cce6395 --- /dev/null +++ b/log_w130.txt @@ -0,0 +1,23 @@ +launch +13:36:59 name=w130 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 77/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:37:05 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:37:05 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:37:15 params 1.08B +13:37:16 resumed step=135000 seen=17.78B round=506 +13:37:16 stream salt=135000 +13:39:27 blocks compiled +13:39:27 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:42:17 step 135001 loss=3.0791 lr=4.77e-05 tok=17.776B dt=171 wait=133 +13:42:41 step 135002 loss=3.1645 lr=4.77e-05 tok=17.776B dt=195 wait=133 +13:43:45 step 135020 loss=3.3655 lr=4.77e-05 tok=17.779B dt=259 wait=134 +13:44:50 step 135040 loss=3.2799 lr=4.76e-05 tok=17.781B dt=323 wait=134 +13:45:53 step 135060 loss=3.3523 lr=4.76e-05 tok=17.784B dt=386 wait=135 +13:46:11 global r=507 load 4.286 -> 4.282 +13:46:43 step 135160 loss=3.3738 lr=4.75e-05 tok=17.785B dt=436 wait=135 +13:47:55 step 135180 loss=3.3727 lr=4.75e-05 tok=17.788B dt=508 wait=135 +13:49:28 step 135200 loss=3.2918 lr=4.75e-05 tok=17.790B dt=602 wait=135 +13:50:54 step 135220 loss=3.4119 lr=4.75e-05 tok=17.793B dt=687 wait=136 +13:51:57 step 135240 loss=3.3024 lr=4.75e-05 tok=17.796B dt=750 wait=136 +13:53:00 step 135260 loss=3.3782 lr=4.74e-05 tok=17.798B dt=813 wait=136 +13:54:03 step 135280 loss=3.3622 lr=4.74e-05 tok=17.801B dt=876 wait=137 +13:55:06 step 135300 loss=3.3365 lr=4.74e-05 tok=17.804B dt=940 wait=137 diff --git a/log_w131.txt b/log_w131.txt new file mode 100644 index 0000000000000000000000000000000000000000..4acaa041fedbcd16e964a7a1f320e8b39bffe8b0 --- /dev/null +++ b/log_w131.txt @@ -0,0 +1,188 @@ +launch +19:43:21 name=w131 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 78/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:43:28 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:43:28 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +19:43:37 params 1.08B +19:43:39 resumed step=139950 seen=18.42B round=536 +19:43:39 stream salt=139950 +19:45:22 blocks compiled +19:45:22 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +19:47:42 step 139951 loss=2.9312 lr=4.33e-05 tok=18.422B dt=140 wait=94 +19:48:07 step 139952 loss=3.0883 lr=4.33e-05 tok=18.422B dt=165 wait=94 +19:48:34 step 139960 loss=3.5097 lr=4.33e-05 tok=18.423B dt=193 wait=94 +19:48:57 global r=537 load 4.331 -> 4.316 +19:49:33 step 140260 loss=3.4885 lr=4.30e-05 tok=18.425B dt=251 wait=95 +19:50:36 step 140280 loss=3.3457 lr=4.30e-05 tok=18.427B dt=314 wait=95 +19:51:39 step 140300 loss=3.3612 lr=4.30e-05 tok=18.430B dt=377 wait=95 +19:52:41 step 140320 loss=3.4487 lr=4.30e-05 tok=18.433B dt=439 wait=95 +19:53:44 step 140340 loss=3.3478 lr=4.30e-05 tok=18.435B dt=502 wait=96 +19:54:47 step 140360 loss=3.4646 lr=4.29e-05 tok=18.438B dt=565 wait=96 +19:55:50 step 140380 loss=3.2795 lr=4.29e-05 tok=18.441B dt=628 wait=96 +19:56:53 step 140400 loss=3.4020 lr=4.29e-05 tok=18.443B dt=691 wait=97 +19:57:06 pushed ckpt_w131.pt,log_w131.txt +19:57:51 waited 45s for global r=538 +19:57:58 global r=538 load 4.289 -> 4.292 +19:59:03 step 140420 loss=3.2777 lr=4.29e-05 tok=18.446B dt=821 wait=97 +20:00:05 step 140440 loss=3.4257 lr=4.29e-05 tok=18.448B dt=884 wait=98 +20:01:08 step 140460 loss=3.2757 lr=4.29e-05 tok=18.451B dt=946 wait=98 +20:02:11 step 140480 loss=3.3636 lr=4.28e-05 tok=18.454B dt=1009 wait=98 +20:03:14 step 140500 loss=3.3836 lr=4.28e-05 tok=18.456B dt=1072 wait=98 +20:04:17 step 140520 loss=3.4298 lr=4.28e-05 tok=18.459B dt=1135 wait=99 +20:05:19 step 140540 loss=3.3706 lr=4.28e-05 tok=18.461B dt=1197 wait=99 +20:06:05 pushed ckpt_w131.pt,log_w131.txt +20:07:21 waited 76s for global r=539 +20:07:29 global r=539 load 4.296 -> 4.291 +20:08:02 step 140560 loss=3.1735 lr=4.28e-05 tok=18.464B dt=1361 wait=99 +20:09:05 step 140580 loss=3.2746 lr=4.28e-05 tok=18.467B dt=1423 wait=100 +20:10:07 step 140600 loss=3.3210 lr=4.27e-05 tok=18.469B dt=1486 wait=100 +20:11:10 step 140620 loss=3.3990 lr=4.27e-05 tok=18.472B dt=1548 wait=100 +20:12:12 step 140640 loss=3.2711 lr=4.27e-05 tok=18.475B dt=1610 wait=101 +20:13:15 step 140660 loss=3.2985 lr=4.27e-05 tok=18.477B dt=1673 wait=101 +20:14:17 step 140680 loss=3.3384 lr=4.27e-05 tok=18.480B dt=1736 wait=101 +20:15:20 step 140700 loss=3.3768 lr=4.27e-05 tok=18.482B dt=1798 wait=102 +20:15:33 pushed ckpt_w131.pt,log_w131.txt +20:16:34 waited 60s for global r=540 +20:16:42 global r=540 load 4.303 -> 4.293 +20:17:47 step 140720 loss=3.3681 lr=4.26e-05 tok=18.485B dt=1945 wait=102 +20:18:50 step 140740 loss=3.3753 lr=4.26e-05 tok=18.488B dt=2008 wait=102 +20:19:53 step 140760 loss=3.3744 lr=4.26e-05 tok=18.490B dt=2071 wait=103 +20:20:56 step 140780 loss=3.3550 lr=4.26e-05 tok=18.493B dt=2134 wait=103 +20:21:59 step 140800 loss=3.3051 lr=4.26e-05 tok=18.496B dt=2197 wait=103 +20:23:02 step 140820 loss=3.4268 lr=4.26e-05 tok=18.498B dt=2260 wait=104 +20:24:04 step 140840 loss=3.3202 lr=4.25e-05 tok=18.501B dt=2322 wait=104 +20:24:48 pushed ckpt_w131.pt,log_w131.txt +20:26:03 waited 76s for global r=541 +20:26:11 global r=541 load 4.310 -> 4.297 +20:26:44 step 140860 loss=3.4578 lr=4.25e-05 tok=18.503B dt=2482 wait=104 +20:27:47 step 140880 loss=3.3539 lr=4.25e-05 tok=18.506B dt=2545 wait=105 +20:28:49 step 140900 loss=3.3433 lr=4.25e-05 tok=18.509B dt=2607 wait=105 +20:29:52 step 140920 loss=3.4231 lr=4.25e-05 tok=18.511B dt=2670 wait=105 +20:30:54 step 140940 loss=3.3601 lr=4.24e-05 tok=18.514B dt=2733 wait=106 +20:31:58 step 140960 loss=3.4303 lr=4.24e-05 tok=18.517B dt=2796 wait=106 +20:33:00 step 140980 loss=3.3478 lr=4.24e-05 tok=18.519B dt=2858 wait=106 +20:34:03 step 141000 loss=3.3234 lr=4.24e-05 tok=18.522B dt=2922 wait=107 +20:34:16 pushed ckpt_w131.pt,log_w131.txt +20:35:31 waited 76s for global r=542 +20:35:37 global r=542 load 4.280 -> 4.283 +20:36:41 step 141020 loss=3.3663 lr=4.24e-05 tok=18.524B dt=3079 wait=107 +20:37:44 step 141040 loss=3.3890 lr=4.24e-05 tok=18.527B dt=3142 wait=107 +20:38:46 step 141060 loss=3.3480 lr=4.23e-05 tok=18.530B dt=3204 wait=108 +20:39:49 step 141080 loss=3.3112 lr=4.23e-05 tok=18.532B dt=3267 wait=108 +20:40:51 step 141100 loss=3.3252 lr=4.23e-05 tok=18.535B dt=3329 wait=108 +20:41:54 step 141120 loss=3.3892 lr=4.23e-05 tok=18.538B dt=3392 wait=109 +20:42:56 step 141140 loss=3.2914 lr=4.23e-05 tok=18.540B dt=3455 wait=109 +20:43:40 pushed ckpt_w131.pt,log_w131.txt +20:44:56 waited 76s for global r=543 +20:45:02 global r=543 load 4.306 -> 4.295 +20:45:35 step 141160 loss=3.4122 lr=4.23e-05 tok=18.543B dt=3613 wait=109 +20:46:37 step 141180 loss=3.3226 lr=4.22e-05 tok=18.545B dt=3675 wait=110 +20:47:40 step 141200 loss=3.3022 lr=4.22e-05 tok=18.548B dt=3738 wait=110 +20:48:42 step 141220 loss=3.4070 lr=4.22e-05 tok=18.551B dt=3801 wait=110 +20:49:45 step 141240 loss=3.3410 lr=4.22e-05 tok=18.553B dt=3863 wait=111 +20:50:47 step 141260 loss=3.4501 lr=4.22e-05 tok=18.556B dt=3925 wait=111 +20:51:50 step 141280 loss=3.3512 lr=4.22e-05 tok=18.558B dt=3988 wait=111 +20:52:52 step 141300 loss=3.3834 lr=4.21e-05 tok=18.561B dt=4051 wait=112 +20:53:05 pushed ckpt_w131.pt,log_w131.txt +20:54:20 waited 76s for global r=544 +20:54:27 global r=544 load 4.300 -> 4.291 +20:55:31 step 141320 loss=3.3553 lr=4.21e-05 tok=18.564B dt=4209 wait=112 +20:56:33 step 141340 loss=3.3395 lr=4.21e-05 tok=18.566B dt=4271 wait=112 +20:57:36 step 141360 loss=3.3783 lr=4.21e-05 tok=18.569B dt=4334 wait=113 +20:58:39 step 141380 loss=3.2862 lr=4.21e-05 tok=18.572B dt=4397 wait=113 +20:59:42 step 141400 loss=3.3067 lr=4.21e-05 tok=18.574B dt=4460 wait=113 +21:00:44 step 141420 loss=3.3701 lr=4.20e-05 tok=18.577B dt=4522 wait=113 +21:01:46 step 141440 loss=3.3037 lr=4.20e-05 tok=18.579B dt=4585 wait=114 +21:02:34 pushed ckpt_w131.pt,log_w131.txt +21:04:04 waited 91s for global r=545 +21:04:10 global r=545 load 4.285 -> 4.292 +21:04:43 step 141460 loss=3.4018 lr=4.20e-05 tok=18.582B dt=4761 wait=114 +21:05:45 step 141480 loss=3.3236 lr=4.20e-05 tok=18.585B dt=4823 wait=114 +21:06:48 step 141500 loss=3.2563 lr=4.20e-05 tok=18.587B dt=4886 wait=115 +21:07:51 step 141520 loss=3.3221 lr=4.20e-05 tok=18.590B dt=4949 wait=115 +21:08:54 step 141540 loss=3.3504 lr=4.19e-05 tok=18.593B dt=5012 wait=115 +21:09:56 step 141560 loss=3.4315 lr=4.19e-05 tok=18.595B dt=5074 wait=116 +21:10:59 step 141580 loss=3.3248 lr=4.19e-05 tok=18.598B dt=5137 wait=116 +21:12:02 step 141600 loss=3.3381 lr=4.19e-05 tok=18.600B dt=5200 wait=116 +21:12:16 pushed ckpt_w131.pt,log_w131.txt +21:13:16 waited 61s for global r=546 +21:13:25 global r=546 load 4.286 -> 4.295 +21:14:29 step 141620 loss=3.3017 lr=4.19e-05 tok=18.603B dt=5347 wait=117 +21:15:32 step 141640 loss=3.4074 lr=4.19e-05 tok=18.606B dt=5410 wait=117 +21:16:34 step 141660 loss=3.3873 lr=4.18e-05 tok=18.608B dt=5473 wait=117 +21:17:37 step 141680 loss=3.3148 lr=4.18e-05 tok=18.611B dt=5535 wait=118 +21:18:39 step 141700 loss=3.3348 lr=4.18e-05 tok=18.614B dt=5598 wait=118 +21:19:43 step 141720 loss=3.4122 lr=4.18e-05 tok=18.616B dt=5661 wait=118 +21:20:56 step 141740 loss=3.2942 lr=4.18e-05 tok=18.619B dt=5734 wait=119 +21:21:40 pushed ckpt_w131.pt,log_w131.txt +21:22:56 waited 76s for global r=547 +21:23:05 global r=547 load 4.284 -> 4.288 +21:23:38 step 141760 loss=3.4328 lr=4.18e-05 tok=18.621B dt=5896 wait=119 +21:24:42 step 141780 loss=2.9030 lr=4.17e-05 tok=18.624B dt=5961 wait=119 +21:25:45 step 141800 loss=3.3349 lr=4.17e-05 tok=18.627B dt=6023 wait=120 +21:26:48 step 141820 loss=3.4042 lr=4.17e-05 tok=18.629B dt=6086 wait=120 +21:27:50 step 141840 loss=3.2543 lr=4.17e-05 tok=18.632B dt=6148 wait=120 +21:28:53 step 141860 loss=3.3848 lr=4.17e-05 tok=18.635B dt=6211 wait=120 +21:29:55 step 141880 loss=3.3661 lr=4.17e-05 tok=18.637B dt=6273 wait=121 +21:30:58 step 141900 loss=3.3502 lr=4.16e-05 tok=18.640B dt=6336 wait=121 +21:31:15 pushed ckpt_w131.pt,log_w131.txt +21:33:00 waited 106s for global r=548 +21:33:08 global r=548 load 4.286 -> 4.289 +21:34:12 step 141920 loss=3.3492 lr=4.16e-05 tok=18.642B dt=6530 wait=121 +21:35:25 step 141940 loss=3.4273 lr=4.16e-05 tok=18.645B dt=6603 wait=122 +21:36:28 step 141960 loss=3.3744 lr=4.16e-05 tok=18.648B dt=6666 wait=122 +21:37:30 step 141980 loss=3.3128 lr=4.16e-05 tok=18.650B dt=6728 wait=122 +21:38:33 step 142000 loss=3.4162 lr=4.15e-05 tok=18.653B dt=6791 wait=123 +21:39:36 step 142020 loss=3.3882 lr=4.15e-05 tok=18.655B dt=6854 wait=123 +21:40:39 step 142040 loss=3.2745 lr=4.15e-05 tok=18.658B dt=6917 wait=123 +21:41:34 pushed ckpt_w131.pt,log_w131.txt +21:42:35 waited 61s for global r=549 +21:42:41 global r=549 load 4.302 -> 4.295 +21:43:13 step 142060 loss=3.4172 lr=4.15e-05 tok=18.661B dt=7071 wait=124 +21:44:16 step 142080 loss=3.3448 lr=4.15e-05 tok=18.663B dt=7134 wait=124 +21:45:18 step 142100 loss=3.3477 lr=4.15e-05 tok=18.666B dt=7197 wait=124 +21:46:21 step 142120 loss=3.3802 lr=4.14e-05 tok=18.669B dt=7259 wait=125 +21:47:23 step 142140 loss=3.2696 lr=4.14e-05 tok=18.671B dt=7321 wait=125 +21:48:26 step 142160 loss=3.3899 lr=4.14e-05 tok=18.674B dt=7384 wait=125 +21:49:28 step 142180 loss=3.4002 lr=4.14e-05 tok=18.676B dt=7446 wait=126 +21:50:31 step 142200 loss=3.2757 lr=4.14e-05 tok=18.679B dt=7509 wait=126 +21:50:46 pushed ckpt_w131.pt,log_w131.txt +21:51:46 waited 61s for global r=550 +21:51:53 global r=550 load 4.311 -> 4.295 +21:52:57 step 142220 loss=3.3437 lr=4.14e-05 tok=18.682B dt=7655 wait=126 +21:53:59 step 142240 loss=3.4316 lr=4.13e-05 tok=18.684B dt=7717 wait=127 +21:55:02 step 142260 loss=3.4156 lr=4.13e-05 tok=18.687B dt=7780 wait=127 +21:56:05 step 142280 loss=3.3719 lr=4.13e-05 tok=18.690B dt=7843 wait=127 +21:57:07 step 142300 loss=3.3379 lr=4.13e-05 tok=18.692B dt=7905 wait=128 +21:58:09 step 142320 loss=3.5033 lr=4.13e-05 tok=18.695B dt=7968 wait=128 +21:59:12 step 142340 loss=3.2943 lr=4.13e-05 tok=18.697B dt=8030 wait=128 +21:59:57 pushed ckpt_w131.pt,log_w131.txt +22:01:13 waited 76s for global r=551 +22:01:20 global r=551 load 4.296 -> 4.284 +22:01:52 step 142360 loss=3.4372 lr=4.12e-05 tok=18.700B dt=8191 wait=129 +22:02:55 step 142380 loss=3.3266 lr=4.12e-05 tok=18.703B dt=8253 wait=129 +22:03:58 step 142400 loss=3.3593 lr=4.12e-05 tok=18.705B dt=8316 wait=129 +22:05:00 step 142420 loss=3.4059 lr=4.12e-05 tok=18.708B dt=8379 wait=129 +22:06:03 step 142440 loss=3.2955 lr=4.12e-05 tok=18.711B dt=8441 wait=130 +22:07:05 step 142460 loss=3.4566 lr=4.12e-05 tok=18.713B dt=8504 wait=130 +22:08:09 step 142480 loss=3.3942 lr=4.11e-05 tok=18.716B dt=8567 wait=130 +22:09:11 step 142500 loss=3.3066 lr=4.11e-05 tok=18.718B dt=8629 wait=131 +22:09:23 pushed ckpt_w131.pt,log_w131.txt +22:13:03 global r=552 load 4.295 -> 4.296 +22:14:05 step 142520 loss=3.2845 lr=4.11e-05 tok=18.722B dt=8923 wait=131 +22:15:08 step 142540 loss=3.3915 lr=4.11e-05 tok=18.725B dt=8986 wait=131 +22:16:10 step 142560 loss=3.4277 lr=4.11e-05 tok=18.728B dt=9048 wait=132 +22:17:13 step 142580 loss=3.3070 lr=4.11e-05 tok=18.730B dt=9111 wait=132 +22:18:15 step 142600 loss=3.3301 lr=4.10e-05 tok=18.733B dt=9174 wait=132 +22:19:18 step 142620 loss=3.4538 lr=4.10e-05 tok=18.735B dt=9236 wait=133 +22:20:20 step 142640 loss=3.3345 lr=4.10e-05 tok=18.738B dt=9298 wait=133 +22:21:03 pushed ckpt_w131.pt,log_w131.txt +22:22:49 waited 106s for global r=553 +22:22:55 global r=553 load 4.278 -> 4.276 +22:23:27 step 142660 loss=3.4083 lr=4.10e-05 tok=18.741B dt=9486 wait=133 +22:24:30 step 142680 loss=3.2868 lr=4.10e-05 tok=18.743B dt=9548 wait=134 +22:25:54 step 142700 loss=3.3223 lr=4.10e-05 tok=18.746B dt=9632 wait=134 +22:27:56 step 142720 loss=3.4444 lr=4.09e-05 tok=18.749B dt=9754 wait=134 +22:29:46 step 142740 loss=3.2858 lr=4.09e-05 tok=18.751B dt=9864 wait=135 +22:31:48 step 142760 loss=3.4389 lr=4.09e-05 tok=18.754B dt=9987 wait=135 +22:33:24 step 142780 loss=3.3403 lr=4.09e-05 tok=18.756B dt=10082 wait=135 +22:34:35 step 142800 loss=3.3743 lr=4.09e-05 tok=18.759B dt=10154 wait=136 diff --git a/log_w132.txt b/log_w132.txt new file mode 100644 index 0000000000000000000000000000000000000000..3bbc96dd26856fbd347b85e3c6704c320e49f0bc --- /dev/null +++ b/log_w132.txt @@ -0,0 +1,393 @@ +launch +19:43:18 name=w132 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 79/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:43:24 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:43:24 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +19:43:34 params 1.08B +19:43:35 resumed step=139950 seen=18.42B round=536 +19:43:35 stream salt=139950 +19:45:20 blocks compiled +19:45:20 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +19:47:59 step 139951 loss=2.9522 lr=4.33e-05 tok=18.422B dt=159 wait=114 +19:48:26 step 139952 loss=3.1517 lr=4.33e-05 tok=18.422B dt=186 wait=114 +19:48:54 step 139960 loss=3.5201 lr=4.33e-05 tok=18.423B dt=214 wait=114 +19:49:09 global r=537 load 4.312 -> 4.316 +19:49:40 step 140260 loss=3.4403 lr=4.30e-05 tok=18.425B dt=259 wait=115 +19:50:43 step 140280 loss=3.3303 lr=4.30e-05 tok=18.427B dt=322 wait=115 +19:51:45 step 140300 loss=3.3325 lr=4.30e-05 tok=18.430B dt=385 wait=115 +19:52:47 step 140320 loss=3.4424 lr=4.30e-05 tok=18.433B dt=447 wait=115 +19:53:50 step 140340 loss=3.3772 lr=4.30e-05 tok=18.435B dt=509 wait=116 +19:54:53 step 140360 loss=3.4027 lr=4.29e-05 tok=18.438B dt=572 wait=116 +19:55:56 step 140380 loss=3.3508 lr=4.29e-05 tok=18.441B dt=635 wait=116 +19:56:58 step 140400 loss=3.3316 lr=4.29e-05 tok=18.443B dt=697 wait=117 +19:57:16 pushed ckpt_w132.pt,log_w132.txt +19:57:46 waited 30s for global r=538 +19:57:53 global r=538 load 4.292 -> 4.292 +19:58:58 step 140420 loss=3.2599 lr=4.29e-05 tok=18.446B dt=817 wait=117 +20:00:00 step 140440 loss=3.3754 lr=4.29e-05 tok=18.448B dt=879 wait=117 +20:01:02 step 140460 loss=3.2971 lr=4.29e-05 tok=18.451B dt=942 wait=118 +20:02:05 step 140480 loss=3.3374 lr=4.28e-05 tok=18.454B dt=1004 wait=118 +20:03:08 step 140500 loss=3.3453 lr=4.28e-05 tok=18.456B dt=1067 wait=118 +20:04:11 step 140520 loss=3.4039 lr=4.28e-05 tok=18.459B dt=1130 wait=119 +20:05:13 step 140540 loss=3.3071 lr=4.28e-05 tok=18.461B dt=1192 wait=119 +20:05:56 pushed ckpt_w132.pt,log_w132.txt +20:07:12 waited 76s for global r=539 +20:07:18 global r=539 load 4.305 -> 4.291 +20:07:51 step 140560 loss=3.4131 lr=4.28e-05 tok=18.464B dt=1350 wait=119 +20:08:53 step 140580 loss=3.3279 lr=4.28e-05 tok=18.467B dt=1412 wait=119 +20:09:56 step 140600 loss=3.3034 lr=4.27e-05 tok=18.469B dt=1475 wait=120 +20:10:58 step 140620 loss=3.3895 lr=4.27e-05 tok=18.472B dt=1538 wait=120 +20:12:01 step 140640 loss=3.3212 lr=4.27e-05 tok=18.475B dt=1600 wait=120 +20:13:03 step 140660 loss=3.4128 lr=4.27e-05 tok=18.477B dt=1663 wait=121 +20:14:06 step 140680 loss=3.4039 lr=4.27e-05 tok=18.480B dt=1726 wait=121 +20:15:09 step 140700 loss=3.3337 lr=4.27e-05 tok=18.482B dt=1789 wait=121 +20:15:21 pushed ckpt_w132.pt,log_w132.txt +20:16:37 waited 76s for global r=540 +20:16:43 global r=540 load 4.291 -> 4.293 +20:17:46 step 140720 loss=3.3476 lr=4.26e-05 tok=18.485B dt=1946 wait=122 +20:18:49 step 140740 loss=3.3732 lr=4.26e-05 tok=18.488B dt=2009 wait=122 +20:19:52 step 140760 loss=3.3207 lr=4.26e-05 tok=18.490B dt=2072 wait=122 +20:20:55 step 140780 loss=3.3362 lr=4.26e-05 tok=18.493B dt=2135 wait=123 +20:21:58 step 140800 loss=3.3641 lr=4.26e-05 tok=18.496B dt=2198 wait=123 +20:23:01 step 140820 loss=3.4146 lr=4.26e-05 tok=18.498B dt=2260 wait=123 +20:24:03 step 140840 loss=3.3043 lr=4.25e-05 tok=18.501B dt=2323 wait=124 +20:24:47 pushed ckpt_w132.pt,log_w132.txt +20:26:02 waited 76s for global r=541 +20:26:10 global r=541 load 4.301 -> 4.297 +20:26:43 step 140860 loss=3.4328 lr=4.25e-05 tok=18.503B dt=2482 wait=124 +20:27:46 step 140880 loss=3.2674 lr=4.25e-05 tok=18.506B dt=2545 wait=124 +20:28:48 step 140900 loss=3.3299 lr=4.25e-05 tok=18.509B dt=2608 wait=124 +20:29:50 step 140920 loss=3.4250 lr=4.25e-05 tok=18.511B dt=2670 wait=125 +20:30:53 step 140940 loss=3.3454 lr=4.24e-05 tok=18.514B dt=2732 wait=125 +20:31:56 step 140960 loss=3.4229 lr=4.24e-05 tok=18.517B dt=2795 wait=125 +20:32:58 step 140980 loss=3.3688 lr=4.24e-05 tok=18.519B dt=2858 wait=126 +20:34:01 step 141000 loss=3.3961 lr=4.24e-05 tok=18.522B dt=2920 wait=126 +20:34:13 pushed ckpt_w132.pt,log_w132.txt +20:35:29 waited 76s for global r=542 +20:35:35 global r=542 load 4.298 -> 4.283 +20:36:39 step 141020 loss=3.3181 lr=4.24e-05 tok=18.524B dt=3078 wait=126 +20:37:42 step 141040 loss=3.3605 lr=4.24e-05 tok=18.527B dt=3141 wait=127 +20:38:45 step 141060 loss=3.3558 lr=4.23e-05 tok=18.530B dt=3204 wait=127 +20:39:47 step 141080 loss=3.3578 lr=4.23e-05 tok=18.532B dt=3266 wait=127 +20:40:49 step 141100 loss=3.3916 lr=4.23e-05 tok=18.535B dt=3329 wait=128 +20:41:52 step 141120 loss=3.4096 lr=4.23e-05 tok=18.538B dt=3392 wait=128 +20:42:54 step 141140 loss=3.3162 lr=4.23e-05 tok=18.540B dt=3454 wait=128 +20:43:38 pushed ckpt_w132.pt,log_w132.txt +20:44:54 waited 76s for global r=543 +20:45:02 global r=543 load 4.310 -> 4.295 +20:45:34 step 141160 loss=3.4377 lr=4.23e-05 tok=18.543B dt=3614 wait=128 +20:46:36 step 141180 loss=3.3509 lr=4.22e-05 tok=18.545B dt=3676 wait=129 +20:47:40 step 141200 loss=3.3245 lr=4.22e-05 tok=18.548B dt=3739 wait=129 +20:48:42 step 141220 loss=3.3949 lr=4.22e-05 tok=18.551B dt=3801 wait=129 +20:49:44 step 141240 loss=3.2932 lr=4.22e-05 tok=18.553B dt=3864 wait=130 +20:50:46 step 141260 loss=3.4550 lr=4.22e-05 tok=18.556B dt=3926 wait=130 +20:51:48 step 141280 loss=3.3279 lr=4.22e-05 tok=18.558B dt=3988 wait=130 +20:52:51 step 141300 loss=3.3890 lr=4.21e-05 tok=18.561B dt=4051 wait=131 +20:53:07 pushed ckpt_w132.pt,log_w132.txt +20:54:22 waited 76s for global r=544 +20:54:29 global r=544 load 4.306 -> 4.291 +20:55:33 step 141320 loss=3.3140 lr=4.21e-05 tok=18.564B dt=4212 wait=131 +20:56:35 step 141340 loss=3.3834 lr=4.21e-05 tok=18.566B dt=4275 wait=131 +20:57:38 step 141360 loss=3.3102 lr=4.21e-05 tok=18.569B dt=4338 wait=132 +20:58:41 step 141380 loss=3.3008 lr=4.21e-05 tok=18.572B dt=4401 wait=132 +20:59:44 step 141400 loss=3.2685 lr=4.21e-05 tok=18.574B dt=4463 wait=132 +21:00:46 step 141420 loss=3.3823 lr=4.20e-05 tok=18.577B dt=4526 wait=133 +21:01:49 step 141440 loss=3.3572 lr=4.20e-05 tok=18.579B dt=4588 wait=133 +21:02:34 pushed ckpt_w132.pt,log_w132.txt +21:04:05 waited 91s for global r=545 +21:04:10 global r=545 load 4.316 -> 4.292 +21:04:43 step 141460 loss=3.4208 lr=4.20e-05 tok=18.582B dt=4763 wait=133 +21:05:45 step 141480 loss=3.3856 lr=4.20e-05 tok=18.585B dt=4824 wait=133 +21:06:47 step 141500 loss=3.2935 lr=4.20e-05 tok=18.587B dt=4887 wait=134 +21:07:51 step 141520 loss=3.3773 lr=4.20e-05 tok=18.590B dt=4950 wait=134 +21:08:53 step 141540 loss=3.3252 lr=4.19e-05 tok=18.593B dt=5013 wait=134 +21:09:56 step 141560 loss=3.4166 lr=4.19e-05 tok=18.595B dt=5075 wait=135 +21:10:58 step 141580 loss=3.3559 lr=4.19e-05 tok=18.598B dt=5137 wait=135 +21:12:01 step 141600 loss=3.3449 lr=4.19e-05 tok=18.600B dt=5200 wait=135 +21:12:13 pushed ckpt_w132.pt,log_w132.txt +21:13:29 waited 76s for global r=546 +21:13:36 global r=546 load 4.311 -> 4.295 +21:14:40 step 141620 loss=3.3088 lr=4.19e-05 tok=18.603B dt=5360 wait=136 +21:15:43 step 141640 loss=3.3943 lr=4.19e-05 tok=18.606B dt=5423 wait=136 +21:16:46 step 141660 loss=3.3714 lr=4.18e-05 tok=18.608B dt=5485 wait=136 +21:17:48 step 141680 loss=3.3633 lr=4.18e-05 tok=18.611B dt=5548 wait=137 +21:18:53 step 141700 loss=2.8215 lr=4.18e-05 tok=18.614B dt=5613 wait=137 +21:19:56 step 141720 loss=3.3715 lr=4.18e-05 tok=18.616B dt=5676 wait=137 +21:20:58 step 141740 loss=3.2574 lr=4.18e-05 tok=18.619B dt=5738 wait=137 +21:21:45 pushed ckpt_w132.pt,log_w132.txt +21:23:00 waited 76s for global r=547 +21:23:08 global r=547 load 4.302 -> 4.288 +21:23:41 step 141760 loss=3.4169 lr=4.18e-05 tok=18.621B dt=5900 wait=138 +21:24:43 step 141780 loss=3.3461 lr=4.17e-05 tok=18.624B dt=5963 wait=138 +21:25:45 step 141800 loss=3.3473 lr=4.17e-05 tok=18.627B dt=6025 wait=138 +21:26:47 step 141820 loss=3.4391 lr=4.17e-05 tok=18.629B dt=6087 wait=139 +21:27:50 step 141840 loss=3.3040 lr=4.17e-05 tok=18.632B dt=6150 wait=139 +21:28:53 step 141860 loss=3.3781 lr=4.17e-05 tok=18.635B dt=6212 wait=139 +21:29:56 step 141880 loss=3.2682 lr=4.17e-05 tok=18.637B dt=6276 wait=139 +21:30:58 step 141900 loss=3.4005 lr=4.16e-05 tok=18.640B dt=6338 wait=140 +21:31:13 pushed ckpt_w132.pt,log_w132.txt +21:32:59 waited 106s for global r=548 +21:33:06 global r=548 load 4.287 -> 4.289 +21:34:10 step 141920 loss=3.3351 lr=4.16e-05 tok=18.642B dt=6530 wait=140 +21:35:13 step 141940 loss=3.3864 lr=4.16e-05 tok=18.645B dt=6592 wait=140 +21:36:15 step 141960 loss=3.4043 lr=4.16e-05 tok=18.648B dt=6655 wait=141 +21:37:18 step 141980 loss=3.2544 lr=4.16e-05 tok=18.650B dt=6718 wait=141 +21:38:21 step 142000 loss=3.4245 lr=4.15e-05 tok=18.653B dt=6781 wait=141 +21:39:24 step 142020 loss=3.4225 lr=4.15e-05 tok=18.655B dt=6843 wait=142 +21:40:27 step 142040 loss=3.2620 lr=4.15e-05 tok=18.658B dt=6906 wait=142 +21:41:11 pushed ckpt_w132.pt,log_w132.txt +21:42:27 waited 76s for global r=549 +21:42:32 global r=549 load 4.295 -> 4.295 +21:43:05 step 142060 loss=3.4377 lr=4.15e-05 tok=18.661B dt=7064 wait=142 +21:44:07 step 142080 loss=3.3763 lr=4.15e-05 tok=18.663B dt=7127 wait=143 +21:45:10 step 142100 loss=3.3890 lr=4.15e-05 tok=18.666B dt=7189 wait=143 +21:46:12 step 142120 loss=3.3645 lr=4.14e-05 tok=18.669B dt=7252 wait=143 +21:47:15 step 142140 loss=3.3596 lr=4.14e-05 tok=18.671B dt=7315 wait=144 +21:48:18 step 142160 loss=3.4151 lr=4.14e-05 tok=18.674B dt=7377 wait=144 +21:49:20 step 142180 loss=3.4541 lr=4.14e-05 tok=18.676B dt=7440 wait=144 +21:50:23 step 142200 loss=3.3285 lr=4.14e-05 tok=18.679B dt=7503 wait=144 +21:50:35 pushed ckpt_w132.pt,log_w132.txt +21:51:51 waited 76s for global r=550 +21:51:57 global r=550 load 4.289 -> 4.295 +21:53:00 step 142220 loss=3.3778 lr=4.14e-05 tok=18.682B dt=7660 wait=145 +21:54:03 step 142240 loss=3.4071 lr=4.13e-05 tok=18.684B dt=7722 wait=145 +21:55:05 step 142260 loss=3.3676 lr=4.13e-05 tok=18.687B dt=7785 wait=145 +21:56:08 step 142280 loss=3.3531 lr=4.13e-05 tok=18.690B dt=7847 wait=146 +21:57:10 step 142300 loss=3.3408 lr=4.13e-05 tok=18.692B dt=7910 wait=146 +21:58:13 step 142320 loss=3.4793 lr=4.13e-05 tok=18.695B dt=7972 wait=146 +21:59:15 step 142340 loss=3.2989 lr=4.13e-05 tok=18.697B dt=8035 wait=147 +22:00:04 pushed ckpt_w132.pt,log_w132.txt +22:01:05 waited 61s for global r=551 +22:01:10 global r=551 load 4.296 -> 4.284 +22:01:42 step 142360 loss=3.3867 lr=4.12e-05 tok=18.700B dt=8182 wait=147 +22:02:45 step 142380 loss=3.2516 lr=4.12e-05 tok=18.703B dt=8245 wait=147 +22:03:48 step 142400 loss=3.3525 lr=4.12e-05 tok=18.705B dt=8308 wait=148 +22:04:51 step 142420 loss=3.3754 lr=4.12e-05 tok=18.708B dt=8371 wait=148 +22:05:53 step 142440 loss=3.3484 lr=4.12e-05 tok=18.711B dt=8433 wait=148 +22:06:56 step 142460 loss=3.3938 lr=4.12e-05 tok=18.713B dt=8495 wait=149 +22:07:59 step 142480 loss=3.2907 lr=4.11e-05 tok=18.716B dt=8558 wait=149 +22:09:02 step 142500 loss=3.3879 lr=4.11e-05 tok=18.718B dt=8621 wait=149 +22:09:13 pushed ckpt_w132.pt,log_w132.txt +22:13:19 step 142520 loss=3.3521 lr=4.11e-05 tok=18.721B dt=8878 wait=150 +22:13:26 global r=552 load 4.297 -> 4.296 +22:14:29 step 142520 loss=3.1628 lr=4.11e-05 tok=18.724B dt=8949 wait=150 +22:15:32 step 142540 loss=3.3311 lr=4.11e-05 tok=18.726B dt=9012 wait=150 +22:16:34 step 142560 loss=3.3838 lr=4.11e-05 tok=18.729B dt=9074 wait=150 +22:17:37 step 142580 loss=3.3635 lr=4.11e-05 tok=18.731B dt=9136 wait=151 +22:18:39 step 142600 loss=3.3121 lr=4.10e-05 tok=18.734B dt=9199 wait=151 +22:19:42 step 142620 loss=3.4288 lr=4.10e-05 tok=18.737B dt=9262 wait=151 +22:20:44 step 142640 loss=3.2863 lr=4.10e-05 tok=18.739B dt=9324 wait=152 +22:21:28 pushed ckpt_w132.pt,log_w132.txt +22:22:58 waited 91s for global r=553 +22:23:03 global r=553 load 4.274 -> 4.276 +22:23:36 step 142660 loss=3.4641 lr=4.10e-05 tok=18.742B dt=9496 wait=152 +22:24:39 step 142680 loss=3.2768 lr=4.10e-05 tok=18.745B dt=9559 wait=152 +22:25:41 step 142700 loss=3.3754 lr=4.10e-05 tok=18.747B dt=9621 wait=153 +22:26:44 step 142720 loss=3.4116 lr=4.09e-05 tok=18.750B dt=9683 wait=153 +22:27:46 step 142740 loss=3.2923 lr=4.09e-05 tok=18.752B dt=9746 wait=153 +22:28:49 step 142760 loss=3.3703 lr=4.09e-05 tok=18.755B dt=9809 wait=154 +22:29:52 step 142780 loss=3.3304 lr=4.09e-05 tok=18.758B dt=9871 wait=154 +22:30:54 step 142800 loss=3.3695 lr=4.09e-05 tok=18.760B dt=9933 wait=154 +22:31:08 pushed ckpt_w132.pt,log_w132.txt +22:34:50 global r=554 load 4.296 -> 4.282 +22:35:52 step 142820 loss=3.3652 lr=4.09e-05 tok=18.764B dt=10232 wait=155 +22:36:55 step 142840 loss=3.3800 lr=4.08e-05 tok=18.767B dt=10294 wait=155 +22:37:58 step 142860 loss=3.3278 lr=4.08e-05 tok=18.770B dt=10357 wait=155 +22:39:00 step 142880 loss=3.2930 lr=4.08e-05 tok=18.772B dt=10420 wait=156 +22:40:02 step 142900 loss=3.3627 lr=4.08e-05 tok=18.775B dt=10482 wait=156 +22:41:05 step 142920 loss=3.4032 lr=4.08e-05 tok=18.777B dt=10545 wait=156 +22:42:08 step 142940 loss=3.2796 lr=4.08e-05 tok=18.780B dt=10608 wait=157 +22:42:51 pushed ckpt_w132.pt,log_w132.txt +22:46:25 step 142960 loss=3.3980 lr=4.07e-05 tok=18.783B dt=10865 wait=157 +22:47:28 step 142980 loss=3.3227 lr=4.07e-05 tok=18.785B dt=10928 wait=157 +22:48:30 step 143000 loss=3.3403 lr=4.07e-05 tok=18.788B dt=10990 wait=157 +22:49:33 step 143020 loss=3.2480 lr=4.07e-05 tok=18.790B dt=11053 wait=158 +22:50:36 step 143040 loss=3.3203 lr=4.07e-05 tok=18.793B dt=11115 wait=158 +22:51:38 step 143060 loss=3.3103 lr=4.07e-05 tok=18.796B dt=11177 wait=158 +22:52:41 step 143080 loss=3.3616 lr=4.06e-05 tok=18.798B dt=11240 wait=159 +22:52:50 global r=555 load 4.302 -> 4.285 +22:53:21 step 142960 loss=3.3231 lr=4.07e-05 tok=18.800B dt=11281 wait=159 +22:54:23 step 142980 loss=3.2742 lr=4.07e-05 tok=18.802B dt=11343 wait=159 +22:55:26 step 143000 loss=3.3002 lr=4.07e-05 tok=18.805B dt=11406 wait=160 +22:56:29 step 143020 loss=3.3660 lr=4.07e-05 tok=18.808B dt=11469 wait=160 +22:57:32 step 143040 loss=3.2519 lr=4.07e-05 tok=18.810B dt=11532 wait=160 +22:58:34 step 143060 loss=3.4364 lr=4.07e-05 tok=18.813B dt=11594 wait=161 +22:59:37 step 143080 loss=3.3593 lr=4.06e-05 tok=18.815B dt=11657 wait=161 +23:00:40 step 143100 loss=3.3707 lr=4.06e-05 tok=18.818B dt=11719 wait=161 +23:00:52 pushed ckpt_w132.pt,log_w132.txt +23:03:23 waited 151s for global r=556 +23:03:32 global r=556 load 4.300 -> 4.289 +23:04:37 step 143120 loss=3.3455 lr=4.06e-05 tok=18.821B dt=11956 wait=162 +23:05:39 step 143140 loss=3.3188 lr=4.06e-05 tok=18.823B dt=12019 wait=162 +23:06:41 step 143160 loss=3.3885 lr=4.06e-05 tok=18.826B dt=12081 wait=162 +23:07:43 step 143180 loss=3.3655 lr=4.06e-05 tok=18.828B dt=12143 wait=162 +23:08:51 step 143200 loss=3.2903 lr=4.05e-05 tok=18.831B dt=12211 wait=163 +23:09:54 step 143220 loss=3.4252 lr=4.05e-05 tok=18.834B dt=12273 wait=163 +23:10:57 step 143240 loss=3.3318 lr=4.05e-05 tok=18.836B dt=12336 wait=163 +23:11:40 pushed ckpt_w132.pt,log_w132.txt +23:12:42 waited 61s for global r=557 +23:12:48 global r=557 load 4.296 -> 4.292 +23:13:22 step 143260 loss=3.3634 lr=4.05e-05 tok=18.839B dt=12482 wait=164 +23:14:25 step 143280 loss=3.3471 lr=4.05e-05 tok=18.842B dt=12545 wait=164 +23:15:27 step 143300 loss=3.3567 lr=4.05e-05 tok=18.844B dt=12606 wait=164 +23:16:29 step 143320 loss=3.3981 lr=4.04e-05 tok=18.847B dt=12669 wait=165 +23:17:32 step 143340 loss=3.3193 lr=4.04e-05 tok=18.849B dt=12731 wait=165 +23:18:34 step 143360 loss=3.3573 lr=4.04e-05 tok=18.852B dt=12794 wait=165 +23:19:37 step 143380 loss=3.2747 lr=4.04e-05 tok=18.855B dt=12857 wait=165 +23:20:39 step 143400 loss=3.3877 lr=4.04e-05 tok=18.857B dt=12919 wait=166 +23:20:51 pushed ckpt_w132.pt,log_w132.txt +23:22:07 waited 76s for global r=558 +23:22:15 global r=558 load 4.301 -> 4.287 +23:23:19 step 143420 loss=3.3752 lr=4.04e-05 tok=18.860B dt=13079 wait=166 +23:24:21 step 143440 loss=3.3832 lr=4.03e-05 tok=18.863B dt=13141 wait=166 +23:25:24 step 143460 loss=3.3540 lr=4.03e-05 tok=18.865B dt=13203 wait=167 +23:26:26 step 143480 loss=3.3421 lr=4.03e-05 tok=18.868B dt=13266 wait=167 +23:27:28 step 143500 loss=3.3888 lr=4.03e-05 tok=18.870B dt=13328 wait=167 +23:28:31 step 143520 loss=3.4346 lr=4.03e-05 tok=18.873B dt=13390 wait=168 +23:29:33 step 143540 loss=3.2787 lr=4.03e-05 tok=18.876B dt=13453 wait=168 +23:30:19 pushed ckpt_w132.pt,log_w132.txt +23:31:19 waited 61s for global r=559 +23:31:27 global r=559 load 4.284 -> 4.295 +23:32:00 step 143560 loss=3.4427 lr=4.02e-05 tok=18.878B dt=13599 wait=168 +23:33:02 step 143580 loss=3.3258 lr=4.02e-05 tok=18.881B dt=13661 wait=169 +23:34:04 step 143600 loss=3.3279 lr=4.02e-05 tok=18.884B dt=13723 wait=169 +23:35:06 step 143620 loss=3.3037 lr=4.02e-05 tok=18.886B dt=13786 wait=169 +23:36:09 step 143640 loss=3.3290 lr=4.02e-05 tok=18.889B dt=13848 wait=170 +23:37:11 step 143660 loss=3.4045 lr=4.02e-05 tok=18.891B dt=13911 wait=170 +23:38:13 step 143680 loss=3.4245 lr=4.01e-05 tok=18.894B dt=13973 wait=170 +23:39:16 step 143700 loss=3.3560 lr=4.01e-05 tok=18.897B dt=14036 wait=170 +23:39:29 pushed ckpt_w132.pt,log_w132.txt +23:40:45 waited 76s for global r=560 +23:40:52 global r=560 load 4.300 -> 4.290 +23:41:55 step 143720 loss=3.3654 lr=4.01e-05 tok=18.899B dt=14195 wait=171 +23:42:58 step 143740 loss=3.3376 lr=4.01e-05 tok=18.902B dt=14258 wait=171 +23:44:01 step 143760 loss=3.3586 lr=4.01e-05 tok=18.905B dt=14320 wait=171 +23:45:08 step 143780 loss=3.3568 lr=4.01e-05 tok=18.907B dt=14388 wait=172 +23:46:11 step 143800 loss=3.2906 lr=4.00e-05 tok=18.910B dt=14450 wait=172 +23:47:14 step 143820 loss=3.3330 lr=4.00e-05 tok=18.912B dt=14514 wait=172 +23:48:16 step 143840 loss=3.2448 lr=4.00e-05 tok=18.915B dt=14576 wait=173 +23:49:03 pushed ckpt_w132.pt,log_w132.txt +23:50:04 waited 61s for global r=561 +23:50:14 global r=561 load 4.314 -> 4.287 +23:50:48 step 143860 loss=3.4145 lr=4.00e-05 tok=18.918B dt=14727 wait=173 +23:51:50 step 143880 loss=3.2708 lr=4.00e-05 tok=18.920B dt=14790 wait=173 +23:52:52 step 143900 loss=3.2746 lr=4.00e-05 tok=18.923B dt=14852 wait=174 +23:53:55 step 143920 loss=3.3716 lr=4.00e-05 tok=18.925B dt=14915 wait=174 +23:54:58 step 143940 loss=3.2693 lr=3.99e-05 tok=18.928B dt=14978 wait=174 +23:56:01 step 143960 loss=3.3477 lr=3.99e-05 tok=18.931B dt=15040 wait=174 +23:57:04 step 143980 loss=3.3052 lr=3.99e-05 tok=18.933B dt=15103 wait=175 +23:58:06 step 144000 loss=3.3690 lr=3.99e-05 tok=18.936B dt=15166 wait=175 +23:58:18 pushed ckpt_w132.pt,log_w132.txt +23:59:18 waited 61s for global r=562 +23:59:31 global r=562 load 4.306 -> 4.292 +00:00:35 step 144020 loss=3.3047 lr=3.99e-05 tok=18.939B dt=15314 wait=175 +00:01:36 step 144040 loss=3.4675 lr=3.99e-05 tok=18.941B dt=15376 wait=176 +00:02:39 step 144060 loss=3.3422 lr=3.98e-05 tok=18.944B dt=15438 wait=176 +00:03:41 step 144080 loss=3.3600 lr=3.98e-05 tok=18.946B dt=15501 wait=176 +00:04:44 step 144100 loss=3.3265 lr=3.98e-05 tok=18.949B dt=15563 wait=177 +00:05:47 step 144120 loss=3.4065 lr=3.98e-05 tok=18.952B dt=15626 wait=177 +00:06:49 step 144140 loss=3.3170 lr=3.98e-05 tok=18.954B dt=15688 wait=177 +00:07:42 pushed ckpt_w132.pt,log_w132.txt +00:08:59 waited 77s for global r=563 +00:09:12 global r=563 load 4.276 -> 4.281 +00:09:46 step 144160 loss=3.4341 lr=3.98e-05 tok=18.957B dt=15866 wait=178 +00:10:49 step 144180 loss=3.3320 lr=3.97e-05 tok=18.960B dt=15928 wait=178 +00:11:51 step 144200 loss=3.3210 lr=3.97e-05 tok=18.962B dt=15990 wait=178 +00:12:53 step 144220 loss=3.3663 lr=3.97e-05 tok=18.965B dt=16052 wait=179 +00:13:55 step 144240 loss=3.2818 lr=3.97e-05 tok=18.967B dt=16115 wait=179 +00:14:58 step 144260 loss=3.4003 lr=3.97e-05 tok=18.970B dt=16178 wait=179 +00:16:01 step 144280 loss=3.3702 lr=3.97e-05 tok=18.973B dt=16240 wait=179 +00:17:03 step 144300 loss=3.3718 lr=3.96e-05 tok=18.975B dt=16303 wait=180 +00:17:18 pushed ckpt_w132.pt,log_w132.txt +00:18:19 waited 61s for global r=564 +00:18:28 global r=564 load 4.285 -> 4.290 +00:19:32 step 144320 loss=3.3581 lr=3.96e-05 tok=18.978B dt=16451 wait=180 +00:20:35 step 144340 loss=3.1698 lr=3.96e-05 tok=18.981B dt=16514 wait=180 +00:21:37 step 144360 loss=3.4140 lr=3.96e-05 tok=18.983B dt=16577 wait=181 +00:22:40 step 144380 loss=3.3196 lr=3.96e-05 tok=18.986B dt=16640 wait=181 +00:23:42 step 144400 loss=3.3585 lr=3.96e-05 tok=18.988B dt=16702 wait=181 +00:24:45 step 144420 loss=3.4387 lr=3.95e-05 tok=18.991B dt=16764 wait=182 +00:25:47 step 144440 loss=3.3316 lr=3.95e-05 tok=18.994B dt=16827 wait=182 +00:26:33 pushed ckpt_w132.pt,log_w132.txt +00:27:34 waited 61s for global r=565 +00:27:42 global r=565 load 4.285 -> 4.285 +00:28:15 step 144460 loss=3.3747 lr=3.95e-05 tok=18.996B dt=16974 wait=182 +00:29:17 step 144480 loss=3.3751 lr=3.95e-05 tok=18.999B dt=17037 wait=183 +00:30:20 step 144500 loss=3.3152 lr=3.95e-05 tok=19.002B dt=17099 wait=183 +00:31:23 step 144520 loss=3.4003 lr=3.95e-05 tok=19.004B dt=17162 wait=183 +00:32:25 step 144540 loss=3.3475 lr=3.94e-05 tok=19.007B dt=17225 wait=183 +00:33:28 step 144560 loss=3.4677 lr=3.94e-05 tok=19.009B dt=17287 wait=184 +00:34:30 step 144580 loss=3.3672 lr=3.94e-05 tok=19.012B dt=17350 wait=184 +00:35:33 step 144600 loss=3.3640 lr=3.94e-05 tok=19.015B dt=17412 wait=184 +00:35:45 pushed ckpt_w132.pt,log_w132.txt +00:36:50 waited 65s for global r=566 +00:37:01 global r=566 load 4.281 -> 4.286 +00:38:05 step 144620 loss=3.3479 lr=3.94e-05 tok=19.017B dt=17564 wait=185 +00:39:07 step 144640 loss=3.3586 lr=3.94e-05 tok=19.020B dt=17627 wait=185 +00:40:10 step 144660 loss=3.3723 lr=3.93e-05 tok=19.022B dt=17689 wait=185 +00:41:12 step 144680 loss=3.3248 lr=3.93e-05 tok=19.025B dt=17752 wait=186 +00:42:15 step 144700 loss=3.3190 lr=3.93e-05 tok=19.028B dt=17814 wait=186 +00:43:18 step 144720 loss=3.3891 lr=3.93e-05 tok=19.030B dt=17878 wait=186 +00:44:21 step 144740 loss=3.3541 lr=3.93e-05 tok=19.033B dt=17940 wait=187 +00:45:14 pushed ckpt_w132.pt,log_w132.txt +00:45:59 waited 46s for global r=567 +00:46:09 global r=567 load 4.281 -> 4.280 +00:46:42 step 144760 loss=3.4276 lr=3.93e-05 tok=19.036B dt=18082 wait=187 +00:47:45 step 144780 loss=3.3019 lr=3.92e-05 tok=19.038B dt=18144 wait=187 +00:48:48 step 144800 loss=3.2044 lr=3.92e-05 tok=19.041B dt=18207 wait=188 +00:49:50 step 144820 loss=3.4375 lr=3.92e-05 tok=19.043B dt=18270 wait=188 +00:50:53 step 144840 loss=3.3406 lr=3.92e-05 tok=19.046B dt=18332 wait=188 +00:51:56 step 144860 loss=3.4121 lr=3.92e-05 tok=19.049B dt=18395 wait=189 +00:52:58 step 144880 loss=3.3460 lr=3.92e-05 tok=19.051B dt=18457 wait=189 +00:54:00 step 144900 loss=3.3703 lr=3.91e-05 tok=19.054B dt=18520 wait=189 +00:54:14 pushed ckpt_w132.pt,log_w132.txt +00:55:44 waited 91s for global r=568 +00:55:54 global r=568 load 4.284 -> 4.282 +00:56:58 step 144920 loss=3.3707 lr=3.91e-05 tok=19.057B dt=18697 wait=190 +00:58:00 step 144940 loss=3.4457 lr=3.91e-05 tok=19.059B dt=18760 wait=190 +00:59:02 step 144960 loss=3.3923 lr=3.91e-05 tok=19.062B dt=18822 wait=190 +01:00:04 step 144980 loss=3.3681 lr=3.91e-05 tok=19.064B dt=18884 wait=191 +01:01:07 step 145000 loss=3.3135 lr=3.91e-05 tok=19.067B dt=18946 wait=191 +01:02:09 step 145020 loss=3.4259 lr=3.91e-05 tok=19.070B dt=19009 wait=191 +01:03:12 step 145040 loss=3.3005 lr=3.90e-05 tok=19.072B dt=19071 wait=192 +01:03:59 pushed ckpt_w132.pt,log_w132.txt +01:05:01 waited 62s for global r=569 +01:05:12 global r=569 load 4.307 -> 4.284 +01:05:45 step 145060 loss=3.4094 lr=3.90e-05 tok=19.075B dt=19224 wait=192 +01:06:47 step 145080 loss=3.3270 lr=3.90e-05 tok=19.078B dt=19286 wait=192 +01:07:49 step 145100 loss=3.3550 lr=3.90e-05 tok=19.080B dt=19349 wait=193 +01:08:52 step 145120 loss=3.4417 lr=3.90e-05 tok=19.083B dt=19411 wait=193 +01:09:54 step 145140 loss=3.3068 lr=3.90e-05 tok=19.085B dt=19474 wait=193 +01:10:58 step 145160 loss=3.6249 lr=3.89e-05 tok=19.088B dt=19537 wait=194 +01:12:00 step 145180 loss=3.3608 lr=3.89e-05 tok=19.091B dt=19599 wait=194 +01:13:02 step 145200 loss=3.3615 lr=3.89e-05 tok=19.093B dt=19662 wait=194 +01:13:14 pushed ckpt_w132.pt,log_w132.txt +01:14:15 waited 61s for global r=570 +01:14:22 global r=570 load 4.301 -> 4.284 +01:15:28 step 145220 loss=3.4228 lr=3.89e-05 tok=19.096B dt=19807 wait=194 +01:16:30 step 145240 loss=3.3867 lr=3.89e-05 tok=19.099B dt=19870 wait=195 +01:17:33 step 145260 loss=3.3713 lr=3.89e-05 tok=19.101B dt=19932 wait=195 +01:18:35 step 145280 loss=3.3131 lr=3.88e-05 tok=19.104B dt=19995 wait=195 +01:19:38 step 145300 loss=3.3161 lr=3.88e-05 tok=19.106B dt=20057 wait=196 +01:20:40 step 145320 loss=3.4302 lr=3.88e-05 tok=19.109B dt=20120 wait=196 +01:21:43 step 145340 loss=3.2172 lr=3.88e-05 tok=19.112B dt=20183 wait=196 +01:22:28 pushed ckpt_w132.pt,log_w132.txt +01:23:28 waited 61s for global r=571 +01:23:41 global r=571 load 4.285 -> 4.283 +01:24:15 step 145360 loss=3.3953 lr=3.88e-05 tok=19.114B dt=20334 wait=197 +01:25:16 step 145380 loss=3.3690 lr=3.88e-05 tok=19.117B dt=20396 wait=197 +01:26:19 step 145400 loss=3.2567 lr=3.87e-05 tok=19.119B dt=20459 wait=197 +01:27:22 step 145420 loss=3.4759 lr=3.87e-05 tok=19.122B dt=20522 wait=198 +01:28:25 step 145440 loss=3.3325 lr=3.87e-05 tok=19.125B dt=20584 wait=198 +01:29:27 step 145460 loss=3.4439 lr=3.87e-05 tok=19.127B dt=20647 wait=198 +01:30:29 step 145480 loss=3.3510 lr=3.87e-05 tok=19.130B dt=20709 wait=199 +01:31:32 step 145500 loss=3.4130 lr=3.87e-05 tok=19.133B dt=20771 wait=199 +01:31:43 pushed ckpt_w132.pt,log_w132.txt +01:32:44 waited 61s for global r=572 +01:32:51 global r=572 load 4.293 -> 4.285 +01:33:55 step 145520 loss=3.3559 lr=3.86e-05 tok=19.135B dt=20914 wait=199 +01:34:57 step 145540 loss=3.4538 lr=3.86e-05 tok=19.138B dt=20977 wait=200 +01:36:00 step 145560 loss=3.3311 lr=3.86e-05 tok=19.140B dt=21039 wait=200 +01:37:02 step 145580 loss=3.3110 lr=3.86e-05 tok=19.143B dt=21102 wait=200 +01:38:04 step 145600 loss=3.3391 lr=3.86e-05 tok=19.146B dt=21164 wait=201 +01:39:07 step 145620 loss=3.3674 lr=3.86e-05 tok=19.148B dt=21227 wait=201 +01:40:09 step 145640 loss=3.2867 lr=3.86e-05 tok=19.151B dt=21289 wait=201 diff --git a/log_w133.txt b/log_w133.txt new file mode 100644 index 0000000000000000000000000000000000000000..45e45f9f34b7ae46ecd93b7b33a2f419d6c59920 --- /dev/null +++ b/log_w133.txt @@ -0,0 +1,233 @@ +launch +22:06:29 name=w133 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 80/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:06:34 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:06:34 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +22:06:43 params 1.08B +22:06:45 resumed step=142350 seen=18.76B round=551 +22:06:45 stream salt=142350 +22:09:20 blocks compiled +22:09:20 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +22:12:41 step 142351 loss=2.9695 lr=4.13e-05 tok=18.760B dt=201 wait=155 +22:13:07 step 142352 loss=3.1321 lr=4.13e-05 tok=18.761B dt=227 wait=155 +22:13:35 step 142360 loss=3.5943 lr=4.12e-05 tok=18.762B dt=255 wait=155 +22:13:54 global r=552 load 4.301 -> 4.296 +22:15:04 step 142520 loss=3.3948 lr=4.11e-05 tok=18.764B dt=344 wait=156 +22:16:08 step 142540 loss=3.4812 lr=4.11e-05 tok=18.767B dt=408 wait=156 +22:17:11 step 142560 loss=3.5207 lr=4.11e-05 tok=18.770B dt=471 wait=156 +22:18:14 step 142580 loss=3.4275 lr=4.11e-05 tok=18.772B dt=534 wait=157 +22:19:17 step 142600 loss=3.3967 lr=4.10e-05 tok=18.775B dt=597 wait=157 +22:20:21 step 142620 loss=3.5947 lr=4.10e-05 tok=18.777B dt=661 wait=157 +22:21:25 step 142640 loss=3.3131 lr=4.10e-05 tok=18.780B dt=725 wait=157 +22:22:09 pushed ckpt_w133.pt,log_w133.txt +22:22:55 waited 45s for global r=553 +22:23:02 global r=553 load 4.270 -> 4.276 +22:23:36 step 142660 loss=3.4563 lr=4.10e-05 tok=18.783B dt=856 wait=158 +22:24:40 step 142680 loss=3.3197 lr=4.10e-05 tok=18.785B dt=920 wait=158 +22:25:43 step 142700 loss=3.3764 lr=4.10e-05 tok=18.788B dt=983 wait=158 +22:26:47 step 142720 loss=3.4638 lr=4.09e-05 tok=18.790B dt=1047 wait=159 +22:27:50 step 142740 loss=3.3695 lr=4.09e-05 tok=18.793B dt=1110 wait=159 +22:28:54 step 142760 loss=3.4563 lr=4.09e-05 tok=18.796B dt=1174 wait=159 +22:29:57 step 142780 loss=3.4104 lr=4.09e-05 tok=18.798B dt=1237 wait=160 +22:31:01 step 142800 loss=3.4426 lr=4.09e-05 tok=18.801B dt=1301 wait=160 +22:31:17 pushed ckpt_w133.pt,log_w133.txt +22:34:19 waited 182s for global r=554 +22:34:27 global r=554 load 4.278 -> 4.282 +22:35:32 step 142820 loss=3.4084 lr=4.09e-05 tok=18.804B dt=1572 wait=160 +22:36:35 step 142840 loss=3.3989 lr=4.08e-05 tok=18.806B dt=1635 wait=161 +22:37:38 step 142860 loss=3.4825 lr=4.08e-05 tok=18.809B dt=1698 wait=161 +22:38:41 step 142880 loss=3.3569 lr=4.08e-05 tok=18.811B dt=1761 wait=161 +22:39:44 step 142900 loss=3.3452 lr=4.08e-05 tok=18.814B dt=1824 wait=162 +22:40:47 step 142920 loss=3.4288 lr=4.08e-05 tok=18.817B dt=1887 wait=162 +22:41:50 step 142940 loss=3.3617 lr=4.08e-05 tok=18.819B dt=1950 wait=162 +22:42:35 pushed ckpt_w133.pt,log_w133.txt +22:46:09 step 142960 loss=3.5241 lr=4.07e-05 tok=18.822B dt=2209 wait=162 +22:47:34 step 142980 loss=3.3880 lr=4.07e-05 tok=18.825B dt=2294 wait=163 +22:49:50 step 143000 loss=3.3011 lr=4.07e-05 tok=18.827B dt=2430 wait=163 +22:51:42 step 143020 loss=3.2654 lr=4.07e-05 tok=18.830B dt=2542 wait=163 +22:52:52 global r=555 load 4.284 -> 4.285 +22:54:00 step 142960 loss=3.5002 lr=4.07e-05 tok=18.832B dt=2680 wait=164 +22:55:42 step 142980 loss=3.3571 lr=4.07e-05 tok=18.835B dt=2782 wait=164 +22:56:55 step 143000 loss=3.1365 lr=4.07e-05 tok=18.838B dt=2855 wait=164 +22:57:58 step 143020 loss=3.5504 lr=4.07e-05 tok=18.840B dt=2918 wait=165 +22:59:01 step 143040 loss=3.3478 lr=4.07e-05 tok=18.843B dt=2981 wait=165 +23:00:05 step 143060 loss=3.4071 lr=4.07e-05 tok=18.846B dt=3045 wait=165 +23:01:08 step 143080 loss=3.4811 lr=4.06e-05 tok=18.848B dt=3108 wait=166 +23:02:11 step 143100 loss=3.4160 lr=4.06e-05 tok=18.851B dt=3171 wait=166 +23:02:28 pushed ckpt_w133.pt,log_w133.txt +23:03:28 waited 61s for global r=556 +23:03:35 global r=556 load 4.286 -> 4.289 +23:04:40 step 143120 loss=3.4057 lr=4.06e-05 tok=18.853B dt=3320 wait=166 +23:05:51 step 143140 loss=3.4159 lr=4.06e-05 tok=18.856B dt=3391 wait=167 +23:07:06 step 143160 loss=3.4481 lr=4.06e-05 tok=18.859B dt=3466 wait=167 +23:08:09 step 143180 loss=3.3869 lr=4.06e-05 tok=18.861B dt=3529 wait=167 +23:09:12 step 143200 loss=3.4504 lr=4.05e-05 tok=18.864B dt=3592 wait=168 +23:10:16 step 143220 loss=3.4745 lr=4.05e-05 tok=18.867B dt=3656 wait=168 +23:11:20 step 143240 loss=3.4026 lr=4.05e-05 tok=18.869B dt=3720 wait=168 +23:12:05 pushed ckpt_w133.pt,log_w133.txt +23:12:50 waited 46s for global r=557 +23:12:57 global r=557 load 4.287 -> 4.292 +23:13:30 step 143260 loss=3.5338 lr=4.05e-05 tok=18.872B dt=3850 wait=169 +23:14:33 step 143280 loss=3.3547 lr=4.05e-05 tok=18.874B dt=3913 wait=169 +23:15:53 step 143300 loss=3.3650 lr=4.05e-05 tok=18.877B dt=3993 wait=169 +23:16:59 step 143320 loss=3.4561 lr=4.04e-05 tok=18.880B dt=4059 wait=170 +23:18:02 step 143340 loss=3.3704 lr=4.04e-05 tok=18.882B dt=4122 wait=170 +23:19:05 step 143360 loss=3.4713 lr=4.04e-05 tok=18.885B dt=4185 wait=170 +23:20:09 step 143380 loss=3.3465 lr=4.04e-05 tok=18.887B dt=4249 wait=171 +23:21:12 step 143400 loss=3.4622 lr=4.04e-05 tok=18.890B dt=4312 wait=171 +23:21:26 pushed ckpt_w133.pt,log_w133.txt +23:22:11 waited 45s for global r=558 +23:22:16 global r=558 load 4.269 -> 4.287 +23:23:40 step 143420 loss=3.3934 lr=4.04e-05 tok=18.893B dt=4460 wait=171 +23:24:43 step 143440 loss=3.4391 lr=4.03e-05 tok=18.895B dt=4523 wait=172 +23:25:46 step 143460 loss=3.3871 lr=4.03e-05 tok=18.898B dt=4586 wait=172 +23:26:49 step 143480 loss=3.3577 lr=4.03e-05 tok=18.901B dt=4649 wait=172 +23:27:53 step 143500 loss=3.3882 lr=4.03e-05 tok=18.903B dt=4713 wait=173 +23:28:56 step 143520 loss=3.5018 lr=4.03e-05 tok=18.906B dt=4776 wait=173 +23:29:58 step 143540 loss=3.3363 lr=4.03e-05 tok=18.908B dt=4838 wait=173 +23:30:42 pushed ckpt_w133.pt,log_w133.txt +23:31:27 waited 45s for global r=559 +23:31:36 global r=559 load 4.302 -> 4.295 +23:32:16 step 143560 loss=3.4255 lr=4.02e-05 tok=18.911B dt=4976 wait=173 +23:33:31 step 143580 loss=3.3233 lr=4.02e-05 tok=18.914B dt=5051 wait=174 +23:34:34 step 143600 loss=3.3519 lr=4.02e-05 tok=18.916B dt=5114 wait=174 +23:35:37 step 143620 loss=3.4039 lr=4.02e-05 tok=18.919B dt=5177 wait=174 +23:36:39 step 143640 loss=3.3331 lr=4.02e-05 tok=18.922B dt=5239 wait=175 +23:37:42 step 143660 loss=3.4342 lr=4.02e-05 tok=18.924B dt=5302 wait=175 +23:38:45 step 143680 loss=3.3306 lr=4.01e-05 tok=18.927B dt=5365 wait=175 +23:39:48 step 143700 loss=3.2874 lr=4.01e-05 tok=18.929B dt=5428 wait=176 +23:40:01 pushed ckpt_w133.pt,log_w133.txt +23:40:46 waited 45s for global r=560 +23:40:53 global r=560 load 4.304 -> 4.290 +23:42:05 step 143720 loss=3.3776 lr=4.01e-05 tok=18.932B dt=5565 wait=176 +23:43:20 step 143740 loss=3.3591 lr=4.01e-05 tok=18.935B dt=5640 wait=177 +23:44:23 step 143760 loss=3.4345 lr=4.01e-05 tok=18.937B dt=5703 wait=177 +23:45:26 step 143780 loss=3.2941 lr=4.01e-05 tok=18.940B dt=5766 wait=177 +23:46:28 step 143800 loss=3.2669 lr=4.00e-05 tok=18.943B dt=5828 wait=177 +23:47:31 step 143820 loss=3.3941 lr=4.00e-05 tok=18.945B dt=5891 wait=178 +23:48:34 step 143840 loss=3.3028 lr=4.00e-05 tok=18.948B dt=5954 wait=178 +23:49:23 pushed ckpt_w133.pt,log_w133.txt +23:50:09 waited 45s for global r=561 +23:50:16 global r=561 load 4.274 -> 4.287 +23:50:48 step 143860 loss=3.4538 lr=4.00e-05 tok=18.950B dt=6088 wait=178 +23:52:08 step 143880 loss=3.3406 lr=4.00e-05 tok=18.953B dt=6168 wait=179 +23:53:13 step 143900 loss=3.3051 lr=4.00e-05 tok=18.956B dt=6233 wait=179 +23:54:17 step 143920 loss=3.4468 lr=4.00e-05 tok=18.958B dt=6297 wait=179 +23:55:19 step 143940 loss=3.3233 lr=3.99e-05 tok=18.961B dt=6359 wait=180 +23:56:22 step 143960 loss=3.3805 lr=3.99e-05 tok=18.963B dt=6422 wait=180 +23:57:27 step 143980 loss=2.9792 lr=3.99e-05 tok=18.966B dt=6487 wait=180 +23:58:30 step 144000 loss=3.3089 lr=3.99e-05 tok=18.969B dt=6550 wait=181 +23:58:44 pushed ckpt_w133.pt,log_w133.txt +23:59:29 waited 45s for global r=562 +23:59:40 global r=562 load 4.286 -> 4.292 +00:00:51 step 144020 loss=3.3038 lr=3.99e-05 tok=18.971B dt=6691 wait=181 +00:02:07 step 144040 loss=3.3406 lr=3.99e-05 tok=18.974B dt=6767 wait=181 +00:03:10 step 144060 loss=3.3559 lr=3.98e-05 tok=18.977B dt=6830 wait=182 +00:04:13 step 144080 loss=3.3683 lr=3.98e-05 tok=18.979B dt=6893 wait=182 +00:05:16 step 144100 loss=3.3393 lr=3.98e-05 tok=18.982B dt=6956 wait=182 +00:06:18 step 144120 loss=3.4084 lr=3.98e-05 tok=18.984B dt=7018 wait=183 +00:07:22 step 144140 loss=3.2960 lr=3.98e-05 tok=18.987B dt=7082 wait=183 +00:08:07 pushed ckpt_w133.pt,log_w133.txt +00:08:53 waited 45s for global r=563 +00:09:03 global r=563 load 4.301 -> 4.281 +00:09:35 step 144160 loss=3.4463 lr=3.98e-05 tok=18.990B dt=7215 wait=183 +00:10:46 step 144180 loss=3.3374 lr=3.97e-05 tok=18.992B dt=7286 wait=183 +00:12:01 step 144200 loss=3.3404 lr=3.97e-05 tok=18.995B dt=7361 wait=184 +00:13:04 step 144220 loss=3.4656 lr=3.97e-05 tok=18.998B dt=7424 wait=184 +00:14:08 step 144240 loss=3.2867 lr=3.97e-05 tok=19.000B dt=7488 wait=184 +00:15:11 step 144260 loss=3.4509 lr=3.97e-05 tok=19.003B dt=7551 wait=185 +00:16:14 step 144280 loss=3.3532 lr=3.97e-05 tok=19.005B dt=7614 wait=185 +00:17:17 step 144300 loss=3.4038 lr=3.96e-05 tok=19.008B dt=7676 wait=185 +00:17:28 pushed ckpt_w133.pt,log_w133.txt +00:18:14 waited 45s for global r=564 +00:18:21 global r=564 load 4.309 -> 4.290 +00:19:25 step 144320 loss=3.3335 lr=3.96e-05 tok=19.011B dt=7805 wait=186 +00:20:42 step 144340 loss=3.3807 lr=3.96e-05 tok=19.013B dt=7882 wait=186 +00:21:50 step 144360 loss=3.3428 lr=3.96e-05 tok=19.016B dt=7950 wait=186 +00:22:53 step 144380 loss=3.3451 lr=3.96e-05 tok=19.019B dt=8013 wait=187 +00:23:56 step 144400 loss=3.3447 lr=3.96e-05 tok=19.021B dt=8076 wait=187 +00:24:58 step 144420 loss=3.4452 lr=3.95e-05 tok=19.024B dt=8138 wait=187 +00:26:01 step 144440 loss=3.2991 lr=3.95e-05 tok=19.026B dt=8201 wait=188 +00:26:44 pushed ckpt_w133.pt,log_w133.txt +00:27:29 waited 45s for global r=565 +00:27:38 global r=565 load 4.284 -> 4.285 +00:28:11 step 144460 loss=3.3987 lr=3.95e-05 tok=19.029B dt=8331 wait=188 +00:29:14 step 144480 loss=3.3187 lr=3.95e-05 tok=19.032B dt=8394 wait=188 +00:30:36 step 144500 loss=3.3485 lr=3.95e-05 tok=19.034B dt=8476 wait=189 +00:31:39 step 144520 loss=3.4012 lr=3.95e-05 tok=19.037B dt=8539 wait=189 +00:32:42 step 144540 loss=3.3372 lr=3.94e-05 tok=19.040B dt=8602 wait=189 +00:33:44 step 144560 loss=3.3940 lr=3.94e-05 tok=19.042B dt=8664 wait=190 +00:34:47 step 144580 loss=3.2493 lr=3.94e-05 tok=19.045B dt=8727 wait=190 +00:35:50 step 144600 loss=3.4274 lr=3.94e-05 tok=19.047B dt=8790 wait=190 +00:36:07 pushed ckpt_w133.pt,log_w133.txt +00:36:53 waited 45s for global r=566 +00:37:01 global r=566 load 4.282 -> 4.286 +00:38:04 step 144620 loss=3.3818 lr=3.94e-05 tok=19.050B dt=8924 wait=191 +00:39:15 step 144640 loss=3.3761 lr=3.94e-05 tok=19.053B dt=8995 wait=191 +00:40:31 step 144660 loss=3.3955 lr=3.93e-05 tok=19.055B dt=9071 wait=191 +00:41:33 step 144680 loss=3.3034 lr=3.93e-05 tok=19.058B dt=9133 wait=191 +00:42:36 step 144700 loss=3.2823 lr=3.93e-05 tok=19.060B dt=9196 wait=192 +00:43:39 step 144720 loss=3.4908 lr=3.93e-05 tok=19.063B dt=9259 wait=192 +00:44:41 step 144740 loss=3.3072 lr=3.93e-05 tok=19.066B dt=9321 wait=192 +00:45:25 pushed ckpt_w133.pt,log_w133.txt +00:46:11 waited 46s for global r=567 +00:46:17 global r=567 load 4.281 -> 4.280 +00:46:58 step 144760 loss=3.4239 lr=3.93e-05 tok=19.068B dt=9458 wait=193 +00:48:13 step 144780 loss=3.3217 lr=3.92e-05 tok=19.071B dt=9533 wait=193 +00:49:16 step 144800 loss=3.3825 lr=3.92e-05 tok=19.074B dt=9596 wait=193 +00:50:19 step 144820 loss=3.4254 lr=3.92e-05 tok=19.076B dt=9659 wait=194 +00:51:21 step 144840 loss=3.3972 lr=3.92e-05 tok=19.079B dt=9721 wait=194 +00:52:24 step 144860 loss=3.3842 lr=3.92e-05 tok=19.081B dt=9784 wait=194 +00:53:27 step 144880 loss=3.3026 lr=3.92e-05 tok=19.084B dt=9847 wait=195 +00:54:30 step 144900 loss=3.3694 lr=3.91e-05 tok=19.087B dt=9910 wait=195 +00:54:47 pushed ckpt_w133.pt,log_w133.txt +00:55:48 waited 61s for global r=568 +00:55:56 global r=568 load 4.281 -> 4.282 +00:57:20 step 144920 loss=3.3305 lr=3.91e-05 tok=19.089B dt=10080 wait=195 +00:58:22 step 144940 loss=3.4067 lr=3.91e-05 tok=19.092B dt=10142 wait=196 +00:59:25 step 144960 loss=3.3761 lr=3.91e-05 tok=19.095B dt=10205 wait=196 +01:00:28 step 144980 loss=3.3212 lr=3.91e-05 tok=19.097B dt=10268 wait=196 +01:01:30 step 145000 loss=3.3591 lr=3.91e-05 tok=19.100B dt=10330 wait=197 +01:02:33 step 145020 loss=3.4064 lr=3.91e-05 tok=19.102B dt=10393 wait=197 +01:03:36 step 145040 loss=3.3035 lr=3.90e-05 tok=19.105B dt=10456 wait=197 +01:04:20 pushed ckpt_w133.pt,log_w133.txt +01:05:06 waited 45s for global r=569 +01:05:13 global r=569 load 4.285 -> 4.284 +01:05:53 step 145060 loss=3.4141 lr=3.90e-05 tok=19.108B dt=10593 wait=198 +01:07:08 step 145080 loss=3.3688 lr=3.90e-05 tok=19.110B dt=10668 wait=198 +01:08:11 step 145100 loss=3.3489 lr=3.90e-05 tok=19.113B dt=10731 wait=198 +01:09:13 step 145120 loss=3.4073 lr=3.90e-05 tok=19.116B dt=10793 wait=199 +01:10:16 step 145140 loss=3.3124 lr=3.90e-05 tok=19.118B dt=10856 wait=199 +01:11:19 step 145160 loss=3.4420 lr=3.89e-05 tok=19.121B dt=10919 wait=199 +01:12:22 step 145180 loss=3.3237 lr=3.89e-05 tok=19.123B dt=10982 wait=200 +01:13:25 step 145200 loss=3.4199 lr=3.89e-05 tok=19.126B dt=11045 wait=200 +01:13:37 pushed ckpt_w133.pt,log_w133.txt +01:14:22 waited 45s for global r=570 +01:14:28 global r=570 load 4.291 -> 4.284 +01:15:40 step 145220 loss=3.3426 lr=3.89e-05 tok=19.129B dt=11180 wait=200 +01:16:55 step 145240 loss=3.3820 lr=3.89e-05 tok=19.131B dt=11255 wait=201 +01:17:58 step 145260 loss=3.2792 lr=3.89e-05 tok=19.134B dt=11318 wait=201 +01:19:01 step 145280 loss=3.3125 lr=3.88e-05 tok=19.137B dt=11381 wait=201 +01:20:04 step 145300 loss=3.3324 lr=3.88e-05 tok=19.139B dt=11444 wait=202 +01:21:07 step 145320 loss=3.4659 lr=3.88e-05 tok=19.142B dt=11507 wait=202 +01:22:10 step 145340 loss=3.2832 lr=3.88e-05 tok=19.144B dt=11570 wait=202 +01:22:55 pushed ckpt_w133.pt,log_w133.txt +01:23:25 waited 30s for global r=571 +01:23:33 global r=571 load 4.283 -> 4.283 +01:24:06 step 145360 loss=3.3934 lr=3.88e-05 tok=19.147B dt=11686 wait=202 +01:25:17 step 145380 loss=3.2950 lr=3.88e-05 tok=19.150B dt=11757 wait=203 +01:26:31 step 145400 loss=3.3299 lr=3.87e-05 tok=19.152B dt=11831 wait=203 +01:27:34 step 145420 loss=3.4098 lr=3.87e-05 tok=19.155B dt=11894 wait=203 +01:28:37 step 145440 loss=3.3135 lr=3.87e-05 tok=19.157B dt=11957 wait=204 +01:29:40 step 145460 loss=3.3852 lr=3.87e-05 tok=19.160B dt=12020 wait=204 +01:30:43 step 145480 loss=3.3981 lr=3.87e-05 tok=19.163B dt=12083 wait=204 +01:31:45 step 145500 loss=3.3346 lr=3.87e-05 tok=19.165B dt=12145 wait=205 +01:31:58 pushed ckpt_w133.pt,log_w133.txt +01:32:43 waited 45s for global r=572 +01:32:52 global r=572 load 4.296 -> 4.285 +01:33:55 step 145520 loss=3.3651 lr=3.86e-05 tok=19.168B dt=12275 wait=205 +01:35:17 step 145540 loss=3.3930 lr=3.86e-05 tok=19.171B dt=12357 wait=205 +01:36:22 step 145560 loss=3.3307 lr=3.86e-05 tok=19.173B dt=12422 wait=206 +01:37:25 step 145580 loss=3.2644 lr=3.86e-05 tok=19.176B dt=12485 wait=206 +01:38:27 step 145600 loss=3.3437 lr=3.86e-05 tok=19.178B dt=12547 wait=206 +01:39:30 step 145620 loss=3.4420 lr=3.86e-05 tok=19.181B dt=12610 wait=207 +01:40:33 step 145640 loss=3.3068 lr=3.86e-05 tok=19.184B dt=12673 wait=207 diff --git a/log_w134.txt b/log_w134.txt new file mode 100644 index 0000000000000000000000000000000000000000..b0de432b5d2ab62d4222e46861dbb7b8034f69ef --- /dev/null +++ b/log_w134.txt @@ -0,0 +1,533 @@ +launch +22:48:48 name=w134 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 81/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:48:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:48:54 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +22:49:04 params 1.08B +22:49:05 resumed step=142800 seen=18.82B round=554 +22:49:05 stream salt=142800 +22:50:50 blocks compiled +22:50:50 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +22:53:26 step 142801 loss=2.7639 lr=4.09e-05 tok=18.819B dt=157 wait=106 +22:53:52 step 142802 loss=3.0064 lr=4.09e-05 tok=18.820B dt=183 wait=106 +22:54:44 global r=555 load 4.284 -> 4.285 +22:55:18 step 142960 loss=3.4548 lr=4.07e-05 tok=18.822B dt=268 wait=106 +22:56:21 step 142980 loss=3.3605 lr=4.07e-05 tok=18.825B dt=331 wait=107 +22:57:24 step 143000 loss=3.3529 lr=4.07e-05 tok=18.827B dt=394 wait=107 +22:58:26 step 143020 loss=3.4743 lr=4.07e-05 tok=18.830B dt=457 wait=108 +22:59:29 step 143040 loss=3.3545 lr=4.07e-05 tok=18.832B dt=519 wait=108 +23:00:33 step 143060 loss=3.4226 lr=4.07e-05 tok=18.835B dt=583 wait=108 +23:01:36 step 143080 loss=3.3733 lr=4.06e-05 tok=18.838B dt=647 wait=108 +23:02:38 step 143100 loss=3.3671 lr=4.06e-05 tok=18.840B dt=709 wait=109 +23:02:52 pushed ckpt_w134.pt,log_w134.txt +23:03:22 waited 30s for global r=556 +23:03:33 global r=556 load 4.294 -> 4.289 +23:04:39 step 143120 loss=3.3306 lr=4.06e-05 tok=18.843B dt=829 wait=109 +23:05:41 step 143140 loss=3.3984 lr=4.06e-05 tok=18.846B dt=892 wait=109 +23:06:44 step 143160 loss=3.3577 lr=4.06e-05 tok=18.848B dt=954 wait=110 +23:07:47 step 143180 loss=3.3842 lr=4.06e-05 tok=18.851B dt=1017 wait=110 +23:08:50 step 143200 loss=3.3917 lr=4.05e-05 tok=18.853B dt=1080 wait=110 +23:09:53 step 143220 loss=3.4315 lr=4.05e-05 tok=18.856B dt=1144 wait=111 +23:10:56 step 143240 loss=3.3473 lr=4.05e-05 tok=18.859B dt=1206 wait=111 +23:11:40 pushed ckpt_w134.pt,log_w134.txt +23:12:41 waited 61s for global r=557 +23:12:47 global r=557 load 4.308 -> 4.292 +23:13:20 step 143260 loss=3.4421 lr=4.05e-05 tok=18.861B dt=1350 wait=111 +23:14:23 step 143280 loss=3.2827 lr=4.05e-05 tok=18.864B dt=1414 wait=112 +23:15:27 step 143300 loss=3.1166 lr=4.05e-05 tok=18.867B dt=1477 wait=112 +23:16:30 step 143320 loss=3.4147 lr=4.04e-05 tok=18.869B dt=1540 wait=112 +23:17:32 step 143340 loss=3.2875 lr=4.04e-05 tok=18.872B dt=1602 wait=113 +23:18:35 step 143360 loss=3.4604 lr=4.04e-05 tok=18.874B dt=1665 wait=113 +23:19:37 step 143380 loss=3.3651 lr=4.04e-05 tok=18.877B dt=1728 wait=113 +23:20:40 step 143400 loss=3.4074 lr=4.04e-05 tok=18.880B dt=1790 wait=114 +23:20:52 pushed ckpt_w134.pt,log_w134.txt +23:22:08 waited 76s for global r=558 +23:22:14 global r=558 load 4.300 -> 4.287 +23:23:18 step 143420 loss=3.3300 lr=4.04e-05 tok=18.882B dt=1948 wait=114 +23:24:21 step 143440 loss=3.4218 lr=4.03e-05 tok=18.885B dt=2012 wait=115 +23:25:24 step 143460 loss=3.3514 lr=4.03e-05 tok=18.887B dt=2075 wait=115 +23:26:28 step 143480 loss=3.3633 lr=4.03e-05 tok=18.890B dt=2138 wait=115 +23:27:31 step 143500 loss=3.3901 lr=4.03e-05 tok=18.893B dt=2201 wait=115 +23:28:33 step 143520 loss=3.4737 lr=4.03e-05 tok=18.895B dt=2264 wait=116 +23:29:36 step 143540 loss=3.3889 lr=4.03e-05 tok=18.898B dt=2326 wait=116 +23:30:19 pushed ckpt_w134.pt,log_w134.txt +23:31:20 waited 60s for global r=559 +23:31:27 global r=559 load 4.309 -> 4.295 +23:32:00 step 143560 loss=3.4511 lr=4.02e-05 tok=18.901B dt=2470 wait=116 +23:33:03 step 143580 loss=3.3162 lr=4.02e-05 tok=18.903B dt=2533 wait=117 +23:34:05 step 143600 loss=3.3372 lr=4.02e-05 tok=18.906B dt=2595 wait=117 +23:35:08 step 143620 loss=3.3415 lr=4.02e-05 tok=18.908B dt=2659 wait=117 +23:36:11 step 143640 loss=3.3161 lr=4.02e-05 tok=18.911B dt=2721 wait=118 +23:37:15 step 143660 loss=3.3505 lr=4.02e-05 tok=18.914B dt=2785 wait=118 +23:38:17 step 143680 loss=3.3492 lr=4.01e-05 tok=18.916B dt=2848 wait=118 +23:39:20 step 143700 loss=3.3996 lr=4.01e-05 tok=18.919B dt=2910 wait=119 +23:39:34 pushed ckpt_w134.pt,log_w134.txt +23:40:35 waited 61s for global r=560 +23:40:41 global r=560 load 4.282 -> 4.290 +23:41:46 step 143720 loss=3.3676 lr=4.01e-05 tok=18.922B dt=3056 wait=119 +23:42:49 step 143740 loss=3.4079 lr=4.01e-05 tok=18.924B dt=3119 wait=119 +23:43:52 step 143760 loss=3.3936 lr=4.01e-05 tok=18.927B dt=3182 wait=120 +23:44:55 step 143780 loss=3.3717 lr=4.01e-05 tok=18.929B dt=3245 wait=120 +23:45:57 step 143800 loss=3.4204 lr=4.00e-05 tok=18.932B dt=3308 wait=120 +23:47:01 step 143820 loss=3.4395 lr=4.00e-05 tok=18.935B dt=3371 wait=121 +23:48:03 step 143840 loss=3.3528 lr=4.00e-05 tok=18.937B dt=3434 wait=121 +23:48:48 pushed ckpt_w134.pt,log_w134.txt +23:50:03 waited 76s for global r=561 +23:50:11 global r=561 load 4.289 -> 4.287 +23:50:43 step 143860 loss=3.4387 lr=4.00e-05 tok=18.940B dt=3594 wait=121 +23:51:46 step 143880 loss=3.3699 lr=4.00e-05 tok=18.943B dt=3656 wait=122 +23:52:49 step 143900 loss=3.3476 lr=4.00e-05 tok=18.945B dt=3719 wait=122 +23:53:52 step 143920 loss=3.4437 lr=4.00e-05 tok=18.948B dt=3782 wait=122 +23:54:55 step 143940 loss=3.3412 lr=3.99e-05 tok=18.950B dt=3845 wait=123 +23:55:57 step 143960 loss=3.4365 lr=3.99e-05 tok=18.953B dt=3907 wait=123 +23:56:59 step 143980 loss=3.3372 lr=3.99e-05 tok=18.956B dt=3970 wait=123 +23:58:02 step 144000 loss=3.3799 lr=3.99e-05 tok=18.958B dt=4033 wait=123 +23:58:14 pushed ckpt_w134.pt,log_w134.txt +23:59:30 waited 76s for global r=562 +23:59:41 global r=562 load 4.295 -> 4.292 +00:00:45 step 144020 loss=3.4309 lr=3.99e-05 tok=18.961B dt=4196 wait=124 +00:01:48 step 144040 loss=3.4048 lr=3.99e-05 tok=18.963B dt=4258 wait=124 +00:02:51 step 144060 loss=3.3545 lr=3.98e-05 tok=18.966B dt=4321 wait=125 +00:03:54 step 144080 loss=3.3339 lr=3.98e-05 tok=18.969B dt=4384 wait=125 +00:04:57 step 144100 loss=3.3655 lr=3.98e-05 tok=18.971B dt=4447 wait=125 +00:06:00 step 144120 loss=3.4810 lr=3.98e-05 tok=18.974B dt=4510 wait=126 +00:07:02 step 144140 loss=3.3831 lr=3.98e-05 tok=18.977B dt=4573 wait=126 +00:07:48 pushed ckpt_w134.pt,log_w134.txt +00:08:48 waited 61s for global r=563 +00:08:59 global r=563 load 4.282 -> 4.281 +00:09:32 step 144160 loss=3.5126 lr=3.98e-05 tok=18.979B dt=4722 wait=126 +00:10:34 step 144180 loss=3.3696 lr=3.97e-05 tok=18.982B dt=4785 wait=126 +00:11:37 step 144200 loss=3.2902 lr=3.97e-05 tok=18.984B dt=4848 wait=127 +00:12:41 step 144220 loss=3.4064 lr=3.97e-05 tok=18.987B dt=4911 wait=127 +00:13:44 step 144240 loss=3.2887 lr=3.97e-05 tok=18.990B dt=4974 wait=127 +00:14:47 step 144260 loss=3.4313 lr=3.97e-05 tok=18.992B dt=5037 wait=128 +00:15:50 step 144280 loss=3.4039 lr=3.97e-05 tok=18.995B dt=5100 wait=128 +00:16:53 step 144300 loss=3.3694 lr=3.96e-05 tok=18.998B dt=5163 wait=128 +00:17:05 pushed ckpt_w134.pt,log_w134.txt +00:18:21 waited 76s for global r=564 +00:18:32 global r=564 load 4.291 -> 4.290 +00:19:37 step 144320 loss=3.2982 lr=3.96e-05 tok=19.000B dt=5327 wait=129 +00:20:39 step 144340 loss=3.3713 lr=3.96e-05 tok=19.003B dt=5390 wait=129 +00:21:42 step 144360 loss=3.3702 lr=3.96e-05 tok=19.005B dt=5453 wait=129 +00:22:45 step 144380 loss=3.3343 lr=3.96e-05 tok=19.008B dt=5515 wait=130 +00:23:48 step 144400 loss=3.3428 lr=3.96e-05 tok=19.011B dt=5578 wait=130 +00:24:51 step 144420 loss=3.4485 lr=3.95e-05 tok=19.013B dt=5641 wait=130 +00:25:54 step 144440 loss=3.3154 lr=3.95e-05 tok=19.016B dt=5704 wait=131 +00:26:37 pushed ckpt_w134.pt,log_w134.txt +00:27:23 waited 46s for global r=565 +00:27:32 global r=565 load 4.297 -> 4.285 +00:28:05 step 144460 loss=3.4434 lr=3.95e-05 tok=19.019B dt=5835 wait=131 +00:29:07 step 144480 loss=3.3523 lr=3.95e-05 tok=19.021B dt=5898 wait=131 +00:30:10 step 144500 loss=3.4100 lr=3.95e-05 tok=19.024B dt=5960 wait=132 +00:31:13 step 144520 loss=3.4921 lr=3.95e-05 tok=19.026B dt=6023 wait=132 +00:32:16 step 144540 loss=3.2691 lr=3.94e-05 tok=19.029B dt=6086 wait=132 +00:33:18 step 144560 loss=3.3933 lr=3.94e-05 tok=19.032B dt=6149 wait=133 +00:34:21 step 144580 loss=3.2861 lr=3.94e-05 tok=19.034B dt=6211 wait=133 +00:35:24 step 144600 loss=3.4292 lr=3.94e-05 tok=19.037B dt=6274 wait=133 +00:35:38 pushed ckpt_w134.pt,log_w134.txt +00:36:53 waited 76s for global r=566 +00:37:00 global r=566 load 4.305 -> 4.286 +00:38:05 step 144620 loss=3.3957 lr=3.94e-05 tok=19.040B dt=6436 wait=133 +00:39:08 step 144640 loss=3.4011 lr=3.94e-05 tok=19.042B dt=6498 wait=134 +00:40:11 step 144660 loss=3.3461 lr=3.93e-05 tok=19.045B dt=6561 wait=134 +00:41:14 step 144680 loss=3.3654 lr=3.93e-05 tok=19.047B dt=6624 wait=134 +00:42:16 step 144700 loss=3.4056 lr=3.93e-05 tok=19.050B dt=6687 wait=135 +00:43:20 step 144720 loss=3.4626 lr=3.93e-05 tok=19.053B dt=6750 wait=135 +00:44:24 step 144740 loss=3.2850 lr=3.93e-05 tok=19.055B dt=6814 wait=135 +00:45:08 pushed ckpt_w134.pt,log_w134.txt +00:46:08 waited 60s for global r=567 +00:46:16 global r=567 load 4.287 -> 4.280 +00:46:50 step 144760 loss=3.3614 lr=3.93e-05 tok=19.058B dt=6960 wait=136 +00:47:52 step 144780 loss=3.3281 lr=3.92e-05 tok=19.060B dt=7023 wait=136 +00:48:55 step 144800 loss=3.3780 lr=3.92e-05 tok=19.063B dt=7085 wait=136 +00:49:58 step 144820 loss=3.4253 lr=3.92e-05 tok=19.066B dt=7148 wait=137 +00:51:00 step 144840 loss=3.2686 lr=3.92e-05 tok=19.068B dt=7211 wait=137 +00:52:03 step 144860 loss=3.4468 lr=3.92e-05 tok=19.071B dt=7273 wait=137 +00:53:06 step 144880 loss=3.4329 lr=3.92e-05 tok=19.074B dt=7336 wait=137 +00:54:09 step 144900 loss=3.2989 lr=3.91e-05 tok=19.076B dt=7399 wait=138 +00:54:52 pushed ckpt_w134.pt,log_w134.txt +00:55:38 waited 45s for global r=568 +00:55:46 global r=568 load 4.291 -> 4.282 +00:56:50 step 144920 loss=3.3403 lr=3.91e-05 tok=19.079B dt=7561 wait=138 +00:57:53 step 144940 loss=3.4036 lr=3.91e-05 tok=19.081B dt=7623 wait=138 +00:58:56 step 144960 loss=3.4237 lr=3.91e-05 tok=19.084B dt=7686 wait=139 +00:59:59 step 144980 loss=3.3173 lr=3.91e-05 tok=19.087B dt=7749 wait=139 +01:01:01 step 145000 loss=3.3815 lr=3.91e-05 tok=19.089B dt=7812 wait=139 +01:02:04 step 145020 loss=3.5035 lr=3.91e-05 tok=19.092B dt=7874 wait=140 +01:03:07 step 145040 loss=3.3494 lr=3.90e-05 tok=19.095B dt=7937 wait=140 +01:03:52 pushed ckpt_w134.pt,log_w134.txt +01:05:08 waited 76s for global r=569 +01:05:17 global r=569 load 4.272 -> 4.284 +01:05:49 step 145060 loss=3.4666 lr=3.90e-05 tok=19.097B dt=8100 wait=140 +01:06:52 step 145080 loss=3.3672 lr=3.90e-05 tok=19.100B dt=8163 wait=141 +01:07:56 step 145100 loss=3.3584 lr=3.90e-05 tok=19.102B dt=8226 wait=141 +01:08:59 step 145120 loss=3.4625 lr=3.90e-05 tok=19.105B dt=8289 wait=141 +01:10:02 step 145140 loss=3.3345 lr=3.90e-05 tok=19.108B dt=8352 wait=142 +01:11:06 step 145160 loss=3.0142 lr=3.89e-05 tok=19.110B dt=8416 wait=142 +01:12:09 step 145180 loss=3.3485 lr=3.89e-05 tok=19.113B dt=8479 wait=142 +01:13:12 step 145200 loss=3.4133 lr=3.89e-05 tok=19.116B dt=8542 wait=143 +01:13:24 pushed ckpt_w134.pt,log_w134.txt +01:14:25 waited 61s for global r=570 +01:14:33 global r=570 load 4.276 -> 4.284 +01:15:37 step 145220 loss=3.3581 lr=3.89e-05 tok=19.118B dt=8687 wait=143 +01:16:40 step 145240 loss=3.4167 lr=3.89e-05 tok=19.121B dt=8750 wait=143 +01:17:43 step 145260 loss=3.3440 lr=3.89e-05 tok=19.123B dt=8813 wait=144 +01:18:45 step 145280 loss=3.3861 lr=3.88e-05 tok=19.126B dt=8876 wait=144 +01:19:48 step 145300 loss=3.4053 lr=3.88e-05 tok=19.129B dt=8938 wait=144 +01:20:50 step 145320 loss=3.4377 lr=3.88e-05 tok=19.131B dt=9001 wait=144 +01:21:53 step 145340 loss=3.3028 lr=3.88e-05 tok=19.134B dt=9063 wait=145 +01:22:39 pushed ckpt_w134.pt,log_w134.txt +01:23:24 waited 45s for global r=571 +01:23:34 global r=571 load 4.291 -> 4.283 +01:24:07 step 145360 loss=3.3848 lr=3.88e-05 tok=19.137B dt=9197 wait=145 +01:25:09 step 145380 loss=3.3948 lr=3.88e-05 tok=19.139B dt=9259 wait=145 +01:26:12 step 145400 loss=3.2849 lr=3.87e-05 tok=19.142B dt=9322 wait=146 +01:27:14 step 145420 loss=3.4520 lr=3.87e-05 tok=19.144B dt=9385 wait=146 +01:28:17 step 145440 loss=3.3724 lr=3.87e-05 tok=19.147B dt=9448 wait=146 +01:29:21 step 145460 loss=3.4082 lr=3.87e-05 tok=19.150B dt=9511 wait=147 +01:30:23 step 145480 loss=3.3855 lr=3.87e-05 tok=19.152B dt=9574 wait=147 +01:31:26 step 145500 loss=3.4148 lr=3.87e-05 tok=19.155B dt=9636 wait=147 +01:31:42 pushed ckpt_w134.pt,log_w134.txt +01:32:42 waited 61s for global r=572 +01:32:49 global r=572 load 4.277 -> 4.285 +01:33:53 step 145520 loss=3.3557 lr=3.86e-05 tok=19.157B dt=9784 wait=148 +01:34:57 step 145540 loss=3.3780 lr=3.86e-05 tok=19.160B dt=9847 wait=148 +01:36:00 step 145560 loss=3.4168 lr=3.86e-05 tok=19.163B dt=9910 wait=148 +01:37:03 step 145580 loss=3.3015 lr=3.86e-05 tok=19.165B dt=9974 wait=149 +01:38:06 step 145600 loss=3.3813 lr=3.86e-05 tok=19.168B dt=10036 wait=149 +01:39:09 step 145620 loss=3.5503 lr=3.86e-05 tok=19.171B dt=10099 wait=149 +01:40:12 step 145640 loss=3.3123 lr=3.86e-05 tok=19.173B dt=10162 wait=150 +01:40:55 pushed ckpt_w134.pt,log_w134.txt +01:41:56 waited 61s for global r=573 +01:42:04 global r=573 load 4.282 -> 4.286 +01:42:37 step 145660 loss=3.4614 lr=3.85e-05 tok=19.176B dt=10308 wait=150 +01:43:40 step 145680 loss=3.3392 lr=3.85e-05 tok=19.178B dt=10370 wait=150 +01:44:43 step 145700 loss=3.3646 lr=3.85e-05 tok=19.181B dt=10433 wait=151 +01:45:46 step 145720 loss=3.4031 lr=3.85e-05 tok=19.184B dt=10496 wait=151 +01:46:49 step 145740 loss=3.3634 lr=3.85e-05 tok=19.186B dt=10559 wait=151 +01:47:51 step 145760 loss=3.4060 lr=3.85e-05 tok=19.189B dt=10622 wait=152 +01:48:54 step 145780 loss=3.3204 lr=3.84e-05 tok=19.192B dt=10685 wait=152 +01:49:58 step 145800 loss=3.3999 lr=3.84e-05 tok=19.194B dt=10748 wait=152 +01:50:12 pushed ckpt_w134.pt,log_w134.txt +01:54:21 step 145820 loss=3.4098 lr=3.84e-05 tok=19.197B dt=11011 wait=153 +01:54:31 global r=574 load 4.272 -> 4.278 +01:55:33 step 145820 loss=3.3951 lr=3.84e-05 tok=19.199B dt=11084 wait=153 +01:56:36 step 145840 loss=3.4089 lr=3.84e-05 tok=19.202B dt=11146 wait=153 +01:57:39 step 145860 loss=3.3926 lr=3.84e-05 tok=19.205B dt=11209 wait=153 +01:58:42 step 145880 loss=3.2906 lr=3.84e-05 tok=19.207B dt=11272 wait=154 +01:59:44 step 145900 loss=3.4186 lr=3.83e-05 tok=19.210B dt=11334 wait=154 +02:00:47 step 145920 loss=3.4138 lr=3.83e-05 tok=19.213B dt=11397 wait=154 +02:01:50 step 145940 loss=3.3425 lr=3.83e-05 tok=19.215B dt=11460 wait=155 +02:02:34 pushed ckpt_w134.pt,log_w134.txt +02:06:08 step 145960 loss=3.4769 lr=3.83e-05 tok=19.218B dt=11719 wait=155 +02:06:20 global r=575 load 4.275 -> 4.277 +02:06:51 step 145960 loss=3.5019 lr=3.83e-05 tok=19.219B dt=11762 wait=155 +02:07:54 step 145980 loss=3.3491 lr=3.83e-05 tok=19.222B dt=11824 wait=155 +02:08:57 step 146000 loss=3.3208 lr=3.83e-05 tok=19.224B dt=11887 wait=156 +02:09:59 step 146020 loss=3.4845 lr=3.82e-05 tok=19.227B dt=11950 wait=156 +02:11:02 step 146040 loss=3.3633 lr=3.82e-05 tok=19.230B dt=12012 wait=156 +02:12:05 step 146060 loss=3.3764 lr=3.82e-05 tok=19.232B dt=12075 wait=157 +02:13:07 step 146080 loss=3.3703 lr=3.82e-05 tok=19.235B dt=12138 wait=157 +02:14:11 step 146100 loss=3.3993 lr=3.82e-05 tok=19.237B dt=12201 wait=157 +02:14:24 pushed ckpt_w134.pt,log_w134.txt +02:17:25 waited 181s for global r=576 +02:17:37 global r=576 load 4.276 -> 4.276 +02:18:42 step 146120 loss=3.3759 lr=3.82e-05 tok=19.240B dt=12472 wait=158 +02:19:45 step 146140 loss=3.4114 lr=3.81e-05 tok=19.243B dt=12535 wait=158 +02:20:47 step 146160 loss=3.3715 lr=3.81e-05 tok=19.245B dt=12597 wait=158 +02:21:50 step 146180 loss=3.3719 lr=3.81e-05 tok=19.248B dt=12660 wait=159 +02:22:53 step 146200 loss=3.3550 lr=3.81e-05 tok=19.251B dt=12723 wait=159 +02:23:55 step 146220 loss=3.4315 lr=3.81e-05 tok=19.253B dt=12786 wait=159 +02:24:58 step 146240 loss=3.2661 lr=3.81e-05 tok=19.256B dt=12848 wait=160 +02:25:44 pushed ckpt_w134.pt,log_w134.txt +02:29:18 step 146260 loss=3.5202 lr=3.81e-05 tok=19.258B dt=13109 wait=160 +02:29:25 global r=577 load 4.268 -> 4.272 +02:29:57 step 146260 loss=3.5079 lr=3.81e-05 tok=19.260B dt=13147 wait=160 +02:30:59 step 146280 loss=3.3243 lr=3.80e-05 tok=19.262B dt=13210 wait=160 +02:32:02 step 146300 loss=3.3504 lr=3.80e-05 tok=19.265B dt=13272 wait=161 +02:33:04 step 146320 loss=3.4657 lr=3.80e-05 tok=19.268B dt=13335 wait=161 +02:34:07 step 146340 loss=3.3003 lr=3.80e-05 tok=19.270B dt=13398 wait=161 +02:35:10 step 146360 loss=3.4498 lr=3.80e-05 tok=19.273B dt=13460 wait=162 +02:36:13 step 146380 loss=3.4139 lr=3.80e-05 tok=19.275B dt=13523 wait=162 +02:37:15 step 146400 loss=3.4158 lr=3.79e-05 tok=19.278B dt=13586 wait=162 +02:37:28 pushed ckpt_w134.pt,log_w134.txt +02:40:29 waited 181s for global r=578 +02:40:39 global r=578 load 4.272 -> 4.272 +02:41:44 step 146420 loss=3.3731 lr=3.79e-05 tok=19.281B dt=13854 wait=163 +02:42:47 step 146440 loss=3.4684 lr=3.79e-05 tok=19.283B dt=13917 wait=163 +02:43:49 step 146460 loss=3.3844 lr=3.79e-05 tok=19.286B dt=13979 wait=163 +02:44:51 step 146480 loss=3.3634 lr=3.79e-05 tok=19.289B dt=14042 wait=164 +02:45:53 step 146500 loss=3.4275 lr=3.79e-05 tok=19.291B dt=14104 wait=164 +02:46:56 step 146520 loss=3.4810 lr=3.78e-05 tok=19.294B dt=14166 wait=164 +02:47:59 step 146540 loss=3.3360 lr=3.78e-05 tok=19.296B dt=14229 wait=165 +02:48:42 pushed ckpt_w134.pt,log_w134.txt +02:52:16 step 146560 loss=3.4403 lr=3.78e-05 tok=19.299B dt=14487 wait=165 +02:52:26 global r=579 load 4.277 -> 4.277 +02:52:57 step 146560 loss=3.4350 lr=3.78e-05 tok=19.300B dt=14527 wait=165 +02:54:00 step 146580 loss=3.2742 lr=3.78e-05 tok=19.303B dt=14590 wait=165 +02:55:03 step 146600 loss=3.3568 lr=3.78e-05 tok=19.306B dt=14653 wait=166 +02:56:05 step 146620 loss=3.4434 lr=3.78e-05 tok=19.308B dt=14716 wait=166 +02:57:08 step 146640 loss=3.3525 lr=3.78e-05 tok=19.311B dt=14778 wait=166 +02:58:10 step 146660 loss=3.4567 lr=3.77e-05 tok=19.313B dt=14841 wait=167 +02:59:13 step 146680 loss=3.3390 lr=3.77e-05 tok=19.316B dt=14903 wait=167 +03:00:16 step 146700 loss=3.3689 lr=3.77e-05 tok=19.319B dt=14966 wait=167 +03:00:30 pushed ckpt_w134.pt,log_w134.txt +03:03:32 waited 181s for global r=580 +03:03:40 global r=580 load 4.293 -> 4.293 +03:04:45 step 146720 loss=3.3585 lr=3.77e-05 tok=19.321B dt=15235 wait=168 +03:05:48 step 146740 loss=3.3995 lr=3.77e-05 tok=19.324B dt=15298 wait=168 +03:06:51 step 146760 loss=3.3341 lr=3.77e-05 tok=19.327B dt=15362 wait=168 +03:07:54 step 146780 loss=3.2358 lr=3.76e-05 tok=19.329B dt=15425 wait=169 +03:08:57 step 146800 loss=3.3978 lr=3.76e-05 tok=19.332B dt=15487 wait=169 +03:10:00 step 146820 loss=3.4905 lr=3.76e-05 tok=19.334B dt=15550 wait=169 +03:11:02 step 146840 loss=3.3612 lr=3.76e-05 tok=19.337B dt=15612 wait=169 +03:11:46 pushed ckpt_w134.pt,log_w134.txt +03:15:20 step 146860 loss=3.4616 lr=3.76e-05 tok=19.340B dt=15870 wait=170 +03:15:27 global r=581 load 4.266 -> 4.263 +03:15:59 step 146860 loss=3.3845 lr=3.76e-05 tok=19.341B dt=15909 wait=170 +03:17:01 step 146880 loss=3.3315 lr=3.76e-05 tok=19.344B dt=15972 wait=170 +03:18:04 step 146900 loss=3.3249 lr=3.75e-05 tok=19.346B dt=16035 wait=171 +03:19:07 step 146920 loss=3.4304 lr=3.75e-05 tok=19.349B dt=16097 wait=171 +03:20:09 step 146940 loss=3.3526 lr=3.75e-05 tok=19.351B dt=16160 wait=171 +03:21:12 step 146960 loss=3.4180 lr=3.75e-05 tok=19.354B dt=16223 wait=172 +03:22:15 step 146980 loss=3.3448 lr=3.75e-05 tok=19.357B dt=16286 wait=172 +03:23:18 step 147000 loss=3.3369 lr=3.75e-05 tok=19.359B dt=16348 wait=172 +03:23:34 pushed ckpt_w134.pt,log_w134.txt +03:26:36 waited 182s for global r=582 +03:26:47 global r=582 load 4.263 -> 4.263 +03:27:52 step 147020 loss=3.4425 lr=3.75e-05 tok=19.362B dt=16623 wait=172 +03:28:55 step 147040 loss=3.3930 lr=3.74e-05 tok=19.365B dt=16686 wait=173 +03:29:58 step 147060 loss=3.3948 lr=3.74e-05 tok=19.367B dt=16748 wait=173 +03:31:01 step 147080 loss=3.3332 lr=3.74e-05 tok=19.370B dt=16811 wait=173 +03:32:04 step 147100 loss=3.3747 lr=3.74e-05 tok=19.372B dt=16875 wait=174 +03:33:07 step 147120 loss=3.4460 lr=3.74e-05 tok=19.375B dt=16937 wait=174 +03:34:10 step 147140 loss=3.2981 lr=3.74e-05 tok=19.378B dt=17000 wait=174 +03:34:54 pushed ckpt_w134.pt,log_w134.txt +03:38:27 step 147160 loss=3.4628 lr=3.73e-05 tok=19.380B dt=17258 wait=175 +03:38:36 global r=583 load 4.258 -> 4.258 +03:39:07 step 147160 loss=3.4808 lr=3.73e-05 tok=19.382B dt=17298 wait=175 +03:40:10 step 147180 loss=3.2833 lr=3.73e-05 tok=19.384B dt=17361 wait=175 +03:41:13 step 147200 loss=3.3998 lr=3.73e-05 tok=19.387B dt=17423 wait=175 +03:42:16 step 147220 loss=3.4147 lr=3.73e-05 tok=19.389B dt=17486 wait=176 +03:43:19 step 147240 loss=3.2577 lr=3.73e-05 tok=19.392B dt=17549 wait=176 +03:44:22 step 147260 loss=3.4432 lr=3.73e-05 tok=19.395B dt=17612 wait=176 +03:45:24 step 147280 loss=3.3871 lr=3.72e-05 tok=19.397B dt=17675 wait=177 +03:46:27 step 147300 loss=3.3630 lr=3.72e-05 tok=19.400B dt=17738 wait=177 +03:46:41 pushed ckpt_w134.pt,log_w134.txt +03:49:42 waited 181s for global r=584 +03:49:52 global r=584 load 4.251 -> 4.251 +03:50:56 step 147320 loss=3.3673 lr=3.72e-05 tok=19.403B dt=18006 wait=177 +03:51:59 step 147340 loss=3.3416 lr=3.72e-05 tok=19.405B dt=18069 wait=178 +03:53:01 step 147360 loss=3.4110 lr=3.72e-05 tok=19.408B dt=18131 wait=178 +03:54:04 step 147380 loss=3.3468 lr=3.72e-05 tok=19.410B dt=18194 wait=178 +03:55:07 step 147400 loss=3.3410 lr=3.72e-05 tok=19.413B dt=18257 wait=179 +03:56:10 step 147420 loss=3.4243 lr=3.71e-05 tok=19.416B dt=18320 wait=179 +03:57:12 step 147440 loss=3.3420 lr=3.71e-05 tok=19.418B dt=18383 wait=179 +03:57:56 pushed ckpt_w134.pt,log_w134.txt +04:01:29 step 147460 loss=3.4585 lr=3.71e-05 tok=19.421B dt=18640 wait=179 +04:01:35 global r=585 load 4.264 -> 4.265 +04:02:07 step 147460 loss=3.4179 lr=3.71e-05 tok=19.422B dt=18677 wait=180 +04:03:10 step 147480 loss=3.2743 lr=3.71e-05 tok=19.425B dt=18740 wait=180 +04:04:13 step 147500 loss=3.2847 lr=3.71e-05 tok=19.427B dt=18803 wait=180 +04:05:15 step 147520 loss=3.4345 lr=3.71e-05 tok=19.430B dt=18866 wait=181 +04:06:19 step 147540 loss=3.2468 lr=3.70e-05 tok=19.433B dt=18929 wait=181 +04:07:21 step 147560 loss=3.4012 lr=3.70e-05 tok=19.435B dt=18992 wait=181 +04:08:24 step 147580 loss=3.3919 lr=3.70e-05 tok=19.438B dt=19054 wait=182 +04:09:27 step 147600 loss=3.3739 lr=3.70e-05 tok=19.441B dt=19117 wait=182 +04:09:41 pushed ckpt_w134.pt,log_w134.txt +04:13:22 global r=586 load 4.256 -> 4.264 +04:14:24 step 147620 loss=3.3960 lr=3.70e-05 tok=19.445B dt=19415 wait=182 +04:15:27 step 147640 loss=3.3484 lr=3.70e-05 tok=19.447B dt=19477 wait=183 +04:16:29 step 147660 loss=3.3597 lr=3.69e-05 tok=19.450B dt=19540 wait=183 +04:17:32 step 147680 loss=3.2777 lr=3.69e-05 tok=19.452B dt=19602 wait=183 +04:18:34 step 147700 loss=3.3361 lr=3.69e-05 tok=19.455B dt=19664 wait=184 +04:19:37 step 147720 loss=3.4181 lr=3.69e-05 tok=19.458B dt=19727 wait=184 +04:20:39 step 147740 loss=3.3826 lr=3.69e-05 tok=19.460B dt=19789 wait=184 +04:21:27 pushed ckpt_w134.pt,log_w134.txt +04:25:00 step 147760 loss=3.4809 lr=3.69e-05 tok=19.463B dt=20051 wait=185 +04:26:03 step 147780 loss=3.4013 lr=3.69e-05 tok=19.466B dt=20114 wait=185 +04:27:06 step 147800 loss=3.3636 lr=3.68e-05 tok=19.468B dt=20176 wait=185 +04:28:08 step 147820 loss=3.2491 lr=3.68e-05 tok=19.471B dt=20238 wait=186 +04:29:10 step 147840 loss=3.3789 lr=3.68e-05 tok=19.473B dt=20301 wait=186 +04:30:13 step 147860 loss=3.3714 lr=3.68e-05 tok=19.476B dt=20363 wait=186 +04:31:15 step 147880 loss=3.3661 lr=3.68e-05 tok=19.479B dt=20426 wait=186 +04:32:19 step 147900 loss=3.4699 lr=3.68e-05 tok=19.481B dt=20489 wait=187 +04:32:37 pushed ckpt_w134.pt,log_w134.txt +04:32:37 waited 0s for global r=587 +04:32:46 global r=587 load 4.267 -> 4.254 +04:33:19 step 147760 loss=3.4763 lr=3.69e-05 tok=19.483B dt=20549 wait=187 +04:34:22 step 147780 loss=3.4096 lr=3.69e-05 tok=19.485B dt=20612 wait=187 +04:35:24 step 147800 loss=3.3879 lr=3.68e-05 tok=19.488B dt=20674 wait=188 +04:36:27 step 147820 loss=3.4177 lr=3.68e-05 tok=19.490B dt=20737 wait=188 +04:37:30 step 147840 loss=3.3614 lr=3.68e-05 tok=19.493B dt=20800 wait=188 +04:38:32 step 147860 loss=3.4665 lr=3.68e-05 tok=19.496B dt=20863 wait=189 +04:39:35 step 147880 loss=3.4035 lr=3.68e-05 tok=19.498B dt=20926 wait=189 +04:40:38 step 147900 loss=3.3864 lr=3.68e-05 tok=19.501B dt=20988 wait=189 +04:40:57 pushed ckpt_w134.pt,log_w134.txt +04:42:03 waited 66s for global r=588 +04:42:13 global r=588 load 4.285 -> 4.269 +04:43:18 step 147920 loss=3.3370 lr=3.67e-05 tok=19.504B dt=21148 wait=190 +04:44:21 step 147940 loss=3.3697 lr=3.67e-05 tok=19.506B dt=21211 wait=190 +04:45:23 step 147960 loss=3.4056 lr=3.67e-05 tok=19.509B dt=21274 wait=190 +04:46:26 step 147980 loss=3.3431 lr=3.67e-05 tok=19.511B dt=21337 wait=190 +04:47:29 step 148000 loss=3.3607 lr=3.67e-05 tok=19.514B dt=21399 wait=191 +04:48:32 step 148020 loss=3.4308 lr=3.67e-05 tok=19.517B dt=21462 wait=191 +04:49:34 step 148040 loss=3.3223 lr=3.67e-05 tok=19.519B dt=21524 wait=191 +04:50:19 pushed ckpt_w134.pt,log_w134.txt +04:51:35 waited 76s for global r=589 +04:51:47 global r=589 load 4.280 -> 4.269 +04:52:20 step 148060 loss=3.4183 lr=3.66e-05 tok=19.522B dt=21690 wait=192 +04:53:23 step 148080 loss=3.3326 lr=3.66e-05 tok=19.524B dt=21753 wait=192 +04:54:25 step 148100 loss=3.3507 lr=3.66e-05 tok=19.527B dt=21816 wait=192 +04:55:28 step 148120 loss=3.4697 lr=3.66e-05 tok=19.530B dt=21878 wait=193 +04:56:31 step 148140 loss=3.4199 lr=3.66e-05 tok=19.532B dt=21941 wait=193 +04:57:34 step 148160 loss=3.4111 lr=3.66e-05 tok=19.535B dt=22004 wait=193 +04:58:36 step 148180 loss=3.3851 lr=3.65e-05 tok=19.538B dt=22066 wait=194 +04:59:39 step 148200 loss=3.3655 lr=3.65e-05 tok=19.540B dt=22129 wait=194 +04:59:56 pushed ckpt_w134.pt,log_w134.txt +05:01:14 waited 78s for global r=590 +05:01:31 global r=590 load 4.303 -> 4.275 +05:02:35 step 148220 loss=3.3838 lr=3.65e-05 tok=19.543B dt=22306 wait=194 +05:03:38 step 148240 loss=3.4317 lr=3.65e-05 tok=19.545B dt=22369 wait=195 +05:04:40 step 148260 loss=3.4307 lr=3.65e-05 tok=19.548B dt=22431 wait=195 +05:05:43 step 148280 loss=3.3221 lr=3.65e-05 tok=19.551B dt=22493 wait=195 +05:06:46 step 148300 loss=3.3132 lr=3.64e-05 tok=19.553B dt=22556 wait=196 +05:07:49 step 148320 loss=3.4444 lr=3.64e-05 tok=19.556B dt=22619 wait=196 +05:08:52 step 148340 loss=3.3577 lr=3.64e-05 tok=19.559B dt=22682 wait=196 +05:09:38 pushed ckpt_w134.pt,log_w134.txt +05:10:39 waited 61s for global r=591 +05:10:51 global r=591 load 4.260 -> 4.262 +05:11:23 step 148360 loss=3.4533 lr=3.64e-05 tok=19.561B dt=22833 wait=196 +05:12:26 step 148380 loss=3.2991 lr=3.64e-05 tok=19.564B dt=22896 wait=197 +05:13:29 step 148400 loss=3.2974 lr=3.64e-05 tok=19.566B dt=22959 wait=197 +05:14:31 step 148420 loss=3.4367 lr=3.64e-05 tok=19.569B dt=23022 wait=197 +05:15:34 step 148440 loss=3.3634 lr=3.63e-05 tok=19.572B dt=23084 wait=198 +05:16:37 step 148460 loss=3.3886 lr=3.63e-05 tok=19.574B dt=23147 wait=198 +05:17:40 step 148480 loss=3.3511 lr=3.63e-05 tok=19.577B dt=23210 wait=198 +05:18:43 step 148500 loss=3.3618 lr=3.63e-05 tok=19.580B dt=23273 wait=199 +05:18:58 pushed ckpt_w134.pt,log_w134.txt +05:20:13 waited 76s for global r=592 +05:20:25 global r=592 load 4.273 -> 4.265 +05:21:30 step 148520 loss=3.2949 lr=3.63e-05 tok=19.582B dt=23440 wait=199 +05:22:33 step 148540 loss=3.4195 lr=3.63e-05 tok=19.585B dt=23503 wait=199 +05:23:35 step 148560 loss=3.4264 lr=3.62e-05 tok=19.587B dt=23565 wait=200 +05:24:37 step 148580 loss=3.3716 lr=3.62e-05 tok=19.590B dt=23628 wait=200 +05:25:40 step 148600 loss=3.3278 lr=3.62e-05 tok=19.593B dt=23690 wait=200 +05:26:43 step 148620 loss=3.4632 lr=3.62e-05 tok=19.595B dt=23753 wait=201 +05:27:46 step 148640 loss=3.4486 lr=3.62e-05 tok=19.598B dt=23816 wait=201 +05:28:36 pushed ckpt_w134.pt,log_w134.txt +05:29:36 waited 61s for global r=593 +05:29:44 global r=593 load 4.270 -> 4.271 +05:30:17 step 148660 loss=3.5733 lr=3.62e-05 tok=19.601B dt=23967 wait=201 +05:31:20 step 148680 loss=3.3244 lr=3.62e-05 tok=19.603B dt=24030 wait=201 +05:32:22 step 148700 loss=3.3567 lr=3.61e-05 tok=19.606B dt=24092 wait=202 +05:33:24 step 148720 loss=3.4065 lr=3.61e-05 tok=19.608B dt=24155 wait=202 +05:34:27 step 148740 loss=3.3407 lr=3.61e-05 tok=19.611B dt=24217 wait=202 +05:35:29 step 148760 loss=3.4387 lr=3.61e-05 tok=19.614B dt=24279 wait=203 +05:36:32 step 148780 loss=3.3971 lr=3.61e-05 tok=19.616B dt=24342 wait=203 +05:37:34 step 148800 loss=3.4128 lr=3.61e-05 tok=19.619B dt=24405 wait=203 +05:37:51 pushed ckpt_w134.pt,log_w134.txt +05:38:54 waited 63s for global r=594 +05:39:02 global r=594 load 4.278 -> 4.266 +05:40:07 step 148820 loss=3.3851 lr=3.60e-05 tok=19.621B dt=24557 wait=204 +05:41:10 step 148840 loss=3.3484 lr=3.60e-05 tok=19.624B dt=24620 wait=204 +05:42:13 step 148860 loss=3.3499 lr=3.60e-05 tok=19.627B dt=24684 wait=204 +05:43:16 step 148880 loss=3.3208 lr=3.60e-05 tok=19.629B dt=24747 wait=204 +05:44:19 step 148900 loss=3.3588 lr=3.60e-05 tok=19.632B dt=24809 wait=205 +05:45:22 step 148920 loss=3.2814 lr=3.60e-05 tok=19.635B dt=24872 wait=205 +05:46:24 step 148940 loss=3.3480 lr=3.60e-05 tok=19.637B dt=24935 wait=205 +05:47:14 pushed ckpt_w134.pt,log_w134.txt +05:48:15 waited 61s for global r=595 +05:48:24 global r=595 load 4.278 -> 4.275 +05:48:58 step 148960 loss=3.3978 lr=3.59e-05 tok=19.640B dt=25088 wait=206 +05:50:00 step 148980 loss=3.3526 lr=3.59e-05 tok=19.642B dt=25151 wait=206 +05:51:03 step 149000 loss=3.3339 lr=3.59e-05 tok=19.645B dt=25213 wait=206 +05:52:06 step 149020 loss=3.4117 lr=3.59e-05 tok=19.648B dt=25277 wait=207 +05:53:09 step 149040 loss=3.3511 lr=3.59e-05 tok=19.650B dt=25339 wait=207 +05:54:11 step 149060 loss=3.4078 lr=3.59e-05 tok=19.653B dt=25402 wait=207 +05:55:14 step 149080 loss=3.3624 lr=3.58e-05 tok=19.656B dt=25464 wait=208 +05:56:17 step 149100 loss=3.4133 lr=3.58e-05 tok=19.658B dt=25527 wait=208 +05:56:28 pushed ckpt_w134.pt,log_w134.txt +05:57:44 waited 76s for global r=596 +05:57:53 global r=596 load 4.288 -> 4.276 +05:58:58 step 149120 loss=3.2455 lr=3.58e-05 tok=19.661B dt=25689 wait=208 +06:00:01 step 149140 loss=3.3990 lr=3.58e-05 tok=19.663B dt=25752 wait=209 +06:01:04 step 149160 loss=3.4053 lr=3.58e-05 tok=19.666B dt=25814 wait=209 +06:02:07 step 149180 loss=3.3656 lr=3.58e-05 tok=19.669B dt=25878 wait=209 +06:03:11 step 149200 loss=3.3945 lr=3.58e-05 tok=19.671B dt=25941 wait=210 +06:04:14 step 149220 loss=3.5238 lr=3.57e-05 tok=19.674B dt=26004 wait=210 +06:05:16 step 149240 loss=3.2968 lr=3.57e-05 tok=19.677B dt=26067 wait=210 +06:06:09 pushed ckpt_w134.pt,log_w134.txt +06:06:58 waited 46s for global r=597 +06:07:09 global r=597 load 4.264 -> 4.263 +06:07:42 step 149260 loss=3.4767 lr=3.57e-05 tok=19.679B dt=26212 wait=210 +06:08:45 step 149280 loss=3.3031 lr=3.57e-05 tok=19.682B dt=26275 wait=211 +06:09:48 step 149300 loss=3.3844 lr=3.57e-05 tok=19.684B dt=26338 wait=211 +06:10:50 step 149320 loss=3.4367 lr=3.57e-05 tok=19.687B dt=26401 wait=211 +06:11:53 step 149340 loss=3.5248 lr=3.56e-05 tok=19.690B dt=26464 wait=212 +06:12:57 step 149360 loss=3.4062 lr=3.56e-05 tok=19.692B dt=26527 wait=212 +06:13:58 step 149380 loss=3.3742 lr=3.56e-05 tok=19.695B dt=26589 wait=212 +06:15:01 step 149400 loss=3.4630 lr=3.56e-05 tok=19.698B dt=26651 wait=213 +06:15:17 pushed ckpt_w134.pt,log_w134.txt +06:16:23 waited 66s for global r=598 +06:16:38 global r=598 load 4.275 -> 4.268 +06:17:42 step 149420 loss=3.3325 lr=3.56e-05 tok=19.700B dt=26812 wait=213 +06:18:44 step 149440 loss=3.4684 lr=3.56e-05 tok=19.703B dt=26875 wait=213 +06:19:47 step 149460 loss=3.4005 lr=3.56e-05 tok=19.705B dt=26937 wait=214 +06:20:50 step 149480 loss=3.3518 lr=3.55e-05 tok=19.708B dt=27000 wait=214 +06:21:52 step 149500 loss=3.3942 lr=3.55e-05 tok=19.711B dt=27063 wait=214 +06:22:55 step 149520 loss=3.4524 lr=3.55e-05 tok=19.713B dt=27125 wait=215 +06:23:58 step 149540 loss=3.3921 lr=3.55e-05 tok=19.716B dt=27188 wait=215 +06:24:49 pushed ckpt_w134.pt,log_w134.txt +06:25:50 waited 61s for global r=599 +06:25:57 global r=599 load 4.285 -> 4.277 +06:26:30 step 149560 loss=3.4835 lr=3.55e-05 tok=19.718B dt=27341 wait=215 +06:27:34 step 149580 loss=3.5675 lr=3.55e-05 tok=19.721B dt=27404 wait=216 +06:28:37 step 149600 loss=3.2901 lr=3.55e-05 tok=19.724B dt=27467 wait=216 +06:29:40 step 149620 loss=3.4102 lr=3.54e-05 tok=19.726B dt=27530 wait=216 +06:30:43 step 149640 loss=3.2830 lr=3.54e-05 tok=19.729B dt=27593 wait=217 +06:31:45 step 149660 loss=3.3903 lr=3.54e-05 tok=19.732B dt=27656 wait=217 +06:32:49 step 149680 loss=3.2999 lr=3.54e-05 tok=19.734B dt=27719 wait=217 +06:33:51 step 149700 loss=3.4179 lr=3.54e-05 tok=19.737B dt=27781 wait=218 +06:34:08 pushed ckpt_w134.pt,log_w134.txt +06:35:09 waited 61s for global r=600 +06:35:18 global r=600 load 4.293 -> 4.284 +06:36:22 step 149720 loss=3.3915 lr=3.54e-05 tok=19.739B dt=27932 wait=218 +06:37:24 step 149740 loss=3.4278 lr=3.53e-05 tok=19.742B dt=27995 wait=218 +06:38:27 step 149760 loss=3.3438 lr=3.53e-05 tok=19.745B dt=28057 wait=219 +06:39:30 step 149780 loss=3.3574 lr=3.53e-05 tok=19.747B dt=28120 wait=219 +06:40:33 step 149800 loss=3.3307 lr=3.53e-05 tok=19.750B dt=28183 wait=219 +06:41:36 step 149820 loss=3.4516 lr=3.53e-05 tok=19.753B dt=28246 wait=220 +06:42:39 step 149840 loss=3.2973 lr=3.53e-05 tok=19.755B dt=28309 wait=220 +06:43:25 pushed ckpt_w134.pt,log_w134.txt +06:44:26 waited 61s for global r=601 +06:44:37 global r=601 load 4.284 -> 4.285 +06:45:10 step 149860 loss=3.4600 lr=3.53e-05 tok=19.758B dt=28461 wait=220 +06:46:13 step 149880 loss=3.3593 lr=3.52e-05 tok=19.760B dt=28523 wait=221 +06:47:15 step 149900 loss=3.3427 lr=3.52e-05 tok=19.763B dt=28586 wait=221 +06:48:18 step 149920 loss=3.3910 lr=3.52e-05 tok=19.766B dt=28648 wait=221 +06:49:20 step 149940 loss=3.2996 lr=3.52e-05 tok=19.768B dt=28711 wait=221 +06:50:23 step 149960 loss=3.4590 lr=3.52e-05 tok=19.771B dt=28773 wait=222 +06:51:25 step 149980 loss=3.4007 lr=3.52e-05 tok=19.774B dt=28835 wait=222 +06:52:27 step 150000 loss=3.3998 lr=3.51e-05 tok=19.776B dt=28897 wait=222 +06:52:42 pushed ckpt_w134.pt,log_w134.txt +06:53:43 waited 61s for global r=602 +06:53:52 global r=602 load 4.293 -> 4.277 +06:54:57 step 150020 loss=3.4498 lr=3.51e-05 tok=19.779B dt=29048 wait=223 +06:56:00 step 150040 loss=3.3862 lr=3.51e-05 tok=19.781B dt=29111 wait=223 +06:56:24 mix source 1 lap 1 +06:57:03 step 150060 loss=3.4274 lr=3.51e-05 tok=19.784B dt=29174 wait=223 +06:58:06 step 150080 loss=3.3688 lr=3.51e-05 tok=19.787B dt=29237 wait=224 +06:59:09 step 150100 loss=3.4074 lr=3.51e-05 tok=19.789B dt=29299 wait=224 +07:00:11 step 150120 loss=3.4323 lr=3.51e-05 tok=19.792B dt=29362 wait=224 +07:01:14 step 150140 loss=3.3693 lr=3.50e-05 tok=19.794B dt=29424 wait=225 +07:02:02 pushed ckpt_w134.pt,log_w134.txt +07:03:05 waited 62s for global r=603 +07:03:13 global r=603 load 4.280 -> 4.279 +07:03:45 step 150160 loss=3.4784 lr=3.50e-05 tok=19.797B dt=29576 wait=225 +07:04:48 step 150180 loss=3.3291 lr=3.50e-05 tok=19.800B dt=29638 wait=225 +07:05:50 step 150200 loss=3.3272 lr=3.50e-05 tok=19.802B dt=29700 wait=226 +07:06:53 step 150220 loss=3.4383 lr=3.50e-05 tok=19.805B dt=29763 wait=226 +07:07:56 step 150240 loss=3.3252 lr=3.50e-05 tok=19.808B dt=29826 wait=226 +07:08:59 step 150260 loss=3.4181 lr=3.50e-05 tok=19.810B dt=29889 wait=227 +07:10:02 step 150280 loss=3.3208 lr=3.49e-05 tok=19.813B dt=29952 wait=227 +07:11:04 step 150300 loss=3.3487 lr=3.49e-05 tok=19.815B dt=30014 wait=227 diff --git a/log_w135.txt b/log_w135.txt new file mode 100644 index 0000000000000000000000000000000000000000..b56517604f08e52edb3a77dd62599eae041c8a0a --- /dev/null +++ b/log_w135.txt @@ -0,0 +1,197 @@ +launch +04:18:34 name=w135 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 82/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +04:18:40 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +04:18:40 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +04:18:50 params 1.08B +04:18:51 resumed step=147600 seen=19.44B round=586 +04:18:51 stream salt=147600 +04:20:36 blocks compiled +04:20:36 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +04:23:01 step 147601 loss=3.1176 lr=3.70e-05 tok=19.441B dt=145 wait=105 +04:23:26 step 147602 loss=3.1899 lr=3.70e-05 tok=19.441B dt=170 wait=105 +04:24:30 step 147620 loss=3.4602 lr=3.70e-05 tok=19.443B dt=234 wait=105 +04:25:34 step 147640 loss=3.4319 lr=3.70e-05 tok=19.446B dt=297 wait=106 +04:26:37 step 147660 loss=3.4487 lr=3.69e-05 tok=19.448B dt=361 wait=106 +04:27:40 step 147680 loss=3.4851 lr=3.69e-05 tok=19.451B dt=423 wait=106 +04:28:43 step 147700 loss=3.5364 lr=3.69e-05 tok=19.454B dt=487 wait=107 +04:29:47 step 147720 loss=3.4443 lr=3.69e-05 tok=19.456B dt=550 wait=107 +04:30:50 step 147740 loss=3.4307 lr=3.69e-05 tok=19.459B dt=614 wait=107 +04:31:38 pushed ckpt_w135.pt,log_w135.txt +04:32:24 waited 45s for global r=587 +04:32:41 global r=587 load 4.252 -> 4.254 +04:33:22 step 147760 loss=3.5700 lr=3.69e-05 tok=19.462B dt=766 wait=108 +04:34:38 step 147780 loss=3.3249 lr=3.69e-05 tok=19.464B dt=842 wait=108 +04:35:41 step 147800 loss=3.3513 lr=3.68e-05 tok=19.467B dt=904 wait=109 +04:36:44 step 147820 loss=3.3956 lr=3.68e-05 tok=19.469B dt=968 wait=109 +04:37:47 step 147840 loss=3.3768 lr=3.68e-05 tok=19.472B dt=1031 wait=109 +04:38:51 step 147860 loss=3.4260 lr=3.68e-05 tok=19.475B dt=1095 wait=110 +04:39:54 step 147880 loss=3.3320 lr=3.68e-05 tok=19.477B dt=1158 wait=110 +04:40:57 step 147900 loss=3.4308 lr=3.68e-05 tok=19.480B dt=1221 wait=110 +04:41:10 pushed ckpt_w135.pt,log_w135.txt +04:41:55 waited 45s for global r=588 +04:42:05 global r=588 load 4.266 -> 4.269 +04:43:28 step 147920 loss=3.3681 lr=3.67e-05 tok=19.483B dt=1372 wait=111 +04:44:33 step 147940 loss=3.4000 lr=3.67e-05 tok=19.485B dt=1437 wait=111 +04:45:36 step 147960 loss=3.4064 lr=3.67e-05 tok=19.488B dt=1500 wait=111 +04:46:39 step 147980 loss=3.3520 lr=3.67e-05 tok=19.490B dt=1563 wait=112 +04:47:41 step 148000 loss=3.4154 lr=3.67e-05 tok=19.493B dt=1625 wait=112 +04:48:45 step 148020 loss=3.4175 lr=3.67e-05 tok=19.496B dt=1688 wait=112 +04:49:47 step 148040 loss=3.2466 lr=3.67e-05 tok=19.498B dt=1751 wait=113 +04:50:35 pushed ckpt_w135.pt,log_w135.txt +04:51:35 waited 60s for global r=589 +04:51:44 global r=589 load 4.281 -> 4.269 +04:52:26 step 148060 loss=3.4828 lr=3.66e-05 tok=19.501B dt=1910 wait=113 +04:53:41 step 148080 loss=3.4009 lr=3.66e-05 tok=19.504B dt=1985 wait=113 +04:54:44 step 148100 loss=3.3622 lr=3.66e-05 tok=19.506B dt=2048 wait=114 +04:55:49 step 148120 loss=3.4448 lr=3.66e-05 tok=19.509B dt=2112 wait=114 +04:56:53 step 148140 loss=3.3160 lr=3.66e-05 tok=19.511B dt=2177 wait=115 +04:57:56 step 148160 loss=3.4529 lr=3.66e-05 tok=19.514B dt=2240 wait=115 +04:59:00 step 148180 loss=3.3347 lr=3.65e-05 tok=19.517B dt=2304 wait=115 +05:00:04 step 148200 loss=3.4456 lr=3.65e-05 tok=19.519B dt=2367 wait=116 +05:00:15 pushed ckpt_w135.pt,log_w135.txt +05:01:16 waited 60s for global r=590 +05:01:22 global r=590 load 4.263 -> 4.275 +05:02:41 step 148220 loss=3.3874 lr=3.65e-05 tok=19.522B dt=2525 wait=116 +05:03:49 step 148240 loss=3.4489 lr=3.65e-05 tok=19.524B dt=2593 wait=116 +05:04:53 step 148260 loss=3.4800 lr=3.65e-05 tok=19.527B dt=2657 wait=117 +05:05:56 step 148280 loss=3.4000 lr=3.65e-05 tok=19.530B dt=2720 wait=117 +05:06:59 step 148300 loss=3.4526 lr=3.64e-05 tok=19.532B dt=2783 wait=117 +05:08:03 step 148320 loss=3.4616 lr=3.64e-05 tok=19.535B dt=2847 wait=117 +05:09:06 step 148340 loss=3.3617 lr=3.64e-05 tok=19.538B dt=2910 wait=118 +05:09:50 pushed ckpt_w135.pt,log_w135.txt +05:10:35 waited 45s for global r=591 +05:10:42 global r=591 load 4.284 -> 4.262 +05:11:16 step 148360 loss=3.4195 lr=3.64e-05 tok=19.540B dt=3039 wait=118 +05:12:37 step 148380 loss=3.3529 lr=3.64e-05 tok=19.543B dt=3121 wait=119 +05:13:41 step 148400 loss=3.3270 lr=3.64e-05 tok=19.545B dt=3185 wait=119 +05:14:45 step 148420 loss=3.4946 lr=3.64e-05 tok=19.548B dt=3249 wait=119 +05:15:48 step 148440 loss=3.4387 lr=3.63e-05 tok=19.551B dt=3312 wait=120 +05:16:51 step 148460 loss=3.4709 lr=3.63e-05 tok=19.553B dt=3375 wait=120 +05:17:54 step 148480 loss=3.3491 lr=3.63e-05 tok=19.556B dt=3438 wait=120 +05:18:58 step 148500 loss=3.4424 lr=3.63e-05 tok=19.559B dt=3502 wait=121 +05:19:12 pushed ckpt_w135.pt,log_w135.txt +05:20:12 waited 60s for global r=592 +05:20:19 global r=592 load 4.267 -> 4.265 +05:21:36 step 148520 loss=3.4178 lr=3.63e-05 tok=19.561B dt=3660 wait=121 +05:22:46 step 148540 loss=3.3952 lr=3.63e-05 tok=19.564B dt=3730 wait=121 +05:23:49 step 148560 loss=3.3617 lr=3.62e-05 tok=19.566B dt=3793 wait=122 +05:24:53 step 148580 loss=3.4462 lr=3.62e-05 tok=19.569B dt=3857 wait=122 +05:25:56 step 148600 loss=3.3315 lr=3.62e-05 tok=19.572B dt=3920 wait=122 +05:26:59 step 148620 loss=3.5293 lr=3.62e-05 tok=19.574B dt=3983 wait=123 +05:28:03 step 148640 loss=3.3858 lr=3.62e-05 tok=19.577B dt=4046 wait=123 +05:28:52 pushed ckpt_w135.pt,log_w135.txt +05:29:22 waited 30s for global r=593 +05:29:28 global r=593 load 4.282 -> 4.271 +05:30:01 step 148660 loss=3.4282 lr=3.62e-05 tok=19.580B dt=4165 wait=123 +05:31:19 step 148680 loss=3.3830 lr=3.62e-05 tok=19.582B dt=4243 wait=124 +05:32:26 step 148700 loss=3.4041 lr=3.61e-05 tok=19.585B dt=4310 wait=124 +05:33:29 step 148720 loss=3.4104 lr=3.61e-05 tok=19.587B dt=4373 wait=124 +05:34:32 step 148740 loss=3.3392 lr=3.61e-05 tok=19.590B dt=4436 wait=125 +05:35:35 step 148760 loss=3.3719 lr=3.61e-05 tok=19.593B dt=4499 wait=125 +05:36:37 step 148780 loss=3.3383 lr=3.61e-05 tok=19.595B dt=4561 wait=125 +05:37:40 step 148800 loss=3.3713 lr=3.61e-05 tok=19.598B dt=4624 wait=125 +05:37:54 pushed ckpt_w135.pt,log_w135.txt +05:38:55 waited 61s for global r=594 +05:39:02 global r=594 load 4.273 -> 4.266 +05:40:13 step 148820 loss=3.3152 lr=3.60e-05 tok=19.601B dt=4777 wait=126 +05:41:28 step 148840 loss=3.3595 lr=3.60e-05 tok=19.603B dt=4852 wait=126 +05:42:30 step 148860 loss=3.3776 lr=3.60e-05 tok=19.606B dt=4914 wait=126 +05:43:35 step 148880 loss=3.2255 lr=3.60e-05 tok=19.608B dt=4978 wait=127 +05:44:37 step 148900 loss=3.3301 lr=3.60e-05 tok=19.611B dt=5041 wait=127 +05:45:40 step 148920 loss=3.3610 lr=3.60e-05 tok=19.614B dt=5104 wait=127 +05:46:42 step 148940 loss=3.3187 lr=3.60e-05 tok=19.616B dt=5165 wait=128 +05:47:29 pushed ckpt_w135.pt,log_w135.txt +05:48:14 waited 45s for global r=595 +05:48:21 global r=595 load 4.287 -> 4.275 +05:48:54 step 148960 loss=3.3448 lr=3.59e-05 tok=19.619B dt=5298 wait=128 +05:50:05 step 148980 loss=3.2923 lr=3.59e-05 tok=19.621B dt=5369 wait=128 +05:51:20 step 149000 loss=3.3275 lr=3.59e-05 tok=19.624B dt=5444 wait=129 +05:52:22 step 149020 loss=3.3972 lr=3.59e-05 tok=19.627B dt=5506 wait=129 +05:53:25 step 149040 loss=3.3315 lr=3.59e-05 tok=19.629B dt=5569 wait=129 +05:54:27 step 149060 loss=3.3723 lr=3.59e-05 tok=19.632B dt=5631 wait=130 +05:55:30 step 149080 loss=3.2884 lr=3.58e-05 tok=19.635B dt=5694 wait=130 +05:56:33 step 149100 loss=3.3666 lr=3.58e-05 tok=19.637B dt=5756 wait=130 +05:56:45 pushed ckpt_w135.pt,log_w135.txt +05:57:45 waited 61s for global r=596 +05:57:51 global r=596 load 4.282 -> 4.276 +05:58:55 step 149120 loss=3.3756 lr=3.58e-05 tok=19.640B dt=5899 wait=131 +06:00:16 step 149140 loss=3.3914 lr=3.58e-05 tok=19.642B dt=5980 wait=131 +06:01:20 step 149160 loss=3.4059 lr=3.58e-05 tok=19.645B dt=6044 wait=131 +06:02:23 step 149180 loss=3.3190 lr=3.58e-05 tok=19.648B dt=6107 wait=132 +06:03:25 step 149200 loss=3.3475 lr=3.58e-05 tok=19.650B dt=6169 wait=132 +06:04:27 step 149220 loss=3.4035 lr=3.57e-05 tok=19.653B dt=6231 wait=132 +06:05:31 step 149240 loss=3.2043 lr=3.57e-05 tok=19.656B dt=6294 wait=133 +06:06:15 pushed ckpt_w135.pt,log_w135.txt +06:07:00 waited 45s for global r=597 +06:07:06 global r=597 load 4.262 -> 4.263 +06:07:39 step 149260 loss=3.4431 lr=3.57e-05 tok=19.658B dt=6423 wait=133 +06:08:42 step 149280 loss=3.2988 lr=3.57e-05 tok=19.661B dt=6486 wait=133 +06:10:04 step 149300 loss=3.3627 lr=3.57e-05 tok=19.663B dt=6568 wait=134 +06:11:07 step 149320 loss=3.4134 lr=3.57e-05 tok=19.666B dt=6631 wait=134 +06:12:10 step 149340 loss=3.2496 lr=3.56e-05 tok=19.669B dt=6694 wait=134 +06:13:12 step 149360 loss=3.3052 lr=3.56e-05 tok=19.671B dt=6756 wait=134 +06:14:15 step 149380 loss=3.3293 lr=3.56e-05 tok=19.674B dt=6819 wait=135 +06:15:18 step 149400 loss=3.3381 lr=3.56e-05 tok=19.677B dt=6882 wait=135 +06:15:33 pushed ckpt_w135.pt,log_w135.txt +06:16:18 waited 45s for global r=598 +06:16:28 global r=598 load 4.265 -> 4.268 +06:17:32 step 149420 loss=3.3630 lr=3.56e-05 tok=19.679B dt=7016 wait=135 +06:18:43 step 149440 loss=3.3897 lr=3.56e-05 tok=19.682B dt=7087 wait=136 +06:20:03 step 149460 loss=3.3694 lr=3.56e-05 tok=19.684B dt=7167 wait=136 +06:21:05 step 149480 loss=3.3351 lr=3.55e-05 tok=19.687B dt=7229 wait=136 +06:22:08 step 149500 loss=3.3576 lr=3.55e-05 tok=19.690B dt=7292 wait=137 +06:23:11 step 149520 loss=3.3715 lr=3.55e-05 tok=19.692B dt=7355 wait=137 +06:24:13 step 149540 loss=3.2874 lr=3.55e-05 tok=19.695B dt=7417 wait=137 +06:25:01 pushed ckpt_w135.pt,log_w135.txt +06:25:46 waited 45s for global r=599 +06:25:54 global r=599 load 4.286 -> 4.277 +06:26:27 step 149560 loss=3.4293 lr=3.55e-05 tok=19.698B dt=7551 wait=138 +06:27:29 step 149580 loss=3.3017 lr=3.55e-05 tok=19.700B dt=7613 wait=138 +06:28:45 step 149600 loss=3.3225 lr=3.55e-05 tok=19.703B dt=7689 wait=138 +06:29:54 step 149620 loss=3.4169 lr=3.54e-05 tok=19.705B dt=7758 wait=139 +06:30:57 step 149640 loss=3.2429 lr=3.54e-05 tok=19.708B dt=7820 wait=139 +06:31:59 step 149660 loss=3.3335 lr=3.54e-05 tok=19.711B dt=7883 wait=139 +06:33:01 step 149680 loss=3.3479 lr=3.54e-05 tok=19.713B dt=7945 wait=140 +06:34:04 step 149700 loss=3.3159 lr=3.54e-05 tok=19.716B dt=8008 wait=140 +06:34:16 pushed ckpt_w135.pt,log_w135.txt +06:35:01 waited 45s for global r=600 +06:35:10 global r=600 load 4.284 -> 4.284 +06:36:31 step 149720 loss=3.1953 lr=3.54e-05 tok=19.718B dt=8155 wait=140 +06:37:37 step 149740 loss=3.3035 lr=3.53e-05 tok=19.721B dt=8221 wait=141 +06:38:40 step 149760 loss=3.2386 lr=3.53e-05 tok=19.724B dt=8284 wait=141 +06:39:43 step 149780 loss=3.3882 lr=3.53e-05 tok=19.726B dt=8347 wait=141 +06:40:46 step 149800 loss=3.3127 lr=3.53e-05 tok=19.729B dt=8410 wait=142 +06:41:48 step 149820 loss=3.4713 lr=3.53e-05 tok=19.732B dt=8472 wait=142 +06:42:51 step 149840 loss=3.2563 lr=3.53e-05 tok=19.734B dt=8535 wait=142 +06:43:38 pushed ckpt_w135.pt,log_w135.txt +06:44:23 waited 45s for global r=601 +06:44:29 global r=601 load 4.301 -> 4.285 +06:45:06 step 149860 loss=3.4030 lr=3.53e-05 tok=19.737B dt=8670 wait=143 +06:46:24 step 149880 loss=3.2744 lr=3.52e-05 tok=19.739B dt=8748 wait=143 +06:47:27 step 149900 loss=3.2698 lr=3.52e-05 tok=19.742B dt=8811 wait=143 +06:48:30 step 149920 loss=3.3800 lr=3.52e-05 tok=19.745B dt=8874 wait=144 +06:49:32 step 149940 loss=3.2623 lr=3.52e-05 tok=19.747B dt=8936 wait=144 +06:50:35 step 149960 loss=3.4574 lr=3.52e-05 tok=19.750B dt=8999 wait=144 +06:51:38 step 149980 loss=3.3319 lr=3.52e-05 tok=19.753B dt=9062 wait=145 +06:52:42 step 150000 loss=3.3217 lr=3.51e-05 tok=19.755B dt=9125 wait=145 +06:52:55 pushed ckpt_w135.pt,log_w135.txt +06:53:41 waited 45s for global r=602 +06:53:47 global r=602 load 4.292 -> 4.277 +06:55:02 step 150020 loss=3.2701 lr=3.51e-05 tok=19.758B dt=9266 wait=145 +06:56:15 step 150040 loss=3.4169 lr=3.51e-05 tok=19.760B dt=9339 wait=146 +06:57:18 step 150060 loss=3.3477 lr=3.51e-05 tok=19.763B dt=9402 wait=146 +06:58:21 step 150080 loss=3.3460 lr=3.51e-05 tok=19.766B dt=9465 wait=146 +06:59:23 step 150100 loss=3.3386 lr=3.51e-05 tok=19.768B dt=9527 wait=146 +07:00:26 step 150120 loss=3.4114 lr=3.51e-05 tok=19.771B dt=9590 wait=147 +07:01:29 step 150140 loss=3.2448 lr=3.50e-05 tok=19.774B dt=9653 wait=147 +07:02:14 pushed ckpt_w135.pt,log_w135.txt +07:02:59 waited 45s for global r=603 +07:03:04 global r=603 load 4.299 -> 4.279 +07:03:37 step 150160 loss=3.4350 lr=3.50e-05 tok=19.776B dt=9781 wait=147 +07:04:48 step 150180 loss=3.3224 lr=3.50e-05 tok=19.779B dt=9851 wait=148 +07:06:02 step 150200 loss=3.2980 lr=3.50e-05 tok=19.781B dt=9926 wait=148 +07:07:04 step 150220 loss=3.4686 lr=3.50e-05 tok=19.784B dt=9988 wait=148 +07:08:07 step 150240 loss=3.2546 lr=3.50e-05 tok=19.787B dt=10051 wait=149 +07:09:10 step 150260 loss=3.4230 lr=3.50e-05 tok=19.789B dt=10114 wait=149 +07:10:12 step 150280 loss=3.3861 lr=3.49e-05 tok=19.792B dt=10176 wait=149 +07:11:15 step 150300 loss=3.3422 lr=3.49e-05 tok=19.794B dt=10239 wait=150 diff --git a/log_w136.txt b/log_w136.txt new file mode 100644 index 0000000000000000000000000000000000000000..21e4442b2cbc459dd2ef67f14c839ad1eced7fd7 --- /dev/null +++ b/log_w136.txt @@ -0,0 +1,197 @@ +launch +04:18:32 name=w136 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 83/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +04:18:39 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +04:18:39 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +04:18:48 params 1.08B +04:18:50 resumed step=147600 seen=19.44B round=586 +04:18:50 stream salt=147600 +04:20:38 blocks compiled +04:20:38 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +04:23:16 step 147601 loss=3.0016 lr=3.70e-05 tok=19.441B dt=158 wait=112 +04:23:42 step 147602 loss=3.1466 lr=3.70e-05 tok=19.441B dt=185 wait=112 +04:24:48 step 147620 loss=3.4782 lr=3.70e-05 tok=19.443B dt=250 wait=113 +04:25:51 step 147640 loss=3.4047 lr=3.70e-05 tok=19.446B dt=314 wait=113 +04:26:54 step 147660 loss=3.4040 lr=3.69e-05 tok=19.448B dt=377 wait=113 +04:27:57 step 147680 loss=3.5032 lr=3.69e-05 tok=19.451B dt=440 wait=114 +04:29:01 step 147700 loss=3.5505 lr=3.69e-05 tok=19.454B dt=503 wait=114 +04:30:04 step 147720 loss=3.3948 lr=3.69e-05 tok=19.456B dt=567 wait=114 +04:31:08 step 147740 loss=3.3764 lr=3.69e-05 tok=19.459B dt=630 wait=115 +04:31:53 pushed ckpt_w136.pt,log_w136.txt +04:32:24 waited 30s for global r=587 +04:32:42 global r=587 load 4.259 -> 4.254 +04:33:17 step 147760 loss=3.5249 lr=3.69e-05 tok=19.462B dt=759 wait=115 +04:34:20 step 147780 loss=3.3219 lr=3.69e-05 tok=19.464B dt=823 wait=116 +04:35:23 step 147800 loss=3.4100 lr=3.68e-05 tok=19.467B dt=885 wait=116 +04:36:26 step 147820 loss=3.4493 lr=3.68e-05 tok=19.469B dt=949 wait=116 +04:37:29 step 147840 loss=3.3396 lr=3.68e-05 tok=19.472B dt=1011 wait=117 +04:38:33 step 147860 loss=3.4507 lr=3.68e-05 tok=19.475B dt=1075 wait=117 +04:39:36 step 147880 loss=3.3562 lr=3.68e-05 tok=19.477B dt=1138 wait=117 +04:40:39 step 147900 loss=3.4158 lr=3.68e-05 tok=19.480B dt=1201 wait=118 +04:40:55 pushed ckpt_w136.pt,log_w136.txt +04:41:56 waited 61s for global r=588 +04:42:05 global r=588 load 4.282 -> 4.269 +04:43:10 step 147920 loss=3.4195 lr=3.67e-05 tok=19.483B dt=1352 wait=118 +04:44:13 step 147940 loss=3.4404 lr=3.67e-05 tok=19.485B dt=1415 wait=119 +04:45:16 step 147960 loss=3.3765 lr=3.67e-05 tok=19.488B dt=1479 wait=119 +04:46:19 step 147980 loss=3.3264 lr=3.67e-05 tok=19.490B dt=1542 wait=119 +04:47:22 step 148000 loss=3.4236 lr=3.67e-05 tok=19.493B dt=1604 wait=120 +04:48:24 step 148020 loss=3.4642 lr=3.67e-05 tok=19.496B dt=1667 wait=120 +04:49:27 step 148040 loss=3.2993 lr=3.67e-05 tok=19.498B dt=1730 wait=120 +04:50:13 pushed ckpt_w136.pt,log_w136.txt +04:51:29 waited 76s for global r=589 +04:51:39 global r=589 load 4.271 -> 4.269 +04:52:12 step 148060 loss=3.4649 lr=3.66e-05 tok=19.501B dt=1894 wait=121 +04:53:15 step 148080 loss=3.3196 lr=3.66e-05 tok=19.504B dt=1957 wait=121 +04:54:19 step 148100 loss=3.3390 lr=3.66e-05 tok=19.506B dt=2021 wait=121 +04:55:22 step 148120 loss=3.4537 lr=3.66e-05 tok=19.509B dt=2084 wait=122 +04:56:26 step 148140 loss=3.3683 lr=3.66e-05 tok=19.511B dt=2148 wait=122 +04:57:29 step 148160 loss=3.4510 lr=3.66e-05 tok=19.514B dt=2211 wait=123 +04:58:32 step 148180 loss=3.3786 lr=3.65e-05 tok=19.517B dt=2275 wait=123 +04:59:35 step 148200 loss=3.4353 lr=3.65e-05 tok=19.519B dt=2337 wait=123 +04:59:49 pushed ckpt_w136.pt,log_w136.txt +05:01:04 waited 76s for global r=590 +05:01:13 global r=590 load 4.273 -> 4.275 +05:02:17 step 148220 loss=3.4006 lr=3.65e-05 tok=19.522B dt=2500 wait=124 +05:03:21 step 148240 loss=3.3693 lr=3.65e-05 tok=19.524B dt=2563 wait=124 +05:04:24 step 148260 loss=3.4133 lr=3.65e-05 tok=19.527B dt=2626 wait=124 +05:05:27 step 148280 loss=3.4006 lr=3.65e-05 tok=19.530B dt=2690 wait=125 +05:06:30 step 148300 loss=3.4098 lr=3.64e-05 tok=19.532B dt=2752 wait=125 +05:07:34 step 148320 loss=3.4507 lr=3.64e-05 tok=19.535B dt=2816 wait=125 +05:08:37 step 148340 loss=3.3735 lr=3.64e-05 tok=19.538B dt=2879 wait=126 +05:09:20 pushed ckpt_w136.pt,log_w136.txt +05:10:36 waited 76s for global r=591 +05:10:44 global r=591 load 4.255 -> 4.262 +05:11:17 step 148360 loss=3.4668 lr=3.64e-05 tok=19.540B dt=3039 wait=126 +05:12:21 step 148380 loss=3.3422 lr=3.64e-05 tok=19.543B dt=3104 wait=126 +05:13:25 step 148400 loss=3.3213 lr=3.64e-05 tok=19.545B dt=3167 wait=127 +05:14:28 step 148420 loss=3.4358 lr=3.64e-05 tok=19.548B dt=3231 wait=127 +05:15:32 step 148440 loss=3.4059 lr=3.63e-05 tok=19.551B dt=3294 wait=127 +05:16:35 step 148460 loss=3.4348 lr=3.63e-05 tok=19.553B dt=3357 wait=128 +05:17:38 step 148480 loss=3.3778 lr=3.63e-05 tok=19.556B dt=3421 wait=128 +05:18:41 step 148500 loss=3.4413 lr=3.63e-05 tok=19.559B dt=3484 wait=129 +05:18:55 pushed ckpt_w136.pt,log_w136.txt +05:20:11 waited 76s for global r=592 +05:20:17 global r=592 load 4.280 -> 4.265 +05:21:22 step 148520 loss=3.4159 lr=3.63e-05 tok=19.561B dt=3644 wait=129 +05:22:25 step 148540 loss=3.5136 lr=3.63e-05 tok=19.564B dt=3707 wait=129 +05:23:29 step 148560 loss=3.3772 lr=3.62e-05 tok=19.566B dt=3771 wait=130 +05:24:32 step 148580 loss=3.3770 lr=3.62e-05 tok=19.569B dt=3834 wait=130 +05:25:35 step 148600 loss=3.4194 lr=3.62e-05 tok=19.572B dt=3898 wait=130 +05:26:38 step 148620 loss=3.4960 lr=3.62e-05 tok=19.574B dt=3961 wait=131 +05:27:41 step 148640 loss=3.3697 lr=3.62e-05 tok=19.577B dt=4024 wait=131 +05:28:25 pushed ckpt_w136.pt,log_w136.txt +05:29:25 waited 60s for global r=593 +05:29:34 global r=593 load 4.261 -> 4.271 +05:30:07 step 148660 loss=3.4354 lr=3.62e-05 tok=19.580B dt=4169 wait=131 +05:31:09 step 148680 loss=3.4032 lr=3.62e-05 tok=19.582B dt=4232 wait=132 +05:32:12 step 148700 loss=3.4092 lr=3.61e-05 tok=19.585B dt=4295 wait=132 +05:33:15 step 148720 loss=3.4280 lr=3.61e-05 tok=19.587B dt=4357 wait=132 +05:34:18 step 148740 loss=3.2673 lr=3.61e-05 tok=19.590B dt=4421 wait=133 +05:35:21 step 148760 loss=3.4290 lr=3.61e-05 tok=19.593B dt=4483 wait=133 +05:36:24 step 148780 loss=3.3757 lr=3.61e-05 tok=19.595B dt=4546 wait=133 +05:37:26 step 148800 loss=3.3473 lr=3.61e-05 tok=19.598B dt=4609 wait=134 +05:37:38 pushed ckpt_w136.pt,log_w136.txt +05:38:54 waited 76s for global r=594 +05:39:06 global r=594 load 4.250 -> 4.266 +05:40:10 step 148820 loss=3.3156 lr=3.60e-05 tok=19.601B dt=4772 wait=134 +05:41:12 step 148840 loss=3.4192 lr=3.60e-05 tok=19.603B dt=4835 wait=134 +05:42:15 step 148860 loss=3.3625 lr=3.60e-05 tok=19.606B dt=4898 wait=135 +05:43:19 step 148880 loss=3.3614 lr=3.60e-05 tok=19.608B dt=4961 wait=135 +05:44:21 step 148900 loss=3.3535 lr=3.60e-05 tok=19.611B dt=5023 wait=136 +05:45:24 step 148920 loss=3.3504 lr=3.60e-05 tok=19.614B dt=5086 wait=136 +05:46:26 step 148940 loss=3.2890 lr=3.60e-05 tok=19.616B dt=5148 wait=136 +05:47:10 pushed ckpt_w136.pt,log_w136.txt +05:48:26 waited 76s for global r=595 +05:48:33 global r=595 load 4.271 -> 4.275 +05:49:05 step 148960 loss=3.2428 lr=3.59e-05 tok=19.619B dt=5308 wait=137 +05:50:07 step 148980 loss=3.3115 lr=3.59e-05 tok=19.621B dt=5370 wait=137 +05:51:10 step 149000 loss=3.2679 lr=3.59e-05 tok=19.624B dt=5433 wait=137 +05:52:13 step 149020 loss=3.3981 lr=3.59e-05 tok=19.627B dt=5495 wait=138 +05:53:15 step 149040 loss=3.3571 lr=3.59e-05 tok=19.629B dt=5558 wait=138 +05:54:18 step 149060 loss=3.3869 lr=3.59e-05 tok=19.632B dt=5621 wait=138 +05:55:21 step 149080 loss=3.3038 lr=3.58e-05 tok=19.635B dt=5683 wait=139 +05:56:23 step 149100 loss=3.3746 lr=3.58e-05 tok=19.637B dt=5746 wait=139 +05:56:38 pushed ckpt_w136.pt,log_w136.txt +05:57:38 waited 60s for global r=596 +05:57:44 global r=596 load 4.268 -> 4.276 +05:58:48 step 149120 loss=3.3724 lr=3.58e-05 tok=19.640B dt=5890 wait=139 +05:59:50 step 149140 loss=3.4152 lr=3.58e-05 tok=19.642B dt=5953 wait=140 +06:00:53 step 149160 loss=3.3278 lr=3.58e-05 tok=19.645B dt=6015 wait=140 +06:01:55 step 149180 loss=3.3132 lr=3.58e-05 tok=19.648B dt=6078 wait=141 +06:02:58 step 149200 loss=3.3404 lr=3.58e-05 tok=19.650B dt=6141 wait=141 +06:04:01 step 149220 loss=3.4433 lr=3.57e-05 tok=19.653B dt=6204 wait=141 +06:05:04 step 149240 loss=3.2793 lr=3.57e-05 tok=19.656B dt=6266 wait=142 +06:05:49 pushed ckpt_w136.pt,log_w136.txt +06:07:05 waited 76s for global r=597 +06:07:11 global r=597 load 4.265 -> 4.263 +06:07:43 step 149260 loss=3.4910 lr=3.57e-05 tok=19.658B dt=6426 wait=142 +06:08:45 step 149280 loss=3.3076 lr=3.57e-05 tok=19.661B dt=6488 wait=142 +06:09:48 step 149300 loss=3.3482 lr=3.57e-05 tok=19.663B dt=6550 wait=143 +06:10:51 step 149320 loss=3.4219 lr=3.57e-05 tok=19.666B dt=6613 wait=143 +06:11:54 step 149340 loss=3.2498 lr=3.56e-05 tok=19.669B dt=6676 wait=143 +06:12:57 step 149360 loss=3.3434 lr=3.56e-05 tok=19.671B dt=6739 wait=144 +06:14:00 step 149380 loss=3.3252 lr=3.56e-05 tok=19.674B dt=6802 wait=144 +06:15:03 step 149400 loss=3.3069 lr=3.56e-05 tok=19.677B dt=6865 wait=144 +06:15:16 pushed ckpt_w136.pt,log_w136.txt +06:16:17 waited 61s for global r=598 +06:16:24 global r=598 load 4.265 -> 4.268 +06:17:28 step 149420 loss=3.3722 lr=3.56e-05 tok=19.679B dt=7010 wait=145 +06:18:30 step 149440 loss=3.3514 lr=3.56e-05 tok=19.682B dt=7073 wait=145 +06:19:33 step 149460 loss=3.3675 lr=3.56e-05 tok=19.684B dt=7135 wait=145 +06:20:36 step 149480 loss=3.3956 lr=3.55e-05 tok=19.687B dt=7198 wait=146 +06:21:39 step 149500 loss=3.2982 lr=3.55e-05 tok=19.690B dt=7261 wait=146 +06:22:41 step 149520 loss=3.3763 lr=3.55e-05 tok=19.692B dt=7324 wait=147 +06:23:44 step 149540 loss=3.3336 lr=3.55e-05 tok=19.695B dt=7386 wait=147 +06:24:29 pushed ckpt_w136.pt,log_w136.txt +06:25:45 waited 76s for global r=599 +06:25:51 global r=599 load 4.281 -> 4.277 +06:26:24 step 149560 loss=3.3558 lr=3.55e-05 tok=19.698B dt=7546 wait=147 +06:27:26 step 149580 loss=3.3192 lr=3.55e-05 tok=19.700B dt=7608 wait=148 +06:28:28 step 149600 loss=3.3318 lr=3.55e-05 tok=19.703B dt=7671 wait=148 +06:29:31 step 149620 loss=3.3561 lr=3.54e-05 tok=19.705B dt=7733 wait=148 +06:30:34 step 149640 loss=3.0209 lr=3.54e-05 tok=19.708B dt=7796 wait=149 +06:31:36 step 149660 loss=3.4097 lr=3.54e-05 tok=19.711B dt=7859 wait=149 +06:32:38 step 149680 loss=3.3776 lr=3.54e-05 tok=19.713B dt=7921 wait=149 +06:33:41 step 149700 loss=3.3628 lr=3.54e-05 tok=19.716B dt=7983 wait=150 +06:33:54 pushed ckpt_w136.pt,log_w136.txt +06:35:09 waited 76s for global r=600 +06:35:15 global r=600 load 4.278 -> 4.284 +06:36:19 step 149720 loss=3.2471 lr=3.54e-05 tok=19.718B dt=8141 wait=150 +06:37:22 step 149740 loss=3.3569 lr=3.53e-05 tok=19.721B dt=8205 wait=150 +06:38:25 step 149760 loss=3.3260 lr=3.53e-05 tok=19.724B dt=8267 wait=151 +06:39:28 step 149780 loss=3.3433 lr=3.53e-05 tok=19.726B dt=8330 wait=151 +06:40:30 step 149800 loss=3.2913 lr=3.53e-05 tok=19.729B dt=8392 wait=151 +06:41:32 step 149820 loss=3.4284 lr=3.53e-05 tok=19.732B dt=8455 wait=152 +06:42:35 step 149840 loss=3.2746 lr=3.53e-05 tok=19.734B dt=8518 wait=152 +06:43:18 pushed ckpt_w136.pt,log_w136.txt +06:44:34 waited 76s for global r=601 +06:44:40 global r=601 load 4.280 -> 4.285 +06:45:12 step 149860 loss=3.3851 lr=3.53e-05 tok=19.737B dt=8675 wait=152 +06:46:15 step 149880 loss=3.3342 lr=3.52e-05 tok=19.739B dt=8737 wait=153 +06:47:17 step 149900 loss=3.3626 lr=3.52e-05 tok=19.742B dt=8800 wait=153 +06:48:21 step 149920 loss=3.3578 lr=3.52e-05 tok=19.745B dt=8863 wait=153 +06:49:23 step 149940 loss=3.3362 lr=3.52e-05 tok=19.747B dt=8926 wait=154 +06:50:26 step 149960 loss=3.4229 lr=3.52e-05 tok=19.750B dt=8988 wait=154 +06:51:28 step 149980 loss=3.3746 lr=3.52e-05 tok=19.753B dt=9051 wait=155 +06:52:31 step 150000 loss=3.3219 lr=3.51e-05 tok=19.755B dt=9113 wait=155 +06:52:43 pushed ckpt_w136.pt,log_w136.txt +06:53:44 waited 61s for global r=602 +06:53:50 global r=602 load 4.267 -> 4.277 +06:54:55 step 150020 loss=2.8431 lr=3.51e-05 tok=19.758B dt=9257 wait=155 +06:55:57 step 150040 loss=3.4022 lr=3.51e-05 tok=19.760B dt=9319 wait=156 +06:57:00 step 150060 loss=3.2870 lr=3.51e-05 tok=19.763B dt=9383 wait=156 +06:58:03 step 150080 loss=3.3162 lr=3.51e-05 tok=19.766B dt=9446 wait=156 +06:59:06 step 150100 loss=3.3206 lr=3.51e-05 tok=19.768B dt=9508 wait=157 +07:00:09 step 150120 loss=3.3830 lr=3.51e-05 tok=19.771B dt=9571 wait=157 +07:01:12 step 150140 loss=3.2523 lr=3.50e-05 tok=19.774B dt=9634 wait=157 +07:01:58 pushed ckpt_w136.pt,log_w136.txt +07:02:59 waited 61s for global r=603 +07:03:04 global r=603 load 4.283 -> 4.279 +07:03:37 step 150160 loss=3.5063 lr=3.50e-05 tok=19.776B dt=9779 wait=158 +07:04:40 step 150180 loss=3.3681 lr=3.50e-05 tok=19.779B dt=9842 wait=158 +07:05:42 step 150200 loss=3.3410 lr=3.50e-05 tok=19.781B dt=9905 wait=158 +07:06:45 step 150220 loss=3.4829 lr=3.50e-05 tok=19.784B dt=9967 wait=159 +07:07:48 step 150240 loss=3.2864 lr=3.50e-05 tok=19.787B dt=10030 wait=159 +07:08:50 step 150260 loss=3.3824 lr=3.50e-05 tok=19.789B dt=10093 wait=159 +07:09:53 step 150280 loss=3.3797 lr=3.49e-05 tok=19.792B dt=10155 wait=160 +07:10:56 step 150300 loss=3.3768 lr=3.49e-05 tok=19.794B dt=10219 wait=160 diff --git a/log_w137.txt b/log_w137.txt new file mode 100644 index 0000000000000000000000000000000000000000..48985319a95d8607776d79ec74f2cea33e607a47 --- /dev/null +++ b/log_w137.txt @@ -0,0 +1,637 @@ +launch +04:18:43 name=w137 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 84/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +04:18:50 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +04:18:50 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +04:18:59 params 1.08B +04:19:01 resumed step=147600 seen=19.44B round=586 +04:19:01 stream salt=147600 +04:20:50 blocks compiled +04:20:50 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +04:23:37 step 147601 loss=2.9118 lr=3.70e-05 tok=19.441B dt=168 wait=123 +04:24:06 step 147602 loss=3.0832 lr=3.70e-05 tok=19.441B dt=196 wait=123 +04:25:07 step 147620 loss=3.4288 lr=3.70e-05 tok=19.443B dt=258 wait=123 +04:26:11 step 147640 loss=3.4282 lr=3.70e-05 tok=19.446B dt=321 wait=124 +04:27:19 step 147660 loss=3.4582 lr=3.69e-05 tok=19.448B dt=390 wait=124 +04:28:22 step 147680 loss=3.4928 lr=3.69e-05 tok=19.451B dt=453 wait=124 +04:29:26 step 147700 loss=3.5970 lr=3.69e-05 tok=19.454B dt=516 wait=125 +04:30:29 step 147720 loss=3.4367 lr=3.69e-05 tok=19.456B dt=580 wait=125 +04:31:33 step 147740 loss=3.4193 lr=3.69e-05 tok=19.459B dt=643 wait=125 +04:32:21 pushed ckpt_w137.pt,log_w137.txt +04:32:21 waited 0s for global r=587 +04:32:36 global r=587 load 4.272 -> 4.254 +04:33:10 step 147760 loss=3.5020 lr=3.69e-05 tok=19.462B dt=740 wait=126 +04:34:14 step 147780 loss=3.3001 lr=3.69e-05 tok=19.464B dt=805 wait=126 +04:35:17 step 147800 loss=3.3013 lr=3.68e-05 tok=19.467B dt=867 wait=127 +04:36:20 step 147820 loss=3.3885 lr=3.68e-05 tok=19.469B dt=931 wait=127 +04:37:23 step 147840 loss=3.3108 lr=3.68e-05 tok=19.472B dt=994 wait=127 +04:38:27 step 147860 loss=3.3990 lr=3.68e-05 tok=19.475B dt=1057 wait=127 +04:39:30 step 147880 loss=3.3498 lr=3.68e-05 tok=19.477B dt=1121 wait=128 +04:40:33 step 147900 loss=3.4073 lr=3.68e-05 tok=19.480B dt=1183 wait=128 +04:40:46 pushed ckpt_w137.pt,log_w137.txt +04:42:02 waited 76s for global r=588 +04:42:10 global r=588 load 4.265 -> 4.269 +04:43:16 step 147920 loss=3.3459 lr=3.67e-05 tok=19.483B dt=1346 wait=128 +04:44:19 step 147940 loss=3.4660 lr=3.67e-05 tok=19.485B dt=1409 wait=129 +04:45:22 step 147960 loss=3.4209 lr=3.67e-05 tok=19.488B dt=1472 wait=129 +04:46:25 step 147980 loss=3.3176 lr=3.67e-05 tok=19.490B dt=1536 wait=129 +04:47:28 step 148000 loss=3.3976 lr=3.67e-05 tok=19.493B dt=1599 wait=130 +04:48:32 step 148020 loss=3.4204 lr=3.67e-05 tok=19.496B dt=1662 wait=130 +04:49:35 step 148040 loss=3.3217 lr=3.67e-05 tok=19.498B dt=1725 wait=130 +04:50:19 pushed ckpt_w137.pt,log_w137.txt +04:51:35 waited 76s for global r=589 +04:51:46 global r=589 load 4.258 -> 4.269 +04:52:19 step 148060 loss=3.2270 lr=3.66e-05 tok=19.501B dt=1890 wait=131 +04:53:22 step 148080 loss=3.4319 lr=3.66e-05 tok=19.504B dt=1953 wait=131 +04:54:26 step 148100 loss=3.3420 lr=3.66e-05 tok=19.506B dt=2017 wait=131 +04:55:29 step 148120 loss=3.4823 lr=3.66e-05 tok=19.509B dt=2080 wait=132 +04:56:32 step 148140 loss=3.3681 lr=3.66e-05 tok=19.511B dt=2143 wait=132 +04:57:36 step 148160 loss=3.4987 lr=3.66e-05 tok=19.514B dt=2206 wait=132 +04:58:39 step 148180 loss=3.3730 lr=3.65e-05 tok=19.517B dt=2270 wait=133 +04:59:42 step 148200 loss=3.3953 lr=3.65e-05 tok=19.519B dt=2333 wait=133 +04:59:56 pushed ckpt_w137.pt,log_w137.txt +05:01:11 waited 76s for global r=590 +05:01:18 global r=590 load 4.277 -> 4.275 +05:02:22 step 148220 loss=3.3818 lr=3.65e-05 tok=19.522B dt=2493 wait=133 +05:03:26 step 148240 loss=3.4330 lr=3.65e-05 tok=19.524B dt=2556 wait=134 +05:04:29 step 148260 loss=3.4837 lr=3.65e-05 tok=19.527B dt=2619 wait=134 +05:05:32 step 148280 loss=3.4164 lr=3.65e-05 tok=19.530B dt=2683 wait=134 +05:06:35 step 148300 loss=3.4396 lr=3.64e-05 tok=19.532B dt=2746 wait=135 +05:07:39 step 148320 loss=3.4320 lr=3.64e-05 tok=19.535B dt=2809 wait=135 +05:08:43 step 148340 loss=3.4301 lr=3.64e-05 tok=19.538B dt=2873 wait=135 +05:09:29 pushed ckpt_w137.pt,log_w137.txt +05:10:45 waited 76s for global r=591 +05:10:52 global r=591 load 4.263 -> 4.262 +05:11:26 step 148360 loss=3.4666 lr=3.64e-05 tok=19.540B dt=3036 wait=135 +05:12:29 step 148380 loss=3.3402 lr=3.64e-05 tok=19.543B dt=3099 wait=136 +05:13:32 step 148400 loss=3.3369 lr=3.64e-05 tok=19.545B dt=3163 wait=136 +05:14:36 step 148420 loss=3.5276 lr=3.64e-05 tok=19.548B dt=3226 wait=136 +05:15:39 step 148440 loss=3.4071 lr=3.63e-05 tok=19.551B dt=3289 wait=137 +05:16:42 step 148460 loss=3.4752 lr=3.63e-05 tok=19.553B dt=3352 wait=137 +05:17:46 step 148480 loss=3.4388 lr=3.63e-05 tok=19.556B dt=3416 wait=137 +05:18:49 step 148500 loss=3.4361 lr=3.63e-05 tok=19.559B dt=3479 wait=138 +05:19:03 pushed ckpt_w137.pt,log_w137.txt +05:20:03 waited 61s for global r=592 +05:20:11 global r=592 load 4.256 -> 4.265 +05:21:15 step 148520 loss=3.4234 lr=3.63e-05 tok=19.561B dt=3625 wait=138 +05:22:18 step 148540 loss=3.5235 lr=3.63e-05 tok=19.564B dt=3688 wait=138 +05:23:21 step 148560 loss=3.4280 lr=3.62e-05 tok=19.566B dt=3752 wait=139 +05:24:24 step 148580 loss=3.3534 lr=3.62e-05 tok=19.569B dt=3815 wait=139 +05:25:28 step 148600 loss=3.4476 lr=3.62e-05 tok=19.572B dt=3878 wait=139 +05:26:31 step 148620 loss=3.5017 lr=3.62e-05 tok=19.574B dt=3942 wait=140 +05:27:34 step 148640 loss=3.3858 lr=3.62e-05 tok=19.577B dt=4005 wait=140 +05:28:20 pushed ckpt_w137.pt,log_w137.txt +05:29:35 waited 76s for global r=593 +05:29:44 global r=593 load 4.289 -> 4.271 +05:30:17 step 148660 loss=3.4695 lr=3.62e-05 tok=19.580B dt=4168 wait=140 +05:31:20 step 148680 loss=3.4395 lr=3.62e-05 tok=19.582B dt=4230 wait=141 +05:32:22 step 148700 loss=3.3737 lr=3.61e-05 tok=19.585B dt=4293 wait=141 +05:33:25 step 148720 loss=3.3403 lr=3.61e-05 tok=19.587B dt=4356 wait=141 +05:34:28 step 148740 loss=3.3171 lr=3.61e-05 tok=19.590B dt=4419 wait=142 +05:35:32 step 148760 loss=3.3639 lr=3.61e-05 tok=19.593B dt=4482 wait=142 +05:36:34 step 148780 loss=3.3545 lr=3.61e-05 tok=19.595B dt=4545 wait=142 +05:37:37 step 148800 loss=3.3669 lr=3.61e-05 tok=19.598B dt=4607 wait=143 +05:37:52 pushed ckpt_w137.pt,log_w137.txt +05:38:53 waited 61s for global r=594 +05:39:01 global r=594 load 4.275 -> 4.266 +05:40:06 step 148820 loss=3.2873 lr=3.60e-05 tok=19.601B dt=4756 wait=143 +05:41:08 step 148840 loss=3.3620 lr=3.60e-05 tok=19.603B dt=4819 wait=143 +05:42:11 step 148860 loss=3.3653 lr=3.60e-05 tok=19.606B dt=4881 wait=144 +05:43:14 step 148880 loss=3.2824 lr=3.60e-05 tok=19.608B dt=4944 wait=144 +05:44:16 step 148900 loss=3.3415 lr=3.60e-05 tok=19.611B dt=5007 wait=144 +05:45:19 step 148920 loss=3.4403 lr=3.60e-05 tok=19.614B dt=5069 wait=145 +05:46:21 step 148940 loss=3.3208 lr=3.60e-05 tok=19.616B dt=5132 wait=145 +05:47:04 pushed ckpt_w137.pt,log_w137.txt +05:48:20 waited 76s for global r=595 +05:48:26 global r=595 load 4.275 -> 4.275 +05:48:59 step 148960 loss=3.3649 lr=3.59e-05 tok=19.619B dt=5289 wait=145 +05:50:02 step 148980 loss=3.2455 lr=3.59e-05 tok=19.621B dt=5352 wait=146 +05:51:04 step 149000 loss=3.2973 lr=3.59e-05 tok=19.624B dt=5415 wait=146 +05:52:07 step 149020 loss=3.4279 lr=3.59e-05 tok=19.627B dt=5478 wait=146 +05:53:10 step 149040 loss=3.2864 lr=3.59e-05 tok=19.629B dt=5541 wait=147 +05:54:13 step 149060 loss=3.3672 lr=3.59e-05 tok=19.632B dt=5604 wait=147 +05:55:16 step 149080 loss=3.3086 lr=3.58e-05 tok=19.635B dt=5666 wait=147 +05:56:18 step 149100 loss=3.3275 lr=3.58e-05 tok=19.637B dt=5729 wait=148 +05:56:30 pushed ckpt_w137.pt,log_w137.txt +05:57:31 waited 61s for global r=596 +05:57:39 global r=596 load 4.283 -> 4.276 +05:58:43 step 149120 loss=3.3480 lr=3.58e-05 tok=19.640B dt=5874 wait=148 +05:59:46 step 149140 loss=3.4057 lr=3.58e-05 tok=19.642B dt=5937 wait=148 +06:00:49 step 149160 loss=3.3945 lr=3.58e-05 tok=19.645B dt=6000 wait=149 +06:01:52 step 149180 loss=3.3809 lr=3.58e-05 tok=19.648B dt=6062 wait=149 +06:02:54 step 149200 loss=3.3735 lr=3.58e-05 tok=19.650B dt=6125 wait=149 +06:03:57 step 149220 loss=3.4873 lr=3.57e-05 tok=19.653B dt=6187 wait=150 +06:05:00 step 149240 loss=3.3526 lr=3.57e-05 tok=19.656B dt=6250 wait=150 +06:05:44 pushed ckpt_w137.pt,log_w137.txt +06:06:59 waited 76s for global r=597 +06:07:05 global r=597 load 4.273 -> 4.263 +06:07:37 step 149260 loss=3.4693 lr=3.57e-05 tok=19.658B dt=6408 wait=150 +06:08:40 step 149280 loss=3.3274 lr=3.57e-05 tok=19.661B dt=6471 wait=151 +06:09:43 step 149300 loss=3.3904 lr=3.57e-05 tok=19.663B dt=6534 wait=151 +06:10:47 step 149320 loss=3.3382 lr=3.57e-05 tok=19.666B dt=6597 wait=151 +06:11:50 step 149340 loss=3.2451 lr=3.56e-05 tok=19.669B dt=6661 wait=152 +06:12:53 step 149360 loss=3.4108 lr=3.56e-05 tok=19.671B dt=6723 wait=152 +06:13:55 step 149380 loss=3.3280 lr=3.56e-05 tok=19.674B dt=6786 wait=152 +06:14:59 step 149400 loss=3.3493 lr=3.56e-05 tok=19.677B dt=6849 wait=153 +06:15:11 pushed ckpt_w137.pt,log_w137.txt +06:16:26 waited 76s for global r=598 +06:16:33 global r=598 load 4.283 -> 4.268 +06:17:38 step 149420 loss=3.2981 lr=3.56e-05 tok=19.679B dt=7008 wait=153 +06:18:40 step 149440 loss=3.3452 lr=3.56e-05 tok=19.682B dt=7071 wait=153 +06:19:43 step 149460 loss=3.3959 lr=3.56e-05 tok=19.684B dt=7133 wait=154 +06:20:45 step 149480 loss=3.3076 lr=3.55e-05 tok=19.687B dt=7196 wait=154 +06:21:48 step 149500 loss=3.3220 lr=3.55e-05 tok=19.690B dt=7259 wait=154 +06:22:51 step 149520 loss=3.4222 lr=3.55e-05 tok=19.692B dt=7321 wait=154 +06:23:53 step 149540 loss=3.3030 lr=3.55e-05 tok=19.695B dt=7384 wait=155 +06:24:40 pushed ckpt_w137.pt,log_w137.txt +06:25:55 waited 76s for global r=599 +06:26:00 global r=599 load 4.269 -> 4.277 +06:26:33 step 149560 loss=3.4090 lr=3.55e-05 tok=19.698B dt=7543 wait=155 +06:27:35 step 149580 loss=3.2849 lr=3.55e-05 tok=19.700B dt=7606 wait=155 +06:28:37 step 149600 loss=3.3282 lr=3.55e-05 tok=19.703B dt=7668 wait=156 +06:29:40 step 149620 loss=3.4296 lr=3.54e-05 tok=19.705B dt=7731 wait=156 +06:30:43 step 149640 loss=3.2960 lr=3.54e-05 tok=19.708B dt=7793 wait=156 +06:31:45 step 149660 loss=3.4411 lr=3.54e-05 tok=19.711B dt=7856 wait=157 +06:32:48 step 149680 loss=3.3702 lr=3.54e-05 tok=19.713B dt=7918 wait=157 +06:33:51 step 149700 loss=3.3549 lr=3.54e-05 tok=19.716B dt=7981 wait=157 +06:34:05 pushed ckpt_w137.pt,log_w137.txt +06:35:06 waited 61s for global r=600 +06:35:11 global r=600 load 4.299 -> 4.284 +06:36:15 step 149720 loss=3.3389 lr=3.54e-05 tok=19.718B dt=8126 wait=158 +06:37:18 step 149740 loss=3.3386 lr=3.53e-05 tok=19.721B dt=8189 wait=158 +06:38:21 step 149760 loss=3.3585 lr=3.53e-05 tok=19.724B dt=8252 wait=158 +06:39:24 step 149780 loss=3.3738 lr=3.53e-05 tok=19.726B dt=8315 wait=159 +06:40:27 step 149800 loss=3.3656 lr=3.53e-05 tok=19.729B dt=8377 wait=159 +06:41:29 step 149820 loss=3.4485 lr=3.53e-05 tok=19.732B dt=8440 wait=159 +06:42:32 step 149840 loss=3.2950 lr=3.53e-05 tok=19.734B dt=8502 wait=160 +06:43:16 pushed ckpt_w137.pt,log_w137.txt +06:44:32 waited 76s for global r=601 +06:44:37 global r=601 load 4.295 -> 4.285 +06:45:10 step 149860 loss=3.3430 lr=3.53e-05 tok=19.737B dt=8661 wait=160 +06:46:13 step 149880 loss=3.3659 lr=3.52e-05 tok=19.739B dt=8723 wait=160 +06:47:16 step 149900 loss=3.3360 lr=3.52e-05 tok=19.742B dt=8786 wait=161 +06:48:20 step 149920 loss=2.9298 lr=3.52e-05 tok=19.745B dt=8851 wait=161 +06:49:23 step 149940 loss=3.2233 lr=3.52e-05 tok=19.747B dt=8913 wait=161 +06:50:25 step 149960 loss=3.4601 lr=3.52e-05 tok=19.750B dt=8976 wait=162 +06:51:28 step 149980 loss=3.3730 lr=3.52e-05 tok=19.753B dt=9039 wait=162 +06:52:31 step 150000 loss=3.3763 lr=3.51e-05 tok=19.755B dt=9101 wait=162 +06:52:44 pushed ckpt_w137.pt,log_w137.txt +06:53:45 waited 60s for global r=602 +06:53:50 global r=602 load 4.269 -> 4.277 +06:54:54 step 150020 loss=3.2967 lr=3.51e-05 tok=19.758B dt=9245 wait=163 +06:55:56 step 150040 loss=3.4090 lr=3.51e-05 tok=19.760B dt=9307 wait=163 +06:56:59 step 150060 loss=3.3214 lr=3.51e-05 tok=19.763B dt=9369 wait=163 +06:58:01 step 150080 loss=3.3159 lr=3.51e-05 tok=19.766B dt=9432 wait=164 +06:59:04 step 150100 loss=3.3276 lr=3.51e-05 tok=19.768B dt=9494 wait=164 +07:00:07 step 150120 loss=3.3972 lr=3.51e-05 tok=19.771B dt=9558 wait=164 +07:01:10 step 150140 loss=3.2547 lr=3.50e-05 tok=19.774B dt=9621 wait=165 +07:01:54 pushed ckpt_w137.pt,log_w137.txt +07:03:09 waited 76s for global r=603 +07:03:17 global r=603 load 4.270 -> 4.279 +07:03:50 step 150160 loss=3.4419 lr=3.50e-05 tok=19.776B dt=9780 wait=165 +07:04:52 step 150180 loss=3.2865 lr=3.50e-05 tok=19.779B dt=9842 wait=165 +07:05:55 step 150200 loss=3.3319 lr=3.50e-05 tok=19.781B dt=9905 wait=165 +07:06:57 step 150220 loss=3.4916 lr=3.50e-05 tok=19.784B dt=9968 wait=166 +07:08:00 step 150240 loss=3.2770 lr=3.50e-05 tok=19.787B dt=10031 wait=166 +07:09:03 step 150260 loss=3.4176 lr=3.50e-05 tok=19.789B dt=10093 wait=166 +07:10:05 step 150280 loss=3.3780 lr=3.49e-05 tok=19.792B dt=10156 wait=167 +07:11:08 step 150300 loss=3.3727 lr=3.49e-05 tok=19.794B dt=10218 wait=167 +07:11:22 pushed ckpt_w137.pt,log_w137.txt +07:12:23 waited 62s for global r=604 +07:12:31 global r=604 load 4.289 -> 4.285 +07:13:35 step 150320 loss=3.3269 lr=3.49e-05 tok=19.797B dt=10366 wait=167 +07:14:38 step 150340 loss=3.3970 lr=3.49e-05 tok=19.800B dt=10428 wait=168 +07:15:41 step 150360 loss=3.3663 lr=3.49e-05 tok=19.802B dt=10491 wait=168 +07:16:43 step 150380 loss=3.3563 lr=3.49e-05 tok=19.805B dt=10554 wait=168 +07:17:46 step 150400 loss=3.3225 lr=3.48e-05 tok=19.808B dt=10617 wait=169 +07:18:49 step 150420 loss=3.4080 lr=3.48e-05 tok=19.810B dt=10680 wait=169 +07:19:52 step 150440 loss=3.2931 lr=3.48e-05 tok=19.813B dt=10743 wait=169 +07:20:37 pushed ckpt_w137.pt,log_w137.txt +07:24:11 step 150460 loss=3.3837 lr=3.48e-05 tok=19.815B dt=11001 wait=170 +07:25:14 step 150480 loss=3.3594 lr=3.48e-05 tok=19.818B dt=11064 wait=170 +07:26:16 step 150500 loss=3.2820 lr=3.48e-05 tok=19.821B dt=11126 wait=170 +07:27:18 step 150520 loss=3.2224 lr=3.48e-05 tok=19.823B dt=11189 wait=171 +07:28:21 step 150540 loss=3.3612 lr=3.47e-05 tok=19.826B dt=11251 wait=171 +07:29:24 step 150560 loss=3.2970 lr=3.47e-05 tok=19.829B dt=11314 wait=171 +07:30:27 step 150580 loss=3.1908 lr=3.47e-05 tok=19.831B dt=11377 wait=171 +07:31:30 step 150600 loss=3.5197 lr=3.47e-05 tok=19.834B dt=11441 wait=172 +07:31:44 pushed ckpt_w137.pt,log_w137.txt +07:35:49 step 150620 loss=3.3015 lr=3.47e-05 tok=19.836B dt=11700 wait=172 +07:36:52 step 150640 loss=3.4342 lr=3.47e-05 tok=19.839B dt=11762 wait=172 +07:37:54 step 150660 loss=3.3632 lr=3.47e-05 tok=19.842B dt=11825 wait=173 +07:38:56 step 150680 loss=3.4004 lr=3.46e-05 tok=19.844B dt=11887 wait=173 +07:39:59 step 150700 loss=3.3636 lr=3.46e-05 tok=19.847B dt=11950 wait=173 +07:41:02 step 150720 loss=3.4015 lr=3.46e-05 tok=19.850B dt=12013 wait=174 +07:42:05 step 150740 loss=3.3338 lr=3.46e-05 tok=19.852B dt=12075 wait=174 +07:42:51 pushed ckpt_w137.pt,log_w137.txt +07:46:26 step 150760 loss=3.3527 lr=3.46e-05 tok=19.855B dt=12336 wait=174 +07:47:28 step 150780 loss=3.3686 lr=3.46e-05 tok=19.857B dt=12399 wait=175 +07:48:31 step 150800 loss=3.3799 lr=3.45e-05 tok=19.860B dt=12462 wait=175 +07:49:33 step 150820 loss=3.3178 lr=3.45e-05 tok=19.863B dt=12524 wait=175 +07:50:36 step 150840 loss=3.3590 lr=3.45e-05 tok=19.865B dt=12587 wait=176 +07:51:39 step 150860 loss=3.4565 lr=3.45e-05 tok=19.868B dt=12650 wait=176 +07:52:42 step 150880 loss=3.3577 lr=3.45e-05 tok=19.871B dt=12713 wait=176 +07:53:45 step 150900 loss=3.3295 lr=3.45e-05 tok=19.873B dt=12776 wait=177 +07:54:04 pushed ckpt_w137.pt,log_w137.txt +07:58:10 step 150920 loss=3.3090 lr=3.45e-05 tok=19.876B dt=13040 wait=177 +07:59:13 step 150940 loss=3.3818 lr=3.44e-05 tok=19.878B dt=13103 wait=177 +08:00:15 step 150960 loss=3.3655 lr=3.44e-05 tok=19.881B dt=13166 wait=178 +08:01:18 step 150980 loss=3.3464 lr=3.44e-05 tok=19.884B dt=13228 wait=178 +08:02:21 step 151000 loss=3.3764 lr=3.44e-05 tok=19.886B dt=13291 wait=178 +08:03:23 step 151020 loss=3.2957 lr=3.44e-05 tok=19.889B dt=13353 wait=178 +08:04:25 step 151040 loss=3.3264 lr=3.44e-05 tok=19.891B dt=13416 wait=179 +08:05:11 pushed ckpt_w137.pt,log_w137.txt +08:08:45 step 151060 loss=3.3374 lr=3.44e-05 tok=19.894B dt=13676 wait=179 +08:09:48 step 151080 loss=3.3269 lr=3.43e-05 tok=19.897B dt=13738 wait=179 +08:10:50 step 151100 loss=3.4428 lr=3.43e-05 tok=19.899B dt=13801 wait=180 +08:11:53 step 151120 loss=3.3801 lr=3.43e-05 tok=19.902B dt=13863 wait=180 +08:12:55 step 151140 loss=3.3060 lr=3.43e-05 tok=19.905B dt=13926 wait=180 +08:13:58 step 151160 loss=3.3802 lr=3.43e-05 tok=19.907B dt=13988 wait=181 +08:15:00 step 151180 loss=3.4275 lr=3.43e-05 tok=19.910B dt=14051 wait=181 +08:16:03 step 151200 loss=3.3234 lr=3.42e-05 tok=19.912B dt=14113 wait=181 +08:16:15 pushed ckpt_w137.pt,log_w137.txt +08:20:21 step 151220 loss=3.3951 lr=3.42e-05 tok=19.915B dt=14371 wait=182 +08:21:23 step 151240 loss=3.3310 lr=3.42e-05 tok=19.918B dt=14434 wait=182 +08:22:26 step 151260 loss=3.2538 lr=3.42e-05 tok=19.920B dt=14496 wait=182 +08:23:28 step 151280 loss=3.3042 lr=3.42e-05 tok=19.923B dt=14559 wait=182 +08:24:31 step 151300 loss=3.2794 lr=3.42e-05 tok=19.926B dt=14621 wait=183 +08:25:34 step 151320 loss=3.3828 lr=3.42e-05 tok=19.928B dt=14684 wait=183 +08:26:36 step 151340 loss=3.3248 lr=3.41e-05 tok=19.931B dt=14747 wait=183 +08:27:19 pushed ckpt_w137.pt,log_w137.txt +08:30:52 step 151360 loss=3.4648 lr=3.41e-05 tok=19.933B dt=15003 wait=184 +08:31:55 step 151380 loss=3.3707 lr=3.41e-05 tok=19.936B dt=15066 wait=184 +08:32:58 step 151400 loss=3.3698 lr=3.41e-05 tok=19.939B dt=15128 wait=184 +08:34:00 step 151420 loss=3.3860 lr=3.41e-05 tok=19.941B dt=15190 wait=185 +08:35:02 step 151440 loss=3.3961 lr=3.41e-05 tok=19.944B dt=15252 wait=185 +08:36:05 step 151460 loss=3.2912 lr=3.41e-05 tok=19.947B dt=15315 wait=185 +08:37:08 step 151480 loss=3.3141 lr=3.40e-05 tok=19.949B dt=15379 wait=186 +08:38:11 step 151500 loss=3.3528 lr=3.40e-05 tok=19.952B dt=15442 wait=186 +08:38:23 pushed ckpt_w137.pt,log_w137.txt +08:42:29 step 151520 loss=3.4087 lr=3.40e-05 tok=19.954B dt=15700 wait=186 +08:43:32 step 151540 loss=3.4358 lr=3.40e-05 tok=19.957B dt=15762 wait=186 +08:44:34 step 151560 loss=3.4493 lr=3.40e-05 tok=19.960B dt=15824 wait=187 +08:45:36 step 151580 loss=3.4356 lr=3.40e-05 tok=19.962B dt=15887 wait=187 +08:46:39 step 151600 loss=3.4406 lr=3.40e-05 tok=19.965B dt=15950 wait=187 +08:47:42 step 151620 loss=3.4028 lr=3.39e-05 tok=19.968B dt=16013 wait=188 +08:48:45 step 151640 loss=3.2913 lr=3.39e-05 tok=19.970B dt=16075 wait=188 +08:49:28 pushed ckpt_w137.pt,log_w137.txt +08:53:03 step 151660 loss=3.4431 lr=3.39e-05 tok=19.973B dt=16333 wait=188 +08:54:05 step 151680 loss=3.2921 lr=3.39e-05 tok=19.975B dt=16396 wait=189 +08:55:08 step 151700 loss=3.3159 lr=3.39e-05 tok=19.978B dt=16459 wait=189 +08:56:11 step 151720 loss=3.3644 lr=3.39e-05 tok=19.981B dt=16521 wait=189 +08:57:13 step 151740 loss=3.3916 lr=3.39e-05 tok=19.983B dt=16584 wait=190 +08:58:16 step 151760 loss=3.2634 lr=3.38e-05 tok=19.986B dt=16647 wait=190 +08:59:19 step 151780 loss=3.3917 lr=3.38e-05 tok=19.988B dt=16710 wait=190 +09:00:22 step 151800 loss=3.2047 lr=3.38e-05 tok=19.991B dt=16773 wait=191 +09:00:36 pushed ckpt_w137.pt,log_w137.txt +09:04:42 step 151820 loss=3.3328 lr=3.38e-05 tok=19.994B dt=17032 wait=191 +09:05:44 step 151840 loss=3.3049 lr=3.38e-05 tok=19.996B dt=17095 wait=191 +09:06:47 step 151860 loss=3.3364 lr=3.38e-05 tok=19.999B dt=17158 wait=192 +09:07:50 step 151880 loss=3.2902 lr=3.37e-05 tok=20.002B dt=17220 wait=192 +09:08:52 step 151900 loss=3.3491 lr=3.37e-05 tok=20.004B dt=17283 wait=192 +09:09:55 step 151920 loss=3.3897 lr=3.37e-05 tok=20.007B dt=17345 wait=193 +09:10:57 step 151940 loss=3.3398 lr=3.37e-05 tok=20.009B dt=17408 wait=193 +09:11:41 pushed ckpt_w137.pt,log_w137.txt +09:15:15 step 151960 loss=3.3225 lr=3.37e-05 tok=20.012B dt=17666 wait=193 +09:16:18 step 151980 loss=3.3534 lr=3.37e-05 tok=20.015B dt=17728 wait=193 +09:17:20 step 152000 loss=3.3350 lr=3.37e-05 tok=20.017B dt=17791 wait=194 +09:18:23 step 152020 loss=3.3725 lr=3.36e-05 tok=20.020B dt=17854 wait=194 +09:19:26 step 152040 loss=3.3018 lr=3.36e-05 tok=20.023B dt=17916 wait=194 +09:20:29 step 152060 loss=3.3696 lr=3.36e-05 tok=20.025B dt=17979 wait=195 +09:21:31 step 152080 loss=3.3408 lr=3.36e-05 tok=20.028B dt=18041 wait=195 +09:22:33 step 152100 loss=3.3921 lr=3.36e-05 tok=20.030B dt=18104 wait=195 +09:22:45 pushed ckpt_w137.pt,log_w137.txt +09:26:51 step 152120 loss=3.2685 lr=3.36e-05 tok=20.033B dt=18361 wait=196 +09:27:53 step 152140 loss=3.2765 lr=3.36e-05 tok=20.036B dt=18424 wait=196 +09:28:56 step 152160 loss=3.3763 lr=3.35e-05 tok=20.038B dt=18486 wait=196 +09:29:59 step 152180 loss=3.3350 lr=3.35e-05 tok=20.041B dt=18549 wait=197 +09:31:01 step 152200 loss=3.4200 lr=3.35e-05 tok=20.044B dt=18611 wait=197 +09:32:04 step 152220 loss=3.3848 lr=3.35e-05 tok=20.046B dt=18674 wait=197 +09:33:06 step 152240 loss=3.3071 lr=3.35e-05 tok=20.049B dt=18737 wait=197 +09:33:49 pushed ckpt_w137.pt,log_w137.txt +09:37:24 step 152260 loss=3.4070 lr=3.35e-05 tok=20.051B dt=18995 wait=198 +09:38:27 step 152280 loss=3.3468 lr=3.35e-05 tok=20.054B dt=19057 wait=198 +09:39:29 step 152300 loss=3.3301 lr=3.34e-05 tok=20.057B dt=19119 wait=198 +09:40:32 step 152320 loss=3.4036 lr=3.34e-05 tok=20.059B dt=19182 wait=199 +09:41:34 step 152340 loss=3.3380 lr=3.34e-05 tok=20.062B dt=19245 wait=199 +09:42:36 step 152360 loss=3.2731 lr=3.34e-05 tok=20.065B dt=19307 wait=199 +09:43:39 step 152380 loss=3.3514 lr=3.34e-05 tok=20.067B dt=19369 wait=200 +09:44:41 step 152400 loss=3.3901 lr=3.34e-05 tok=20.070B dt=19431 wait=200 +09:44:55 pushed ckpt_w137.pt,log_w137.txt +09:49:01 step 152420 loss=3.3484 lr=3.34e-05 tok=20.072B dt=19691 wait=200 +09:50:04 step 152440 loss=3.3383 lr=3.33e-05 tok=20.075B dt=19754 wait=200 +09:51:06 step 152460 loss=3.3864 lr=3.33e-05 tok=20.078B dt=19816 wait=201 +09:52:08 step 152480 loss=3.4171 lr=3.33e-05 tok=20.080B dt=19878 wait=201 +09:53:10 step 152500 loss=3.4176 lr=3.33e-05 tok=20.083B dt=19941 wait=201 +09:54:13 step 152520 loss=3.4628 lr=3.33e-05 tok=20.085B dt=20003 wait=202 +09:55:16 step 152540 loss=3.3829 lr=3.33e-05 tok=20.088B dt=20066 wait=202 +09:56:01 pushed ckpt_w137.pt,log_w137.txt +09:59:36 step 152560 loss=3.3202 lr=3.33e-05 tok=20.091B dt=20327 wait=202 +10:00:39 step 152580 loss=3.4305 lr=3.32e-05 tok=20.093B dt=20389 wait=203 +10:01:41 step 152600 loss=3.4148 lr=3.32e-05 tok=20.096B dt=20452 wait=203 +10:02:44 step 152620 loss=3.4887 lr=3.32e-05 tok=20.099B dt=20515 wait=203 +10:03:47 step 152640 loss=3.4280 lr=3.32e-05 tok=20.101B dt=20577 wait=203 +10:04:50 step 152660 loss=3.3073 lr=3.32e-05 tok=20.104B dt=20640 wait=204 +10:05:52 step 152680 loss=3.4285 lr=3.32e-05 tok=20.106B dt=20703 wait=204 +10:06:55 step 152700 loss=3.3792 lr=3.32e-05 tok=20.109B dt=20765 wait=204 +10:07:08 pushed ckpt_w137.pt,log_w137.txt +10:11:14 step 152720 loss=3.2856 lr=3.31e-05 tok=20.112B dt=21025 wait=205 +10:12:17 step 152740 loss=3.2711 lr=3.31e-05 tok=20.114B dt=21087 wait=205 +10:13:19 step 152760 loss=3.3570 lr=3.31e-05 tok=20.117B dt=21150 wait=205 +10:14:23 step 152780 loss=3.3444 lr=3.31e-05 tok=20.120B dt=21213 wait=206 +10:15:25 step 152800 loss=3.4418 lr=3.31e-05 tok=20.122B dt=21276 wait=206 +10:16:28 step 152820 loss=3.4623 lr=3.31e-05 tok=20.125B dt=21339 wait=206 +10:17:31 step 152840 loss=3.3612 lr=3.30e-05 tok=20.127B dt=21401 wait=207 +10:18:15 pushed ckpt_w137.pt,log_w137.txt +10:21:51 step 152860 loss=3.3978 lr=3.30e-05 tok=20.130B dt=21661 wait=207 +10:22:54 step 152880 loss=3.3271 lr=3.30e-05 tok=20.133B dt=21724 wait=207 +10:23:56 step 152900 loss=3.4084 lr=3.30e-05 tok=20.135B dt=21786 wait=208 +10:24:59 step 152920 loss=3.3526 lr=3.30e-05 tok=20.138B dt=21849 wait=208 +10:26:01 step 152940 loss=3.3568 lr=3.30e-05 tok=20.141B dt=21912 wait=208 +10:27:04 step 152960 loss=3.4226 lr=3.30e-05 tok=20.143B dt=21975 wait=209 +10:28:07 step 152980 loss=3.3563 lr=3.29e-05 tok=20.146B dt=22037 wait=209 +10:29:09 step 153000 loss=3.4429 lr=3.29e-05 tok=20.148B dt=22100 wait=209 +10:29:22 pushed ckpt_w137.pt,log_w137.txt +10:33:29 step 153020 loss=3.3642 lr=3.29e-05 tok=20.151B dt=22359 wait=210 +10:34:31 step 153040 loss=3.3494 lr=3.29e-05 tok=20.154B dt=22421 wait=210 +10:35:33 step 153060 loss=3.3716 lr=3.29e-05 tok=20.156B dt=22484 wait=210 +10:36:36 step 153080 loss=3.3300 lr=3.29e-05 tok=20.159B dt=22547 wait=211 +10:37:39 step 153100 loss=3.3944 lr=3.29e-05 tok=20.161B dt=22609 wait=211 +10:38:41 step 153120 loss=3.2886 lr=3.28e-05 tok=20.164B dt=22671 wait=211 +10:39:44 step 153140 loss=3.4637 lr=3.28e-05 tok=20.167B dt=22734 wait=211 +10:40:31 pushed ckpt_w137.pt,log_w137.txt +10:44:06 step 153160 loss=3.3686 lr=3.28e-05 tok=20.169B dt=22997 wait=212 +10:45:09 step 153180 loss=3.2624 lr=3.28e-05 tok=20.172B dt=23060 wait=212 +10:46:12 step 153200 loss=3.4378 lr=3.28e-05 tok=20.175B dt=23122 wait=212 +10:47:14 step 153220 loss=3.3107 lr=3.28e-05 tok=20.177B dt=23185 wait=213 +10:48:17 step 153240 loss=3.4082 lr=3.28e-05 tok=20.180B dt=23248 wait=213 +10:49:20 step 153260 loss=3.5001 lr=3.27e-05 tok=20.182B dt=23311 wait=213 +10:50:23 step 153280 loss=3.4226 lr=3.27e-05 tok=20.185B dt=23374 wait=214 +10:51:27 step 153300 loss=3.4160 lr=3.27e-05 tok=20.188B dt=23437 wait=214 +10:51:38 pushed ckpt_w137.pt,log_w137.txt +10:55:45 step 153320 loss=3.3394 lr=3.27e-05 tok=20.190B dt=23696 wait=214 +10:56:47 step 153340 loss=3.4045 lr=3.27e-05 tok=20.193B dt=23758 wait=215 +10:57:50 step 153360 loss=3.4095 lr=3.27e-05 tok=20.196B dt=23820 wait=215 +10:58:53 step 153380 loss=3.3402 lr=3.27e-05 tok=20.198B dt=23884 wait=215 +10:59:56 step 153400 loss=3.4759 lr=3.26e-05 tok=20.201B dt=23947 wait=215 +11:00:59 step 153420 loss=3.3871 lr=3.26e-05 tok=20.203B dt=24010 wait=216 +11:02:02 step 153440 loss=3.4083 lr=3.26e-05 tok=20.206B dt=24073 wait=216 +11:02:47 pushed ckpt_w137.pt,log_w137.txt +11:06:22 step 153460 loss=3.3863 lr=3.26e-05 tok=20.209B dt=24333 wait=216 +11:07:25 step 153480 loss=3.2838 lr=3.26e-05 tok=20.211B dt=24395 wait=217 +11:08:27 step 153500 loss=3.4012 lr=3.26e-05 tok=20.214B dt=24457 wait=217 +11:09:29 step 153520 loss=3.3146 lr=3.26e-05 tok=20.217B dt=24520 wait=217 +11:10:32 step 153540 loss=3.3877 lr=3.25e-05 tok=20.219B dt=24582 wait=218 +11:11:35 step 153560 loss=3.2727 lr=3.25e-05 tok=20.222B dt=24645 wait=218 +11:12:37 step 153580 loss=3.2623 lr=3.25e-05 tok=20.224B dt=24708 wait=218 +11:13:40 step 153600 loss=3.3057 lr=3.25e-05 tok=20.227B dt=24770 wait=219 +11:13:53 pushed ckpt_w137.pt,log_w137.txt +11:18:00 step 153620 loss=3.3766 lr=3.25e-05 tok=20.230B dt=25030 wait=219 +11:19:03 step 153640 loss=3.4275 lr=3.25e-05 tok=20.232B dt=25093 wait=219 +11:20:05 step 153660 loss=3.3332 lr=3.25e-05 tok=20.235B dt=25156 wait=220 +11:21:08 step 153680 loss=3.4101 lr=3.24e-05 tok=20.238B dt=25218 wait=220 +11:22:11 step 153700 loss=3.3998 lr=3.24e-05 tok=20.240B dt=25282 wait=220 +11:23:13 step 153720 loss=3.4100 lr=3.24e-05 tok=20.243B dt=25344 wait=221 +11:24:16 step 153740 loss=3.4128 lr=3.24e-05 tok=20.245B dt=25406 wait=221 +11:25:01 pushed ckpt_w137.pt,log_w137.txt +11:28:38 step 153760 loss=3.2668 lr=3.24e-05 tok=20.248B dt=25668 wait=221 +11:29:40 step 153780 loss=3.3373 lr=3.24e-05 tok=20.251B dt=25731 wait=222 +11:30:42 step 153800 loss=3.3879 lr=3.24e-05 tok=20.253B dt=25793 wait=222 +11:31:45 step 153820 loss=3.3478 lr=3.23e-05 tok=20.256B dt=25855 wait=222 +11:32:48 step 153840 loss=3.4361 lr=3.23e-05 tok=20.258B dt=25918 wait=223 +11:33:50 step 153860 loss=3.3773 lr=3.23e-05 tok=20.261B dt=25981 wait=223 +11:34:53 step 153880 loss=3.2763 lr=3.23e-05 tok=20.264B dt=26043 wait=223 +11:35:55 step 153900 loss=3.1068 lr=3.23e-05 tok=20.266B dt=26106 wait=223 +11:36:07 pushed ckpt_w137.pt,log_w137.txt +11:40:14 step 153920 loss=3.2009 lr=3.23e-05 tok=20.269B dt=26364 wait=224 +11:41:16 step 153940 loss=3.4506 lr=3.23e-05 tok=20.272B dt=26427 wait=224 +11:42:19 step 153960 loss=3.3949 lr=3.22e-05 tok=20.274B dt=26489 wait=224 +11:43:21 step 153980 loss=3.4003 lr=3.22e-05 tok=20.277B dt=26552 wait=225 +11:44:24 step 154000 loss=3.3590 lr=3.22e-05 tok=20.279B dt=26614 wait=225 +11:45:26 step 154020 loss=3.2927 lr=3.22e-05 tok=20.282B dt=26677 wait=225 +11:46:31 step 154040 loss=3.2685 lr=3.22e-05 tok=20.285B dt=26741 wait=226 +11:47:14 pushed ckpt_w137.pt,log_w137.txt +11:50:49 step 154060 loss=3.3428 lr=3.22e-05 tok=20.287B dt=26999 wait=226 +11:51:51 step 154080 loss=3.3494 lr=3.22e-05 tok=20.290B dt=27062 wait=226 +11:52:54 step 154100 loss=3.3225 lr=3.21e-05 tok=20.293B dt=27124 wait=226 +11:53:57 step 154120 loss=3.3872 lr=3.21e-05 tok=20.295B dt=27187 wait=227 +11:54:59 step 154140 loss=3.3331 lr=3.21e-05 tok=20.298B dt=27250 wait=227 +11:56:02 step 154160 loss=3.3823 lr=3.21e-05 tok=20.300B dt=27312 wait=227 +11:57:05 step 154180 loss=3.3256 lr=3.21e-05 tok=20.303B dt=27375 wait=228 +11:58:07 step 154200 loss=3.3219 lr=3.21e-05 tok=20.306B dt=27437 wait=228 +11:58:19 pushed ckpt_w137.pt,log_w137.txt +12:02:25 step 154220 loss=3.4020 lr=3.21e-05 tok=20.308B dt=27696 wait=228 +12:03:27 step 154240 loss=3.2213 lr=3.21e-05 tok=20.311B dt=27758 wait=229 +12:04:31 step 154260 loss=3.3373 lr=3.20e-05 tok=20.314B dt=27821 wait=229 +12:05:33 step 154280 loss=3.3345 lr=3.20e-05 tok=20.316B dt=27883 wait=229 +12:06:36 step 154300 loss=3.2747 lr=3.20e-05 tok=20.319B dt=27946 wait=230 +12:07:38 step 154320 loss=3.2674 lr=3.20e-05 tok=20.321B dt=28009 wait=230 +12:08:41 step 154340 loss=3.3876 lr=3.20e-05 tok=20.324B dt=28071 wait=230 +12:09:24 pushed ckpt_w137.pt,log_w137.txt +12:12:59 step 154360 loss=3.3831 lr=3.20e-05 tok=20.327B dt=28329 wait=230 +12:14:01 step 154380 loss=3.3736 lr=3.20e-05 tok=20.329B dt=28392 wait=231 +12:15:04 step 154400 loss=3.2561 lr=3.19e-05 tok=20.332B dt=28454 wait=231 +12:16:07 step 154420 loss=3.4635 lr=3.19e-05 tok=20.335B dt=28517 wait=231 +12:17:09 step 154440 loss=3.4006 lr=3.19e-05 tok=20.337B dt=28580 wait=232 +12:18:12 step 154460 loss=3.4127 lr=3.19e-05 tok=20.340B dt=28642 wait=232 +12:19:15 step 154480 loss=3.2930 lr=3.19e-05 tok=20.342B dt=28705 wait=232 +12:20:18 step 154500 loss=3.3078 lr=3.19e-05 tok=20.345B dt=28768 wait=233 +12:20:32 pushed ckpt_w137.pt,log_w137.txt +12:24:39 step 154520 loss=3.4033 lr=3.19e-05 tok=20.348B dt=29029 wait=233 +12:25:42 step 154540 loss=3.2853 lr=3.18e-05 tok=20.350B dt=29092 wait=233 +12:26:44 step 154560 loss=3.3285 lr=3.18e-05 tok=20.353B dt=29155 wait=233 +12:27:47 step 154580 loss=3.3748 lr=3.18e-05 tok=20.355B dt=29218 wait=234 +12:28:50 step 154600 loss=3.4141 lr=3.18e-05 tok=20.358B dt=29280 wait=234 +12:29:52 step 154620 loss=3.3567 lr=3.18e-05 tok=20.361B dt=29342 wait=234 +12:30:54 step 154640 loss=3.3288 lr=3.18e-05 tok=20.363B dt=29405 wait=235 +12:31:38 pushed ckpt_w137.pt,log_w137.txt +12:35:13 step 154660 loss=3.2777 lr=3.18e-05 tok=20.366B dt=29664 wait=235 +12:36:16 step 154680 loss=3.4051 lr=3.17e-05 tok=20.369B dt=29727 wait=235 +12:37:19 step 154700 loss=3.3533 lr=3.17e-05 tok=20.371B dt=29789 wait=236 +12:38:22 step 154720 loss=3.3957 lr=3.17e-05 tok=20.374B dt=29852 wait=236 +12:39:25 step 154740 loss=3.3019 lr=3.17e-05 tok=20.376B dt=29915 wait=236 +12:40:27 step 154760 loss=3.4284 lr=3.17e-05 tok=20.379B dt=29978 wait=237 +12:41:30 step 154780 loss=3.1514 lr=3.17e-05 tok=20.382B dt=30041 wait=237 +12:42:33 step 154800 loss=3.1732 lr=3.17e-05 tok=20.384B dt=30104 wait=237 +12:42:46 pushed ckpt_w137.pt,log_w137.txt +12:46:53 step 154820 loss=3.1377 lr=3.16e-05 tok=20.387B dt=30364 wait=238 +12:47:55 step 154840 loss=3.3226 lr=3.16e-05 tok=20.390B dt=30426 wait=238 +12:48:58 step 154860 loss=3.1869 lr=3.16e-05 tok=20.392B dt=30489 wait=238 +12:50:01 step 154880 loss=3.3375 lr=3.16e-05 tok=20.395B dt=30552 wait=239 +12:51:03 step 154900 loss=3.4298 lr=3.16e-05 tok=20.397B dt=30614 wait=239 +12:52:05 step 154920 loss=3.3843 lr=3.16e-05 tok=20.400B dt=30676 wait=239 +12:52:24 mix source 1 lap 1 +12:53:08 step 154940 loss=3.3304 lr=3.16e-05 tok=20.403B dt=30739 wait=240 +12:53:53 pushed ckpt_w137.pt,log_w137.txt +12:57:29 step 154960 loss=3.2169 lr=3.15e-05 tok=20.405B dt=30999 wait=240 +12:58:31 step 154980 loss=3.2088 lr=3.15e-05 tok=20.408B dt=31062 wait=240 +12:59:33 step 155000 loss=3.4758 lr=3.15e-05 tok=20.411B dt=31124 wait=240 +13:00:36 step 155020 loss=3.3695 lr=3.15e-05 tok=20.413B dt=31186 wait=241 +13:01:38 step 155040 loss=3.3900 lr=3.15e-05 tok=20.416B dt=31249 wait=241 +13:02:40 step 155060 loss=3.3177 lr=3.15e-05 tok=20.418B dt=31311 wait=241 +13:03:43 step 155080 loss=3.3144 lr=3.15e-05 tok=20.421B dt=31373 wait=242 +13:04:45 step 155100 loss=3.3287 lr=3.14e-05 tok=20.424B dt=31435 wait=242 +13:04:59 pushed ckpt_w137.pt,log_w137.txt +13:09:06 step 155120 loss=3.3074 lr=3.14e-05 tok=20.426B dt=31696 wait=242 +13:10:08 step 155140 loss=3.3119 lr=3.14e-05 tok=20.429B dt=31759 wait=243 +13:11:11 step 155160 loss=3.2826 lr=3.14e-05 tok=20.432B dt=31822 wait=243 +13:12:15 step 155180 loss=3.2924 lr=3.14e-05 tok=20.434B dt=31885 wait=243 +13:13:17 step 155200 loss=3.5096 lr=3.14e-05 tok=20.437B dt=31947 wait=244 +13:14:20 step 155220 loss=3.3791 lr=3.14e-05 tok=20.439B dt=32011 wait=244 +13:15:23 step 155240 loss=3.3165 lr=3.14e-05 tok=20.442B dt=32073 wait=244 +13:16:06 pushed ckpt_w137.pt,log_w137.txt +13:19:42 step 155260 loss=3.2910 lr=3.13e-05 tok=20.445B dt=32332 wait=245 +13:20:45 step 155280 loss=3.3125 lr=3.13e-05 tok=20.447B dt=32395 wait=245 +13:21:48 step 155300 loss=3.2881 lr=3.13e-05 tok=20.450B dt=32458 wait=245 +13:22:50 step 155320 loss=3.4437 lr=3.13e-05 tok=20.452B dt=32521 wait=245 +13:23:53 step 155340 loss=3.3618 lr=3.13e-05 tok=20.455B dt=32583 wait=246 +13:24:55 step 155360 loss=3.3590 lr=3.13e-05 tok=20.458B dt=32646 wait=246 +13:25:58 step 155380 loss=3.3043 lr=3.13e-05 tok=20.460B dt=32709 wait=246 +13:27:01 step 155400 loss=3.3923 lr=3.12e-05 tok=20.463B dt=32771 wait=247 +13:27:13 pushed ckpt_w137.pt,log_w137.txt +13:31:20 step 155420 loss=3.3628 lr=3.12e-05 tok=20.466B dt=33030 wait=247 +13:32:22 step 155440 loss=3.3410 lr=3.12e-05 tok=20.468B dt=33093 wait=247 +13:33:25 step 155460 loss=3.3139 lr=3.12e-05 tok=20.471B dt=33155 wait=248 +13:34:27 step 155480 loss=3.3082 lr=3.12e-05 tok=20.473B dt=33218 wait=248 +13:35:31 step 155500 loss=3.3885 lr=3.12e-05 tok=20.476B dt=33281 wait=248 +13:36:33 step 155520 loss=3.3372 lr=3.12e-05 tok=20.479B dt=33344 wait=249 +13:37:35 step 155540 loss=3.4928 lr=3.11e-05 tok=20.481B dt=33406 wait=249 +13:38:19 pushed ckpt_w137.pt,log_w137.txt +13:41:54 step 155560 loss=3.4175 lr=3.11e-05 tok=20.484B dt=33665 wait=249 +13:42:57 step 155580 loss=3.3314 lr=3.11e-05 tok=20.487B dt=33728 wait=250 +13:44:00 step 155600 loss=3.3897 lr=3.11e-05 tok=20.489B dt=33791 wait=250 +13:45:02 step 155620 loss=3.3291 lr=3.11e-05 tok=20.492B dt=33853 wait=250 +13:46:05 step 155640 loss=3.3465 lr=3.11e-05 tok=20.494B dt=33915 wait=251 +13:47:07 step 155660 loss=3.3997 lr=3.11e-05 tok=20.497B dt=33977 wait=251 +13:48:09 step 155680 loss=3.4996 lr=3.10e-05 tok=20.500B dt=34040 wait=251 +13:49:11 step 155700 loss=3.2220 lr=3.10e-05 tok=20.502B dt=34102 wait=252 +13:49:27 pushed ckpt_w137.pt,log_w137.txt +13:53:33 step 155720 loss=3.3330 lr=3.10e-05 tok=20.505B dt=34364 wait=252 +13:54:36 step 155740 loss=3.2501 lr=3.10e-05 tok=20.508B dt=34426 wait=252 +13:55:38 step 155760 loss=3.2695 lr=3.10e-05 tok=20.510B dt=34489 wait=252 +13:56:42 step 155780 loss=3.0848 lr=3.10e-05 tok=20.513B dt=34552 wait=253 +13:57:44 step 155800 loss=3.2657 lr=3.10e-05 tok=20.515B dt=34614 wait=253 +13:58:46 step 155820 loss=3.3719 lr=3.10e-05 tok=20.518B dt=34676 wait=253 +13:59:49 step 155840 loss=3.4355 lr=3.09e-05 tok=20.521B dt=34739 wait=254 +14:00:35 pushed ckpt_w137.pt,log_w137.txt +14:04:10 step 155860 loss=3.4460 lr=3.09e-05 tok=20.523B dt=35001 wait=254 +14:05:13 step 155880 loss=3.4058 lr=3.09e-05 tok=20.526B dt=35063 wait=254 +14:06:15 step 155900 loss=3.3538 lr=3.09e-05 tok=20.528B dt=35126 wait=255 +14:07:18 step 155920 loss=3.4080 lr=3.09e-05 tok=20.531B dt=35189 wait=255 +14:08:21 step 155940 loss=3.3286 lr=3.09e-05 tok=20.534B dt=35251 wait=255 +14:09:23 step 155960 loss=3.3336 lr=3.09e-05 tok=20.536B dt=35314 wait=256 +14:10:26 step 155980 loss=3.3388 lr=3.08e-05 tok=20.539B dt=35377 wait=256 +14:11:29 step 156000 loss=3.4553 lr=3.08e-05 tok=20.542B dt=35439 wait=256 +14:11:41 pushed ckpt_w137.pt,log_w137.txt +14:15:47 step 156020 loss=3.3310 lr=3.08e-05 tok=20.544B dt=35697 wait=256 +14:16:49 step 156040 loss=3.4345 lr=3.08e-05 tok=20.547B dt=35760 wait=257 +14:17:52 step 156060 loss=3.2857 lr=3.08e-05 tok=20.549B dt=35822 wait=257 +14:18:54 step 156080 loss=3.4351 lr=3.08e-05 tok=20.552B dt=35885 wait=257 +14:19:57 step 156100 loss=3.2957 lr=3.08e-05 tok=20.555B dt=35947 wait=258 +14:21:00 step 156120 loss=3.4395 lr=3.07e-05 tok=20.557B dt=36010 wait=258 +14:22:02 step 156140 loss=3.4759 lr=3.07e-05 tok=20.560B dt=36073 wait=258 +14:22:45 pushed ckpt_w137.pt,log_w137.txt +14:26:21 step 156160 loss=3.4108 lr=3.07e-05 tok=20.563B dt=36331 wait=259 +14:27:24 step 156180 loss=3.3107 lr=3.07e-05 tok=20.565B dt=36394 wait=259 +14:28:26 step 156200 loss=3.3960 lr=3.07e-05 tok=20.568B dt=36457 wait=259 +14:29:29 step 156220 loss=3.3532 lr=3.07e-05 tok=20.570B dt=36519 wait=260 +14:30:31 step 156240 loss=3.2951 lr=3.07e-05 tok=20.573B dt=36582 wait=260 +14:31:34 step 156260 loss=3.5741 lr=3.07e-05 tok=20.576B dt=36644 wait=260 +14:32:36 step 156280 loss=3.3398 lr=3.06e-05 tok=20.578B dt=36707 wait=261 +14:33:39 step 156300 loss=3.3217 lr=3.06e-05 tok=20.581B dt=36769 wait=261 +14:33:53 pushed ckpt_w137.pt,log_w137.txt +14:37:59 step 156320 loss=3.3735 lr=3.06e-05 tok=20.584B dt=37030 wait=261 +14:39:02 step 156340 loss=3.3175 lr=3.06e-05 tok=20.586B dt=37092 wait=262 +14:40:04 step 156360 loss=3.5083 lr=3.06e-05 tok=20.589B dt=37155 wait=262 +14:41:07 step 156380 loss=3.4413 lr=3.06e-05 tok=20.591B dt=37217 wait=262 +14:42:09 step 156400 loss=3.3664 lr=3.06e-05 tok=20.594B dt=37280 wait=263 +14:43:12 step 156420 loss=3.4189 lr=3.05e-05 tok=20.597B dt=37343 wait=263 +14:44:15 step 156440 loss=3.4688 lr=3.05e-05 tok=20.599B dt=37405 wait=263 +14:45:04 pushed ckpt_w137.pt,log_w137.txt +14:48:39 step 156460 loss=3.3440 lr=3.05e-05 tok=20.602B dt=37669 wait=263 +14:49:41 step 156480 loss=3.2274 lr=3.05e-05 tok=20.605B dt=37731 wait=264 +14:50:43 step 156500 loss=3.4684 lr=3.05e-05 tok=20.607B dt=37794 wait=264 +14:51:45 step 156520 loss=3.4290 lr=3.05e-05 tok=20.610B dt=37856 wait=264 +14:52:48 step 156540 loss=3.4853 lr=3.05e-05 tok=20.612B dt=37918 wait=265 +14:53:50 step 156560 loss=3.3650 lr=3.05e-05 tok=20.615B dt=37981 wait=265 +14:54:53 step 156580 loss=3.4101 lr=3.04e-05 tok=20.618B dt=38043 wait=265 +14:55:55 step 156600 loss=3.3835 lr=3.04e-05 tok=20.620B dt=38105 wait=265 +14:56:07 pushed ckpt_w137.pt,log_w137.txt +15:00:13 step 156620 loss=3.3534 lr=3.04e-05 tok=20.623B dt=38364 wait=266 +15:01:16 step 156640 loss=3.4198 lr=3.04e-05 tok=20.625B dt=38426 wait=266 +15:02:19 step 156660 loss=3.1879 lr=3.04e-05 tok=20.628B dt=38489 wait=266 +15:03:21 step 156680 loss=3.3752 lr=3.04e-05 tok=20.631B dt=38551 wait=267 +15:03:59 global r=605 load 4.292 -> 4.294 +15:05:01 step 156620 loss=3.4086 lr=3.04e-05 tok=20.635B dt=38652 wait=267 +15:06:04 step 156640 loss=3.4066 lr=3.04e-05 tok=20.637B dt=38714 wait=268 +15:07:07 step 156660 loss=3.2973 lr=3.04e-05 tok=20.640B dt=38777 wait=268 +15:08:10 step 156680 loss=3.3440 lr=3.04e-05 tok=20.643B dt=38840 wait=268 +15:09:12 step 156700 loss=3.3153 lr=3.04e-05 tok=20.645B dt=38903 wait=268 +15:10:15 step 156720 loss=3.3936 lr=3.03e-05 tok=20.648B dt=38965 wait=269 +15:11:17 step 156740 loss=3.3646 lr=3.03e-05 tok=20.650B dt=39028 wait=269 +15:12:03 pushed ckpt_w137.pt,log_w137.txt +15:15:38 step 156760 loss=3.4571 lr=3.03e-05 tok=20.653B dt=39288 wait=269 +15:16:41 step 156780 loss=3.2808 lr=3.03e-05 tok=20.656B dt=39351 wait=270 +15:17:43 step 156800 loss=3.3398 lr=3.03e-05 tok=20.658B dt=39414 wait=270 +15:18:46 step 156820 loss=3.2696 lr=3.03e-05 tok=20.661B dt=39476 wait=270 +15:19:48 step 156840 loss=3.2797 lr=3.03e-05 tok=20.664B dt=39539 wait=271 +15:19:59 global r=606 load 4.280 -> 4.299 +15:20:30 step 156760 loss=3.2893 lr=3.03e-05 tok=20.665B dt=39580 wait=271 +15:21:33 step 156780 loss=3.3703 lr=3.03e-05 tok=20.667B dt=39643 wait=271 +15:22:36 step 156800 loss=3.3443 lr=3.03e-05 tok=20.670B dt=39707 wait=271 +15:23:39 step 156820 loss=3.4217 lr=3.03e-05 tok=20.673B dt=39769 wait=272 +15:24:42 step 156840 loss=3.2229 lr=3.03e-05 tok=20.675B dt=39832 wait=272 +15:25:44 step 156860 loss=3.4025 lr=3.02e-05 tok=20.678B dt=39895 wait=272 +15:26:46 step 156880 loss=3.3573 lr=3.02e-05 tok=20.681B dt=39957 wait=273 +15:27:49 step 156900 loss=3.3342 lr=3.02e-05 tok=20.683B dt=40020 wait=273 +15:28:07 pushed ckpt_w137.pt,log_w137.txt +15:29:07 waited 61s for global r=607 +15:29:17 global r=607 load 4.297 -> 4.299 +15:30:21 step 156920 loss=3.3124 lr=3.02e-05 tok=20.686B dt=40171 wait=273 +15:31:24 step 156940 loss=3.3782 lr=3.02e-05 tok=20.688B dt=40234 wait=274 +15:32:26 step 156960 loss=3.3810 lr=3.02e-05 tok=20.691B dt=40297 wait=274 +15:33:29 step 156980 loss=3.3718 lr=3.02e-05 tok=20.694B dt=40359 wait=274 +15:34:31 step 157000 loss=3.3751 lr=3.02e-05 tok=20.696B dt=40422 wait=275 +15:35:34 step 157020 loss=3.4369 lr=3.01e-05 tok=20.699B dt=40484 wait=275 +15:36:37 step 157040 loss=3.2900 lr=3.01e-05 tok=20.702B dt=40547 wait=275 +15:37:25 pushed ckpt_w137.pt,log_w137.txt +15:38:26 waited 62s for global r=608 +15:38:35 global r=608 load 4.293 -> 4.293 +15:39:08 step 157060 loss=3.4082 lr=3.01e-05 tok=20.704B dt=40698 wait=276 +15:40:11 step 157080 loss=3.2463 lr=3.01e-05 tok=20.707B dt=40762 wait=276 +15:41:14 step 157100 loss=3.3184 lr=3.01e-05 tok=20.709B dt=40825 wait=276 +15:42:17 step 157120 loss=3.3958 lr=3.01e-05 tok=20.712B dt=40887 wait=276 +15:43:20 step 157140 loss=3.2506 lr=3.01e-05 tok=20.715B dt=40950 wait=277 +15:44:23 step 157160 loss=3.4004 lr=3.00e-05 tok=20.717B dt=41013 wait=277 +15:45:25 step 157180 loss=3.3141 lr=3.00e-05 tok=20.720B dt=41076 wait=277 +15:46:28 step 157200 loss=3.3978 lr=3.00e-05 tok=20.722B dt=41139 wait=278 +15:46:44 pushed ckpt_w137.pt,log_w137.txt +15:47:45 waited 61s for global r=609 +15:47:53 global r=609 load 4.287 -> 4.290 +15:48:57 step 157220 loss=3.3635 lr=3.00e-05 tok=20.725B dt=41287 wait=278 +15:49:59 step 157240 loss=3.3901 lr=3.00e-05 tok=20.728B dt=41350 wait=278 +15:51:02 step 157260 loss=3.4092 lr=3.00e-05 tok=20.730B dt=41412 wait=279 +15:52:05 step 157280 loss=3.3285 lr=3.00e-05 tok=20.733B dt=41475 wait=279 +15:53:07 step 157300 loss=3.3678 lr=3.00e-05 tok=20.736B dt=41538 wait=279 +15:54:10 step 157320 loss=3.4603 lr=2.99e-05 tok=20.738B dt=41600 wait=279 +15:55:13 step 157340 loss=3.2887 lr=2.99e-05 tok=20.741B dt=41663 wait=280 +15:56:03 pushed ckpt_w137.pt,log_w137.txt +15:57:06 waited 62s for global r=610 +15:57:15 global r=610 load 4.288 -> 4.288 +15:57:48 step 157360 loss=3.4490 lr=2.99e-05 tok=20.743B dt=41818 wait=280 +15:58:51 step 157380 loss=3.3185 lr=2.99e-05 tok=20.746B dt=41881 wait=280 +15:59:54 step 157400 loss=3.2885 lr=2.99e-05 tok=20.749B dt=41944 wait=281 +16:00:57 step 157420 loss=3.3829 lr=2.99e-05 tok=20.751B dt=42007 wait=281 +16:02:00 step 157440 loss=3.3258 lr=2.99e-05 tok=20.754B dt=42070 wait=281 +16:03:03 step 157460 loss=3.3645 lr=2.98e-05 tok=20.757B dt=42133 wait=282 +16:04:05 step 157480 loss=3.3276 lr=2.98e-05 tok=20.759B dt=42196 wait=282 +16:05:09 step 157500 loss=3.4016 lr=2.98e-05 tok=20.762B dt=42259 wait=282 +16:05:21 pushed ckpt_w137.pt,log_w137.txt +16:06:37 waited 76s for global r=611 +16:06:44 global r=611 load 4.298 -> 4.294 +16:07:49 step 157520 loss=3.2299 lr=2.98e-05 tok=20.764B dt=42420 wait=283 +16:08:52 step 157540 loss=3.4062 lr=2.98e-05 tok=20.767B dt=42482 wait=283 +16:09:54 step 157560 loss=3.3636 lr=2.98e-05 tok=20.770B dt=42545 wait=283 +16:10:57 step 157580 loss=3.3628 lr=2.98e-05 tok=20.772B dt=42608 wait=283 +16:12:00 step 157600 loss=3.3863 lr=2.98e-05 tok=20.775B dt=42670 wait=284 +16:13:02 step 157620 loss=3.4159 lr=2.97e-05 tok=20.778B dt=42732 wait=284 +16:14:04 step 157640 loss=3.2654 lr=2.97e-05 tok=20.780B dt=42795 wait=284 diff --git a/log_w138.txt b/log_w138.txt new file mode 100644 index 0000000000000000000000000000000000000000..6e70621a39dd7860611c8dc68d644883f9960bbe --- /dev/null +++ b/log_w138.txt @@ -0,0 +1,167 @@ +launch +15:05:20 name=w138 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 85/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:05:26 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:05:26 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:05:36 params 1.08B +15:05:37 resumed step=156600 seen=20.62B round=605 +15:05:37 stream salt=156600 +15:07:41 blocks compiled +15:07:41 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +15:10:42 step 156601 loss=3.1379 lr=3.04e-05 tok=20.620B dt=181 wait=141 +15:11:04 step 156602 loss=3.2573 lr=3.04e-05 tok=20.621B dt=203 wait=141 +15:12:05 step 156620 loss=3.4047 lr=3.04e-05 tok=20.623B dt=265 wait=142 +15:13:08 step 156640 loss=3.3542 lr=3.04e-05 tok=20.625B dt=328 wait=142 +15:14:10 step 156660 loss=3.3973 lr=3.04e-05 tok=20.628B dt=390 wait=142 +15:15:12 step 156680 loss=3.4126 lr=3.04e-05 tok=20.631B dt=451 wait=143 +15:16:15 step 156700 loss=3.5030 lr=3.04e-05 tok=20.633B dt=514 wait=143 +15:17:18 step 156720 loss=3.4166 lr=3.03e-05 tok=20.636B dt=577 wait=143 +15:18:21 step 156740 loss=3.3394 lr=3.03e-05 tok=20.639B dt=640 wait=144 +15:19:09 pushed ckpt_w138.pt,log_w138.txt +15:19:39 waited 30s for global r=606 +15:19:55 global r=606 load 4.280 -> 4.299 +15:20:29 step 156760 loss=3.4797 lr=3.03e-05 tok=20.641B dt=768 wait=144 +15:21:35 step 156780 loss=3.2414 lr=3.03e-05 tok=20.644B dt=834 wait=144 +15:22:54 step 156800 loss=3.2995 lr=3.03e-05 tok=20.646B dt=913 wait=145 +15:23:56 step 156820 loss=3.4058 lr=3.03e-05 tok=20.649B dt=976 wait=145 +15:24:59 step 156840 loss=3.2813 lr=3.03e-05 tok=20.652B dt=1038 wait=145 +15:26:01 step 156860 loss=3.3918 lr=3.02e-05 tok=20.654B dt=1101 wait=146 +15:27:04 step 156880 loss=3.2622 lr=3.02e-05 tok=20.657B dt=1163 wait=146 +15:28:06 step 156900 loss=3.3672 lr=3.02e-05 tok=20.660B dt=1225 wait=146 +15:28:20 pushed ckpt_w138.pt,log_w138.txt +15:29:05 waited 45s for global r=607 +15:29:13 global r=607 load 4.309 -> 4.299 +15:30:17 step 156920 loss=3.3286 lr=3.02e-05 tok=20.662B dt=1356 wait=147 +15:31:28 step 156940 loss=3.3986 lr=3.02e-05 tok=20.665B dt=1427 wait=147 +15:32:42 step 156960 loss=3.3534 lr=3.02e-05 tok=20.667B dt=1501 wait=147 +15:33:44 step 156980 loss=3.3728 lr=3.02e-05 tok=20.670B dt=1564 wait=148 +15:34:46 step 157000 loss=3.3304 lr=3.02e-05 tok=20.673B dt=1626 wait=148 +15:35:49 step 157020 loss=3.4263 lr=3.01e-05 tok=20.675B dt=1688 wait=148 +15:36:51 step 157040 loss=3.2796 lr=3.01e-05 tok=20.678B dt=1750 wait=149 +15:37:35 pushed ckpt_w138.pt,log_w138.txt +15:38:20 waited 45s for global r=608 +15:38:31 global r=608 load 4.298 -> 4.293 +15:39:11 step 157060 loss=3.4277 lr=3.01e-05 tok=20.681B dt=1890 wait=149 +15:40:25 step 157080 loss=3.3441 lr=3.01e-05 tok=20.683B dt=1964 wait=149 +15:41:28 step 157100 loss=3.2684 lr=3.01e-05 tok=20.686B dt=2027 wait=150 +15:42:30 step 157120 loss=3.3955 lr=3.01e-05 tok=20.688B dt=2089 wait=150 +15:43:33 step 157140 loss=3.3115 lr=3.01e-05 tok=20.691B dt=2152 wait=150 +15:44:36 step 157160 loss=3.4153 lr=3.00e-05 tok=20.694B dt=2215 wait=151 +15:45:38 step 157180 loss=3.3034 lr=3.00e-05 tok=20.696B dt=2277 wait=151 +15:46:41 step 157200 loss=3.3597 lr=3.00e-05 tok=20.699B dt=2340 wait=151 +15:46:55 pushed ckpt_w138.pt,log_w138.txt +15:47:40 waited 46s for global r=609 +15:47:47 global r=609 load 4.304 -> 4.290 +15:49:05 step 157220 loss=3.3406 lr=3.00e-05 tok=20.702B dt=2484 wait=151 +15:50:13 step 157240 loss=3.4045 lr=3.00e-05 tok=20.704B dt=2552 wait=152 +15:51:15 step 157260 loss=3.3620 lr=3.00e-05 tok=20.707B dt=2614 wait=152 +15:52:17 step 157280 loss=3.3175 lr=3.00e-05 tok=20.709B dt=2677 wait=152 +15:53:20 step 157300 loss=3.4041 lr=3.00e-05 tok=20.712B dt=2739 wait=153 +15:54:22 step 157320 loss=3.3646 lr=2.99e-05 tok=20.715B dt=2801 wait=153 +15:55:24 step 157340 loss=3.2676 lr=2.99e-05 tok=20.717B dt=2863 wait=153 +15:56:08 pushed ckpt_w138.pt,log_w138.txt +15:57:08 waited 60s for global r=610 +15:57:16 global r=610 load 4.296 -> 4.288 +15:57:52 step 157360 loss=3.3800 lr=2.99e-05 tok=20.720B dt=3011 wait=154 +15:59:10 step 157380 loss=3.3040 lr=2.99e-05 tok=20.722B dt=3089 wait=154 +16:00:13 step 157400 loss=3.2604 lr=2.99e-05 tok=20.725B dt=3152 wait=154 +16:01:16 step 157420 loss=3.3312 lr=2.99e-05 tok=20.728B dt=3215 wait=155 +16:02:18 step 157440 loss=3.2424 lr=2.99e-05 tok=20.730B dt=3278 wait=155 +16:03:20 step 157460 loss=3.3934 lr=2.98e-05 tok=20.733B dt=3340 wait=155 +16:04:23 step 157480 loss=3.3312 lr=2.98e-05 tok=20.736B dt=3402 wait=156 +16:05:25 step 157500 loss=3.3408 lr=2.98e-05 tok=20.738B dt=3464 wait=156 +16:05:37 pushed ckpt_w138.pt,log_w138.txt +16:06:38 waited 61s for global r=611 +16:06:44 global r=611 load 4.289 -> 4.294 +16:07:56 step 157520 loss=3.3564 lr=2.98e-05 tok=20.741B dt=3615 wait=156 +16:09:10 step 157540 loss=3.3789 lr=2.98e-05 tok=20.743B dt=3689 wait=157 +16:10:12 step 157560 loss=3.3773 lr=2.98e-05 tok=20.746B dt=3751 wait=157 +16:11:14 step 157580 loss=3.3125 lr=2.98e-05 tok=20.749B dt=3814 wait=157 +16:12:17 step 157600 loss=3.3290 lr=2.98e-05 tok=20.751B dt=3876 wait=158 +16:13:19 step 157620 loss=3.3569 lr=2.97e-05 tok=20.754B dt=3939 wait=158 +16:14:22 step 157640 loss=3.3236 lr=2.97e-05 tok=20.757B dt=4001 wait=158 +16:15:05 pushed ckpt_w138.pt,log_w138.txt +16:15:50 waited 45s for global r=612 +16:15:57 global r=612 load 4.300 -> 4.291 +16:16:30 step 157660 loss=3.3530 lr=2.97e-05 tok=20.759B dt=4129 wait=159 +16:17:41 step 157680 loss=3.3732 lr=2.97e-05 tok=20.762B dt=4200 wait=159 +16:18:53 step 157700 loss=3.3353 lr=2.97e-05 tok=20.764B dt=4272 wait=159 +16:19:56 step 157720 loss=3.3513 lr=2.97e-05 tok=20.767B dt=4335 wait=160 +16:20:59 step 157740 loss=3.2608 lr=2.97e-05 tok=20.770B dt=4398 wait=160 +16:22:01 step 157760 loss=3.3430 lr=2.96e-05 tok=20.772B dt=4460 wait=160 +16:23:04 step 157780 loss=3.2900 lr=2.96e-05 tok=20.775B dt=4523 wait=161 +16:24:06 step 157800 loss=3.4343 lr=2.96e-05 tok=20.778B dt=4585 wait=161 +16:24:18 pushed ckpt_w138.pt,log_w138.txt +16:28:23 step 157820 loss=3.3109 lr=2.96e-05 tok=20.780B dt=4842 wait=161 +16:29:25 step 157840 loss=3.3784 lr=2.96e-05 tok=20.783B dt=4904 wait=162 +16:30:29 step 157860 loss=3.2510 lr=2.96e-05 tok=20.785B dt=4968 wait=162 +16:30:37 global r=613 load 4.285 -> 4.292 +16:31:40 step 157820 loss=3.2729 lr=2.96e-05 tok=20.788B dt=5039 wait=162 +16:32:42 step 157840 loss=3.4074 lr=2.96e-05 tok=20.791B dt=5101 wait=163 +16:33:56 step 157860 loss=3.3641 lr=2.96e-05 tok=20.793B dt=5175 wait=163 +16:34:59 step 157880 loss=3.3458 lr=2.96e-05 tok=20.796B dt=5238 wait=163 +16:36:01 step 157900 loss=3.3462 lr=2.96e-05 tok=20.799B dt=5300 wait=164 +16:37:04 step 157920 loss=3.4235 lr=2.95e-05 tok=20.801B dt=5363 wait=164 +16:38:06 step 157940 loss=3.2620 lr=2.95e-05 tok=20.804B dt=5425 wait=164 +16:38:51 pushed ckpt_w138.pt,log_w138.txt +16:41:37 waited 166s for global r=614 +16:41:49 global r=614 load 4.300 -> 4.295 +16:42:24 step 157960 loss=3.3365 lr=2.95e-05 tok=20.806B dt=5683 wait=165 +16:43:38 step 157980 loss=3.2592 lr=2.95e-05 tok=20.809B dt=5757 wait=165 +16:44:41 step 158000 loss=3.2767 lr=2.95e-05 tok=20.812B dt=5820 wait=165 +16:45:43 step 158020 loss=3.3381 lr=2.95e-05 tok=20.814B dt=5882 wait=166 +16:46:46 step 158040 loss=3.2685 lr=2.95e-05 tok=20.817B dt=5945 wait=166 +16:47:49 step 158060 loss=3.3141 lr=2.95e-05 tok=20.819B dt=6008 wait=166 +16:48:51 step 158080 loss=3.3321 lr=2.94e-05 tok=20.822B dt=6070 wait=167 +16:49:53 step 158100 loss=3.3866 lr=2.94e-05 tok=20.825B dt=6132 wait=167 +16:50:07 pushed ckpt_w138.pt,log_w138.txt +16:52:54 waited 166s for global r=615 +16:53:06 global r=615 load 4.298 -> 4.293 +16:54:24 step 158120 loss=3.3612 lr=2.94e-05 tok=20.827B dt=6403 wait=167 +16:55:31 step 158140 loss=3.2763 lr=2.94e-05 tok=20.830B dt=6471 wait=167 +16:56:33 step 158160 loss=3.3764 lr=2.94e-05 tok=20.833B dt=6533 wait=168 +16:57:36 step 158180 loss=3.2950 lr=2.94e-05 tok=20.835B dt=6596 wait=168 +16:58:38 step 158200 loss=3.3981 lr=2.94e-05 tok=20.838B dt=6658 wait=168 +16:59:41 step 158220 loss=3.4392 lr=2.93e-05 tok=20.840B dt=6720 wait=169 +17:00:43 step 158240 loss=3.3128 lr=2.93e-05 tok=20.843B dt=6782 wait=169 +17:01:28 pushed ckpt_w138.pt,log_w138.txt +17:05:01 step 158260 loss=3.4982 lr=2.93e-05 tok=20.846B dt=7041 wait=169 +17:05:07 global r=616 load 4.281 -> 4.291 +17:05:38 step 158260 loss=3.4289 lr=2.93e-05 tok=20.847B dt=7078 wait=170 +17:06:41 step 158280 loss=3.2912 lr=2.93e-05 tok=20.850B dt=7140 wait=170 +17:08:00 step 158300 loss=3.2764 lr=2.93e-05 tok=20.852B dt=7219 wait=170 +17:09:05 step 158320 loss=3.3765 lr=2.93e-05 tok=20.855B dt=7285 wait=171 +17:10:08 step 158340 loss=3.2642 lr=2.93e-05 tok=20.857B dt=7347 wait=171 +17:11:11 step 158360 loss=3.3817 lr=2.93e-05 tok=20.860B dt=7410 wait=171 +17:12:14 step 158380 loss=3.3352 lr=2.92e-05 tok=20.863B dt=7473 wait=172 +17:13:16 step 158400 loss=3.3616 lr=2.92e-05 tok=20.865B dt=7535 wait=172 +17:13:30 pushed ckpt_w138.pt,log_w138.txt +17:16:16 waited 166s for global r=617 +17:16:22 global r=617 load 4.294 -> 4.291 +17:17:26 step 158420 loss=3.3480 lr=2.92e-05 tok=20.868B dt=7785 wait=172 +17:18:30 step 158440 loss=3.3135 lr=2.92e-05 tok=20.871B dt=7849 wait=173 +17:19:51 step 158460 loss=3.3838 lr=2.92e-05 tok=20.873B dt=7930 wait=173 +17:20:53 step 158480 loss=3.3131 lr=2.92e-05 tok=20.876B dt=7992 wait=173 +17:21:55 step 158500 loss=3.3408 lr=2.92e-05 tok=20.878B dt=8055 wait=174 +17:22:59 step 158520 loss=3.3194 lr=2.92e-05 tok=20.881B dt=8118 wait=174 +17:24:01 step 158540 loss=3.2954 lr=2.91e-05 tok=20.884B dt=8180 wait=174 +17:24:46 pushed ckpt_w138.pt,log_w138.txt +17:27:32 waited 166s for global r=618 +17:27:41 global r=618 load 4.301 -> 4.298 +17:28:20 step 158560 loss=3.3863 lr=2.91e-05 tok=20.886B dt=8439 wait=174 +17:29:35 step 158580 loss=3.3230 lr=2.91e-05 tok=20.889B dt=8514 wait=175 +17:30:38 step 158600 loss=3.3171 lr=2.91e-05 tok=20.892B dt=8577 wait=175 +17:31:40 step 158620 loss=3.4400 lr=2.91e-05 tok=20.894B dt=8639 wait=175 +17:32:42 step 158640 loss=3.2878 lr=2.91e-05 tok=20.897B dt=8701 wait=176 +17:33:44 step 158660 loss=3.3267 lr=2.91e-05 tok=20.899B dt=8763 wait=176 +17:34:47 step 158680 loss=3.2730 lr=2.90e-05 tok=20.902B dt=8826 wait=176 +17:35:49 step 158700 loss=3.3275 lr=2.90e-05 tok=20.905B dt=8889 wait=177 +17:36:04 pushed ckpt_w138.pt,log_w138.txt +17:39:06 waited 181s for global r=619 +17:39:14 global r=619 load 4.294 -> 4.290 +17:40:32 step 158720 loss=3.3179 lr=2.90e-05 tok=20.907B dt=9171 wait=177 +17:41:40 step 158740 loss=3.4172 lr=2.90e-05 tok=20.910B dt=9239 wait=177 +17:42:42 step 158760 loss=3.3526 lr=2.90e-05 tok=20.913B dt=9301 wait=178 +17:43:45 step 158780 loss=3.3705 lr=2.90e-05 tok=20.915B dt=9364 wait=178 +17:44:47 step 158800 loss=3.3524 lr=2.90e-05 tok=20.918B dt=9426 wait=178 +17:45:49 step 158820 loss=3.4684 lr=2.90e-05 tok=20.920B dt=9489 wait=179 +17:46:52 step 158840 loss=3.2936 lr=2.89e-05 tok=20.923B dt=9551 wait=179 diff --git a/log_w139.txt b/log_w139.txt new file mode 100644 index 0000000000000000000000000000000000000000..77481d9fd68b8493979cd7ee356acb655041a1a2 --- /dev/null +++ b/log_w139.txt @@ -0,0 +1,35 @@ +launch +18:15:20 name=w139 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 86/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:15:26 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:15:27 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:15:36 params 1.08B +18:15:37 resumed step=158850 seen=20.93B round=620 +18:15:37 stream salt=158850 +18:17:25 blocks compiled +18:17:25 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +18:20:07 mix source 1 paused 30 min ((Request ID: Root=1-6abaafd7-288bfd0c644ac86659007672;1b68b6bd-1ac1-41ee-8534-d4) +18:20:08 mix source 5 paused 30 min ((Request ID: Root=1-6abaafd8-456aecf3779f991677aa47f4;908b7adb-587a-4769-b975-38) +18:20:08 mix source 4 paused 30 min ((Request ID: Root=1-6abaafd8-2e494f7f434b8ac86881f7ad;57988ce3-465f-44b8-8b63-f5) +18:20:08 mix source 3 paused 30 min ((Request ID: Root=1-6abaafd8-63b30785796990dc0f58b21f;d48bbbe2-76df-4815-9811-b4) +18:20:43 step 158851 loss=3.1921 lr=2.89e-05 tok=20.928B dt=198 wait=163 +18:21:05 step 158852 loss=3.2773 lr=2.89e-05 tok=20.929B dt=220 wait=163 +18:21:32 step 158860 loss=3.6959 lr=2.89e-05 tok=20.930B dt=247 wait=163 +18:22:37 step 158880 loss=3.5034 lr=2.89e-05 tok=20.932B dt=311 wait=163 +18:23:39 step 158900 loss=3.5494 lr=2.89e-05 tok=20.935B dt=374 wait=164 +18:24:42 step 158920 loss=3.5869 lr=2.89e-05 tok=20.937B dt=437 wait=164 +18:25:44 step 158940 loss=3.6193 lr=2.89e-05 tok=20.940B dt=499 wait=164 +18:26:47 step 158960 loss=3.6728 lr=2.89e-05 tok=20.943B dt=561 wait=165 +18:27:49 step 158980 loss=3.5352 lr=2.89e-05 tok=20.945B dt=624 wait=165 +18:28:52 step 159000 loss=3.6105 lr=2.88e-05 tok=20.948B dt=687 wait=166 +18:29:08 pushed ckpt_w139.pt,log_w139.txt +18:33:17 step 159020 loss=3.6314 lr=2.88e-05 tok=20.951B dt=952 wait=166 +18:34:20 step 159040 loss=3.5822 lr=2.88e-05 tok=20.953B dt=1015 wait=166 +18:35:23 step 159060 loss=3.5340 lr=2.88e-05 tok=20.956B dt=1077 wait=167 +18:35:40 global r=621 load 4.279 -> 4.275 +18:36:43 step 159020 loss=3.5405 lr=2.88e-05 tok=20.958B dt=1158 wait=167 +18:37:46 step 159040 loss=3.5493 lr=2.88e-05 tok=20.961B dt=1220 wait=167 +18:38:48 step 159060 loss=3.6127 lr=2.88e-05 tok=20.964B dt=1283 wait=168 +18:39:51 step 159080 loss=3.5514 lr=2.88e-05 tok=20.966B dt=1346 wait=168 +18:40:53 step 159100 loss=3.5451 lr=2.88e-05 tok=20.969B dt=1408 wait=168 +18:41:56 step 159120 loss=3.6302 lr=2.88e-05 tok=20.972B dt=1471 wait=169 +18:42:59 step 159140 loss=3.5059 lr=2.88e-05 tok=20.974B dt=1533 wait=169 diff --git a/log_w140.txt b/log_w140.txt new file mode 100644 index 0000000000000000000000000000000000000000..170dd51926aa6d03b7f801ef1a2a7aa6ffadce2f --- /dev/null +++ b/log_w140.txt @@ -0,0 +1,83 @@ +launch +22:10:34 name=w140 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 87/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:10:41 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:10:41 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +22:10:50 params 1.08B +22:10:52 resumed step=159150 seen=20.98B round=622 +22:10:52 stream salt=159150 +22:12:37 blocks compiled +22:12:37 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +22:16:03 step 159151 loss=3.3141 lr=2.87e-05 tok=20.976B dt=205 wait=180 +22:16:19 step 159152 loss=3.4007 lr=2.87e-05 tok=20.976B dt=221 wait=180 +22:16:43 step 159160 loss=3.5267 lr=2.87e-05 tok=20.977B dt=246 wait=180 +22:17:00 global r=623 load 4.285 -> 4.289 +22:18:04 step 159320 loss=3.4203 lr=2.86e-05 tok=20.979B dt=326 wait=180 +22:19:08 step 159340 loss=3.4412 lr=2.86e-05 tok=20.982B dt=390 wait=180 +22:20:11 step 159360 loss=3.5003 lr=2.86e-05 tok=20.985B dt=453 wait=180 +22:21:14 step 159380 loss=3.4431 lr=2.86e-05 tok=20.987B dt=516 wait=181 +22:22:17 step 159400 loss=3.3598 lr=2.86e-05 tok=20.990B dt=580 wait=181 +22:23:21 step 159420 loss=3.4337 lr=2.86e-05 tok=20.992B dt=643 wait=181 +22:24:24 step 159440 loss=3.3619 lr=2.86e-05 tok=20.995B dt=707 wait=181 +22:25:11 pushed ckpt_w140.pt,log_w140.txt +22:25:57 waited 45s for global r=624 +22:26:06 global r=624 load 4.290 -> 4.296 +22:26:40 step 159460 loss=3.4874 lr=2.85e-05 tok=20.998B dt=842 wait=181 +22:27:43 step 159480 loss=3.3099 lr=2.85e-05 tok=21.000B dt=905 wait=182 +22:28:46 step 159500 loss=3.3815 lr=2.85e-05 tok=21.003B dt=968 wait=182 +22:29:50 step 159520 loss=3.4149 lr=2.85e-05 tok=21.006B dt=1033 wait=182 +22:30:54 step 159540 loss=3.3156 lr=2.85e-05 tok=21.008B dt=1096 wait=182 +22:31:58 step 159560 loss=3.5408 lr=2.85e-05 tok=21.011B dt=1160 wait=182 +22:33:02 step 159580 loss=3.3456 lr=2.85e-05 tok=21.013B dt=1224 wait=183 +22:34:06 step 159600 loss=3.3944 lr=2.85e-05 tok=21.016B dt=1288 wait=183 +22:34:20 pushed ckpt_w140.pt,log_w140.txt +22:35:36 waited 76s for global r=625 +22:35:42 global r=625 load 4.308 -> 4.293 +22:36:49 step 159620 loss=3.4404 lr=2.84e-05 tok=21.019B dt=1452 wait=183 +22:37:53 step 159640 loss=3.4237 lr=2.84e-05 tok=21.021B dt=1516 wait=183 +22:38:58 step 159660 loss=3.3484 lr=2.84e-05 tok=21.024B dt=1581 wait=183 +22:40:02 step 159680 loss=3.3892 lr=2.84e-05 tok=21.027B dt=1644 wait=183 +22:41:05 step 159700 loss=3.3416 lr=2.84e-05 tok=21.029B dt=1708 wait=184 +22:42:10 step 159720 loss=3.4164 lr=2.84e-05 tok=21.032B dt=1772 wait=184 +22:43:14 step 159740 loss=3.2884 lr=2.84e-05 tok=21.034B dt=1836 wait=184 +22:44:04 pushed ckpt_w140.pt,log_w140.txt +22:45:04 waited 61s for global r=626 +22:45:11 global r=626 load 4.302 -> 4.294 +22:45:44 step 159760 loss=3.5428 lr=2.84e-05 tok=21.037B dt=1987 wait=184 +22:46:48 step 159780 loss=3.2900 lr=2.83e-05 tok=21.040B dt=2050 wait=184 +22:47:52 step 159800 loss=3.2459 lr=2.83e-05 tok=21.042B dt=2114 wait=185 +22:48:56 step 159820 loss=3.3065 lr=2.83e-05 tok=21.045B dt=2179 wait=185 +22:50:00 step 159840 loss=3.2741 lr=2.83e-05 tok=21.048B dt=2243 wait=185 +22:51:03 step 159860 loss=3.3979 lr=2.83e-05 tok=21.050B dt=2306 wait=185 +22:52:07 step 159880 loss=3.3474 lr=2.83e-05 tok=21.053B dt=2369 wait=185 +22:53:10 step 159900 loss=3.3967 lr=2.83e-05 tok=21.055B dt=2432 wait=186 +22:53:22 pushed ckpt_w140.pt,log_w140.txt +22:54:22 waited 61s for global r=627 +22:54:28 global r=627 load 4.289 -> 4.289 +22:55:32 step 159920 loss=3.2844 lr=2.83e-05 tok=21.058B dt=2575 wait=186 +22:56:35 step 159940 loss=3.3188 lr=2.82e-05 tok=21.061B dt=2638 wait=186 +22:57:38 step 159960 loss=3.2499 lr=2.82e-05 tok=21.063B dt=2701 wait=186 +22:58:41 step 159980 loss=3.1760 lr=2.82e-05 tok=21.066B dt=2764 wait=186 +22:59:45 step 160000 loss=3.1572 lr=2.82e-05 tok=21.069B dt=2827 wait=186 +23:00:48 step 160020 loss=3.2441 lr=2.82e-05 tok=21.071B dt=2891 wait=187 +23:01:51 step 160040 loss=3.1958 lr=2.82e-05 tok=21.074B dt=2953 wait=187 +23:02:36 pushed ckpt_w140.pt,log_w140.txt +23:03:51 waited 76s for global r=628 +23:03:58 global r=628 load 4.285 -> 4.288 +23:04:31 step 160060 loss=3.3152 lr=2.82e-05 tok=21.076B dt=3114 wait=187 +23:05:34 step 160080 loss=3.2127 lr=2.82e-05 tok=21.079B dt=3176 wait=187 +23:06:37 step 160100 loss=3.1783 lr=2.81e-05 tok=21.082B dt=3239 wait=187 +23:07:40 step 160120 loss=3.3440 lr=2.81e-05 tok=21.084B dt=3302 wait=188 +23:08:42 step 160140 loss=3.1929 lr=2.81e-05 tok=21.087B dt=3365 wait=188 +23:09:45 step 160160 loss=3.2993 lr=2.81e-05 tok=21.089B dt=3427 wait=188 +23:10:47 step 160180 loss=3.2271 lr=2.81e-05 tok=21.092B dt=3490 wait=188 +23:11:50 step 160200 loss=3.1391 lr=2.81e-05 tok=21.095B dt=3553 wait=188 +23:12:07 pushed ckpt_w140.pt,log_w140.txt +23:13:07 waited 61s for global r=629 +23:13:13 global r=629 load 4.293 -> 4.293 +23:14:17 step 160220 loss=3.1351 lr=2.81e-05 tok=21.097B dt=3700 wait=189 +23:15:21 step 160240 loss=3.1947 lr=2.81e-05 tok=21.100B dt=3763 wait=189 +23:16:24 step 160260 loss=3.1799 lr=2.80e-05 tok=21.103B dt=3826 wait=189 +23:17:27 step 160280 loss=3.1523 lr=2.80e-05 tok=21.105B dt=3890 wait=189 +23:18:30 step 160300 loss=3.2509 lr=2.80e-05 tok=21.108B dt=3952 wait=189 +23:19:33 step 160320 loss=3.2272 lr=2.80e-05 tok=21.110B dt=4016 wait=190 +23:20:36 step 160340 loss=3.0725 lr=2.80e-05 tok=21.113B dt=4079 wait=190 diff --git a/log_w141.txt b/log_w141.txt new file mode 100644 index 0000000000000000000000000000000000000000..94e1387d6547a4ac99d44a6e56e549f7d5f362bd --- /dev/null +++ b/log_w141.txt @@ -0,0 +1,83 @@ +launch +22:07:51 name=w141 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 88/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:07:57 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:07:57 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +22:08:07 params 1.08B +22:08:08 resumed step=159150 seen=20.98B round=622 +22:08:08 stream salt=159150 +22:09:55 blocks compiled +22:09:55 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +22:12:54 step 159151 loss=2.9996 lr=2.87e-05 tok=20.976B dt=178 wait=137 +22:13:17 step 159152 loss=3.0823 lr=2.87e-05 tok=20.976B dt=202 wait=137 +22:13:44 step 159160 loss=3.4371 lr=2.87e-05 tok=20.977B dt=229 wait=138 +22:14:04 global r=623 load 4.299 -> 4.289 +22:15:08 step 159320 loss=3.2520 lr=2.86e-05 tok=20.979B dt=313 wait=138 +22:16:13 step 159340 loss=3.2126 lr=2.86e-05 tok=20.982B dt=377 wait=138 +22:17:16 step 159360 loss=3.2926 lr=2.86e-05 tok=20.985B dt=441 wait=138 +22:18:19 step 159380 loss=3.2861 lr=2.86e-05 tok=20.987B dt=504 wait=138 +22:19:23 step 159400 loss=3.2486 lr=2.86e-05 tok=20.990B dt=568 wait=139 +22:20:27 step 159420 loss=3.4199 lr=2.86e-05 tok=20.992B dt=632 wait=139 +22:21:31 step 159440 loss=3.2460 lr=2.86e-05 tok=20.995B dt=696 wait=139 +22:22:16 pushed ckpt_w141.pt,log_w141.txt +22:25:52 step 159460 loss=3.2782 lr=2.85e-05 tok=20.998B dt=957 wait=139 +22:26:01 global r=624 load 4.311 -> 4.296 +22:26:33 step 159460 loss=3.3602 lr=2.85e-05 tok=20.999B dt=998 wait=139 +22:27:37 step 159480 loss=3.2905 lr=2.85e-05 tok=21.002B dt=1062 wait=140 +22:28:41 step 159500 loss=3.2432 lr=2.85e-05 tok=21.004B dt=1126 wait=140 +22:29:45 step 159520 loss=3.3365 lr=2.85e-05 tok=21.007B dt=1190 wait=140 +22:30:49 step 159540 loss=3.3587 lr=2.85e-05 tok=21.010B dt=1254 wait=140 +22:31:54 step 159560 loss=3.3928 lr=2.85e-05 tok=21.012B dt=1319 wait=140 +22:32:57 step 159580 loss=3.3061 lr=2.85e-05 tok=21.015B dt=1382 wait=141 +22:34:01 step 159600 loss=3.4015 lr=2.85e-05 tok=21.017B dt=1446 wait=141 +22:34:13 pushed ckpt_w141.pt,log_w141.txt +22:35:44 waited 91s for global r=625 +22:35:54 global r=625 load 4.281 -> 4.293 +22:37:00 step 159620 loss=3.3667 lr=2.84e-05 tok=21.020B dt=1625 wait=141 +22:38:04 step 159640 loss=3.4865 lr=2.84e-05 tok=21.023B dt=1689 wait=141 +22:39:08 step 159660 loss=3.2289 lr=2.84e-05 tok=21.025B dt=1753 wait=141 +22:40:12 step 159680 loss=3.3262 lr=2.84e-05 tok=21.028B dt=1817 wait=142 +22:41:16 step 159700 loss=3.1954 lr=2.84e-05 tok=21.031B dt=1880 wait=142 +22:42:19 step 159720 loss=3.2999 lr=2.84e-05 tok=21.033B dt=1944 wait=142 +22:43:23 step 159740 loss=3.2166 lr=2.84e-05 tok=21.036B dt=2008 wait=142 +22:44:09 pushed ckpt_w141.pt,log_w141.txt +22:45:11 waited 62s for global r=626 +22:45:20 global r=626 load 4.291 -> 4.294 +22:45:53 step 159760 loss=3.3483 lr=2.84e-05 tok=21.038B dt=2158 wait=142 +22:46:57 step 159780 loss=3.2350 lr=2.83e-05 tok=21.041B dt=2222 wait=143 +22:48:01 step 159800 loss=3.1824 lr=2.83e-05 tok=21.044B dt=2286 wait=143 +22:49:04 step 159820 loss=3.2015 lr=2.83e-05 tok=21.046B dt=2349 wait=143 +22:50:08 step 159840 loss=3.2056 lr=2.83e-05 tok=21.049B dt=2413 wait=143 +22:51:11 step 159860 loss=3.3289 lr=2.83e-05 tok=21.051B dt=2476 wait=143 +22:52:14 step 159880 loss=3.3232 lr=2.83e-05 tok=21.054B dt=2539 wait=144 +22:53:16 step 159900 loss=3.3537 lr=2.83e-05 tok=21.057B dt=2601 wait=144 +22:53:30 pushed ckpt_w141.pt,log_w141.txt +22:54:30 waited 61s for global r=627 +22:54:39 global r=627 load 4.292 -> 4.289 +22:55:44 step 159920 loss=3.2083 lr=2.83e-05 tok=21.059B dt=2749 wait=144 +22:56:47 step 159940 loss=3.4260 lr=2.82e-05 tok=21.062B dt=2812 wait=144 +22:57:51 step 159960 loss=3.4161 lr=2.82e-05 tok=21.065B dt=2876 wait=144 +22:58:54 step 159980 loss=3.2869 lr=2.82e-05 tok=21.067B dt=2939 wait=144 +22:59:59 step 160000 loss=3.3418 lr=2.82e-05 tok=21.070B dt=3004 wait=145 +23:01:02 step 160020 loss=3.3618 lr=2.82e-05 tok=21.072B dt=3067 wait=145 +23:02:05 step 160040 loss=3.2841 lr=2.82e-05 tok=21.075B dt=3130 wait=145 +23:02:56 pushed ckpt_w141.pt,log_w141.txt +23:03:42 waited 46s for global r=628 +23:03:52 global r=628 load 4.279 -> 4.288 +23:04:26 step 160060 loss=3.3965 lr=2.82e-05 tok=21.078B dt=3270 wait=145 +23:05:29 step 160080 loss=3.2064 lr=2.82e-05 tok=21.080B dt=3334 wait=145 +23:06:32 step 160100 loss=3.2337 lr=2.81e-05 tok=21.083B dt=3397 wait=146 +23:07:35 step 160120 loss=3.3069 lr=2.81e-05 tok=21.086B dt=3460 wait=146 +23:08:38 step 160140 loss=3.1345 lr=2.81e-05 tok=21.088B dt=3523 wait=146 +23:09:41 step 160160 loss=3.2038 lr=2.81e-05 tok=21.091B dt=3586 wait=146 +23:10:44 step 160180 loss=3.2047 lr=2.81e-05 tok=21.093B dt=3649 wait=146 +23:11:48 step 160200 loss=3.2355 lr=2.81e-05 tok=21.096B dt=3713 wait=147 +23:12:01 pushed ckpt_w141.pt,log_w141.txt +23:13:02 waited 61s for global r=629 +23:13:13 global r=629 load 4.292 -> 4.293 +23:14:18 step 160220 loss=3.1963 lr=2.81e-05 tok=21.099B dt=3863 wait=147 +23:15:22 step 160240 loss=3.1593 lr=2.81e-05 tok=21.101B dt=3927 wait=147 +23:16:25 step 160260 loss=3.2290 lr=2.80e-05 tok=21.104B dt=3990 wait=147 +23:17:30 step 160280 loss=3.2808 lr=2.80e-05 tok=21.107B dt=4055 wait=147 +23:18:34 step 160300 loss=3.3561 lr=2.80e-05 tok=21.109B dt=4118 wait=148 +23:19:38 step 160320 loss=3.3088 lr=2.80e-05 tok=21.112B dt=4183 wait=148 +23:20:42 step 160340 loss=3.3346 lr=2.80e-05 tok=21.114B dt=4247 wait=148 diff --git a/log_w143.txt b/log_w143.txt new file mode 100644 index 0000000000000000000000000000000000000000..194e484228695b6271fc0e9818aac9715e347ecd --- /dev/null +++ b/log_w143.txt @@ -0,0 +1,93 @@ +launch +21:54:28 name=w143 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 90/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:54:35 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:54:35 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +21:54:44 params 1.08B +21:54:46 resumed step=159150 seen=20.98B round=622 +21:54:46 stream salt=159150 +21:56:30 blocks compiled +21:56:30 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +21:59:48 step 159151 loss=3.1153 lr=2.87e-05 tok=20.976B dt=198 wait=169 +22:00:04 step 159152 loss=3.3905 lr=2.87e-05 tok=20.976B dt=214 wait=169 +22:00:30 step 159160 loss=3.5498 lr=2.87e-05 tok=20.977B dt=240 wait=169 +22:01:35 step 159180 loss=3.2799 lr=2.87e-05 tok=20.979B dt=305 wait=169 +22:02:39 step 159200 loss=3.2650 lr=2.87e-05 tok=20.982B dt=369 wait=169 +22:03:43 step 159220 loss=3.3128 lr=2.87e-05 tok=20.985B dt=433 wait=169 +22:04:47 step 159240 loss=3.4747 lr=2.87e-05 tok=20.987B dt=497 wait=170 +22:05:52 step 159260 loss=3.4561 lr=2.87e-05 tok=20.990B dt=562 wait=170 +22:06:56 step 159280 loss=3.3481 lr=2.87e-05 tok=20.992B dt=626 wait=170 +22:08:01 step 159300 loss=3.4054 lr=2.86e-05 tok=20.995B dt=691 wait=170 +22:08:18 pushed ckpt_w143.pt,log_w143.txt +22:12:11 global r=623 load 4.288 -> 4.289 +22:13:16 step 159320 loss=3.2731 lr=2.86e-05 tok=20.999B dt=1006 wait=170 +22:14:20 step 159340 loss=3.3934 lr=2.86e-05 tok=21.002B dt=1070 wait=171 +22:15:25 step 159360 loss=3.3745 lr=2.86e-05 tok=21.004B dt=1135 wait=171 +22:16:30 step 159380 loss=3.2850 lr=2.86e-05 tok=21.007B dt=1200 wait=171 +22:17:34 step 159400 loss=3.2880 lr=2.86e-05 tok=21.010B dt=1265 wait=171 +22:18:40 step 159420 loss=3.3806 lr=2.86e-05 tok=21.012B dt=1330 wait=171 +22:19:43 step 159440 loss=3.2738 lr=2.86e-05 tok=21.015B dt=1393 wait=172 +22:20:27 pushed ckpt_w143.pt,log_w143.txt +22:24:05 step 159460 loss=3.3857 lr=2.85e-05 tok=21.017B dt=1655 wait=172 +22:25:10 step 159480 loss=3.2128 lr=2.85e-05 tok=21.020B dt=1720 wait=172 +22:26:19 step 159500 loss=3.1619 lr=2.85e-05 tok=21.023B dt=1789 wait=172 +22:26:33 global r=624 load 4.304 -> 4.296 +22:27:18 step 159460 loss=3.2337 lr=2.85e-05 tok=21.024B dt=1848 wait=172 +22:28:22 step 159480 loss=3.1843 lr=2.85e-05 tok=21.027B dt=1912 wait=172 +22:29:25 step 159500 loss=3.1524 lr=2.85e-05 tok=21.029B dt=1976 wait=173 +22:30:30 step 159520 loss=3.1454 lr=2.85e-05 tok=21.032B dt=2041 wait=173 +22:31:35 step 159540 loss=3.0811 lr=2.85e-05 tok=21.034B dt=2105 wait=173 +22:32:39 step 159560 loss=3.2194 lr=2.85e-05 tok=21.037B dt=2169 wait=173 +22:33:44 step 159580 loss=3.1197 lr=2.85e-05 tok=21.040B dt=2234 wait=173 +22:34:49 step 159600 loss=3.1475 lr=2.85e-05 tok=21.042B dt=2299 wait=174 +22:35:00 pushed ckpt_w143.pt,log_w143.txt +22:35:45 waited 45s for global r=625 +22:35:58 global r=625 load 4.298 -> 4.293 +22:37:04 step 159620 loss=3.1984 lr=2.84e-05 tok=21.045B dt=2434 wait=174 +22:38:16 step 159640 loss=3.1571 lr=2.84e-05 tok=21.048B dt=2506 wait=174 +22:39:33 step 159660 loss=3.2008 lr=2.84e-05 tok=21.050B dt=2583 wait=174 +22:40:37 step 159680 loss=3.1517 lr=2.84e-05 tok=21.053B dt=2647 wait=175 +22:41:41 step 159700 loss=3.1615 lr=2.84e-05 tok=21.055B dt=2711 wait=175 +22:42:45 step 159720 loss=3.2729 lr=2.84e-05 tok=21.058B dt=2775 wait=175 +22:43:48 step 159740 loss=3.2227 lr=2.84e-05 tok=21.061B dt=2839 wait=175 +22:44:35 pushed ckpt_w143.pt,log_w143.txt +22:45:05 waited 30s for global r=626 +22:45:10 global r=626 load 4.296 -> 4.294 +22:45:44 step 159760 loss=3.3071 lr=2.84e-05 tok=21.063B dt=2954 wait=175 +22:46:47 step 159780 loss=3.2582 lr=2.83e-05 tok=21.066B dt=3017 wait=176 +22:47:58 step 159800 loss=3.2680 lr=2.83e-05 tok=21.069B dt=3088 wait=176 +22:49:15 step 159820 loss=3.3540 lr=2.83e-05 tok=21.071B dt=3165 wait=176 +22:50:18 step 159840 loss=3.1912 lr=2.83e-05 tok=21.074B dt=3228 wait=176 +22:51:21 step 159860 loss=3.3719 lr=2.83e-05 tok=21.076B dt=3291 wait=176 +22:52:25 step 159880 loss=3.3353 lr=2.83e-05 tok=21.079B dt=3355 wait=177 +22:53:30 step 159900 loss=3.3466 lr=2.83e-05 tok=21.082B dt=3420 wait=177 +22:53:40 pushed ckpt_w143.pt,log_w143.txt +22:54:25 waited 45s for global r=627 +22:54:32 global r=627 load 4.295 -> 4.289 +22:55:51 step 159920 loss=3.3217 lr=2.83e-05 tok=21.084B dt=3562 wait=177 +22:57:01 step 159940 loss=3.3500 lr=2.82e-05 tok=21.087B dt=3631 wait=177 +22:58:05 step 159960 loss=3.3539 lr=2.82e-05 tok=21.089B dt=3695 wait=177 +22:59:09 step 159980 loss=3.2707 lr=2.82e-05 tok=21.092B dt=3759 wait=178 +23:00:13 step 160000 loss=3.3006 lr=2.82e-05 tok=21.095B dt=3823 wait=178 +23:01:17 step 160020 loss=3.3735 lr=2.82e-05 tok=21.097B dt=3887 wait=178 +23:02:22 step 160040 loss=3.1171 lr=2.82e-05 tok=21.100B dt=3952 wait=178 +23:03:08 pushed ckpt_w143.pt,log_w143.txt +23:03:38 waited 30s for global r=628 +23:03:45 global r=628 load 4.309 -> 4.288 +23:04:19 step 160060 loss=3.3738 lr=2.82e-05 tok=21.103B dt=4069 wait=178 +23:05:43 step 160080 loss=3.3720 lr=2.82e-05 tok=21.105B dt=4153 wait=178 +23:06:48 step 160100 loss=3.2995 lr=2.81e-05 tok=21.108B dt=4218 wait=179 +23:07:52 step 160120 loss=3.3874 lr=2.81e-05 tok=21.110B dt=4282 wait=179 +23:08:56 step 160140 loss=3.3333 lr=2.81e-05 tok=21.113B dt=4346 wait=179 +23:10:00 step 160160 loss=3.3071 lr=2.81e-05 tok=21.116B dt=4410 wait=179 +23:11:03 step 160180 loss=3.3101 lr=2.81e-05 tok=21.118B dt=4474 wait=180 +23:12:07 step 160200 loss=3.3583 lr=2.81e-05 tok=21.121B dt=4537 wait=180 +23:12:20 pushed ckpt_w143.pt,log_w143.txt +23:13:06 waited 46s for global r=629 +23:13:11 global r=629 load 4.304 -> 4.293 +23:14:23 step 160220 loss=3.3032 lr=2.81e-05 tok=21.124B dt=4674 wait=180 +23:15:41 step 160240 loss=3.4101 lr=2.81e-05 tok=21.126B dt=4751 wait=180 +23:16:45 step 160260 loss=3.4135 lr=2.80e-05 tok=21.129B dt=4815 wait=180 +23:17:49 step 160280 loss=3.3996 lr=2.80e-05 tok=21.131B dt=4879 wait=181 +23:18:52 step 160300 loss=3.3053 lr=2.80e-05 tok=21.134B dt=4942 wait=181 +23:19:55 step 160320 loss=3.5043 lr=2.80e-05 tok=21.137B dt=5006 wait=181 +23:20:59 step 160340 loss=3.3330 lr=2.80e-05 tok=21.139B dt=5069 wait=181 diff --git a/log_w146.txt b/log_w146.txt new file mode 100644 index 0000000000000000000000000000000000000000..54caf73393f523245eb60b632cc8a7e34c778640 --- /dev/null +++ b/log_w146.txt @@ -0,0 +1,174 @@ +launch +05:07:19 name=w146 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 93/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +05:07:25 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +05:07:25 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +05:07:35 params 1.08B +05:07:36 resumed step=160350 seen=21.14B round=630 +05:07:36 stream salt=160350 +05:09:19 blocks compiled +05:09:19 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:13:36 step 160351 loss=3.0198 lr=2.80e-05 tok=21.141B dt=257 wait=227 +05:13:52 step 160352 loss=3.1865 lr=2.80e-05 tok=21.141B dt=273 wait=227 +05:14:18 step 160360 loss=3.5810 lr=2.80e-05 tok=21.142B dt=298 wait=227 +05:15:23 step 160380 loss=3.4218 lr=2.80e-05 tok=21.145B dt=364 wait=228 +05:16:28 step 160400 loss=3.4108 lr=2.80e-05 tok=21.147B dt=429 wait=228 +05:17:32 step 160420 loss=3.3503 lr=2.79e-05 tok=21.150B dt=493 wait=228 +05:18:37 step 160440 loss=3.3863 lr=2.79e-05 tok=21.152B dt=557 wait=228 +05:19:41 step 160460 loss=3.4846 lr=2.79e-05 tok=21.155B dt=622 wait=228 +05:20:46 step 160480 loss=3.4424 lr=2.79e-05 tok=21.158B dt=687 wait=229 +05:21:50 step 160500 loss=3.4876 lr=2.79e-05 tok=21.160B dt=751 wait=229 +05:22:05 pushed ckpt_w146.pt,log_w146.txt +05:26:02 global r=631 load 4.286 -> 4.281 +05:27:07 step 160520 loss=3.2236 lr=2.79e-05 tok=21.164B dt=1068 wait=229 +05:28:10 step 160540 loss=3.3417 lr=2.79e-05 tok=21.167B dt=1131 wait=229 +05:29:14 step 160560 loss=3.2637 lr=2.79e-05 tok=21.169B dt=1195 wait=229 +05:30:18 step 160580 loss=3.1546 lr=2.78e-05 tok=21.172B dt=1259 wait=230 +05:31:22 step 160600 loss=3.2624 lr=2.78e-05 tok=21.175B dt=1323 wait=230 +05:32:26 step 160620 loss=3.2559 lr=2.78e-05 tok=21.177B dt=1386 wait=230 +05:33:28 step 160640 loss=3.1965 lr=2.78e-05 tok=21.180B dt=1449 wait=230 +05:34:12 pushed ckpt_w146.pt,log_w146.txt +05:37:48 step 160660 loss=3.3199 lr=2.78e-05 tok=21.183B dt=1709 wait=230 +05:37:56 global r=632 load 4.306 -> 4.301 +05:38:27 step 160660 loss=3.2832 lr=2.78e-05 tok=21.184B dt=1748 wait=230 +05:39:30 step 160680 loss=3.1470 lr=2.78e-05 tok=21.186B dt=1810 wait=231 +05:40:33 step 160700 loss=3.0869 lr=2.78e-05 tok=21.189B dt=1873 wait=231 +05:41:36 step 160720 loss=3.2175 lr=2.78e-05 tok=21.192B dt=1936 wait=231 +05:42:38 step 160740 loss=3.1411 lr=2.77e-05 tok=21.194B dt=1999 wait=231 +05:43:43 step 160760 loss=3.1590 lr=2.77e-05 tok=21.197B dt=2064 wait=231 +05:44:47 step 160780 loss=3.2459 lr=2.77e-05 tok=21.200B dt=2128 wait=232 +05:45:52 step 160800 loss=3.2594 lr=2.77e-05 tok=21.202B dt=2193 wait=232 +05:46:06 pushed ckpt_w146.pt,log_w146.txt +05:49:51 global r=633 load 4.296 -> 4.300 +05:50:56 step 160820 loss=3.1064 lr=2.77e-05 tok=21.206B dt=2497 wait=232 +05:52:01 step 160840 loss=3.3213 lr=2.77e-05 tok=21.209B dt=2562 wait=232 +05:53:06 step 160860 loss=3.2653 lr=2.77e-05 tok=21.211B dt=2627 wait=233 +05:54:11 step 160880 loss=3.2972 lr=2.77e-05 tok=21.214B dt=2692 wait=233 +05:55:16 step 160900 loss=3.2559 lr=2.76e-05 tok=21.217B dt=2757 wait=233 +05:56:21 step 160920 loss=3.4751 lr=2.76e-05 tok=21.219B dt=2822 wait=233 +05:57:25 step 160940 loss=3.3308 lr=2.76e-05 tok=21.222B dt=2886 wait=233 +05:58:13 pushed ckpt_w146.pt,log_w146.txt +06:01:50 step 160960 loss=3.4880 lr=2.76e-05 tok=21.224B dt=3151 wait=234 +06:03:08 global r=634 load 4.292 -> 4.292 +06:03:41 step 160960 loss=3.4951 lr=2.76e-05 tok=21.226B dt=3261 wait=234 +06:04:45 step 160980 loss=3.3612 lr=2.76e-05 tok=21.228B dt=3326 wait=234 +06:05:50 step 161000 loss=3.3745 lr=2.76e-05 tok=21.231B dt=3390 wait=234 +06:06:54 step 161020 loss=3.4088 lr=2.76e-05 tok=21.234B dt=3455 wait=234 +06:07:59 step 161040 loss=3.3636 lr=2.76e-05 tok=21.236B dt=3520 wait=234 +06:09:04 step 161060 loss=3.4420 lr=2.75e-05 tok=21.239B dt=3584 wait=235 +06:10:07 step 161080 loss=3.2823 lr=2.75e-05 tok=21.242B dt=3648 wait=235 +06:11:11 step 161100 loss=3.3146 lr=2.75e-05 tok=21.244B dt=3712 wait=235 +06:11:25 pushed ckpt_w146.pt,log_w146.txt +06:14:27 waited 182s for global r=635 +06:14:36 global r=635 load 4.282 -> 4.284 +06:15:42 step 161120 loss=3.2368 lr=2.75e-05 tok=21.247B dt=3983 wait=235 +06:16:47 step 161140 loss=3.2283 lr=2.75e-05 tok=21.249B dt=4047 wait=235 +06:17:50 step 161160 loss=3.2407 lr=2.75e-05 tok=21.252B dt=4111 wait=236 +06:18:54 step 161180 loss=3.2777 lr=2.75e-05 tok=21.255B dt=4175 wait=236 +06:19:58 step 161200 loss=3.2546 lr=2.75e-05 tok=21.257B dt=4239 wait=236 +06:21:02 step 161220 loss=3.3233 lr=2.74e-05 tok=21.260B dt=4303 wait=236 +06:22:07 step 161240 loss=3.1233 lr=2.74e-05 tok=21.262B dt=4367 wait=237 +06:22:53 pushed ckpt_w146.pt,log_w146.txt +06:26:29 step 161260 loss=3.3134 lr=2.74e-05 tok=21.265B dt=4629 wait=237 +06:26:36 global r=636 load 4.289 -> 4.285 +06:27:08 step 161260 loss=3.3261 lr=2.74e-05 tok=21.266B dt=4669 wait=237 +06:28:12 step 161280 loss=3.2916 lr=2.74e-05 tok=21.269B dt=4733 wait=237 +06:29:16 step 161300 loss=3.1340 lr=2.74e-05 tok=21.272B dt=4797 wait=237 +06:30:20 step 161320 loss=3.3428 lr=2.74e-05 tok=21.274B dt=4860 wait=238 +06:31:24 step 161340 loss=3.1286 lr=2.74e-05 tok=21.277B dt=4925 wait=238 +06:32:28 step 161360 loss=3.2365 lr=2.74e-05 tok=21.280B dt=4989 wait=238 +06:33:31 step 161380 loss=3.1815 lr=2.73e-05 tok=21.282B dt=5052 wait=238 +06:34:34 step 161400 loss=3.2104 lr=2.73e-05 tok=21.285B dt=5115 wait=238 +06:34:53 pushed ckpt_w146.pt,log_w146.txt +06:37:54 waited 181s for global r=637 +06:38:04 global r=637 load 4.297 -> 4.283 +06:39:09 step 161420 loss=3.1172 lr=2.73e-05 tok=21.287B dt=5390 wait=239 +06:40:13 step 161440 loss=3.2108 lr=2.73e-05 tok=21.290B dt=5454 wait=239 +06:41:17 step 161460 loss=3.2510 lr=2.73e-05 tok=21.293B dt=5518 wait=239 +06:42:20 step 161480 loss=3.1054 lr=2.73e-05 tok=21.295B dt=5581 wait=239 +06:43:24 step 161500 loss=3.2769 lr=2.73e-05 tok=21.298B dt=5644 wait=239 +06:44:28 step 161520 loss=3.3162 lr=2.73e-05 tok=21.301B dt=5709 wait=240 +06:45:31 step 161540 loss=3.2781 lr=2.73e-05 tok=21.303B dt=5772 wait=240 +06:46:16 pushed ckpt_w146.pt,log_w146.txt +06:47:18 waited 61s for global r=638 +06:47:26 global r=638 load 4.285 -> 4.279 +06:48:00 step 161560 loss=3.3682 lr=2.72e-05 tok=21.306B dt=5921 wait=240 +06:49:05 step 161580 loss=3.2715 lr=2.72e-05 tok=21.308B dt=5986 wait=240 +06:50:08 step 161600 loss=3.2427 lr=2.72e-05 tok=21.311B dt=6049 wait=240 +06:51:12 step 161620 loss=3.3741 lr=2.72e-05 tok=21.314B dt=6113 wait=241 +06:52:15 step 161640 loss=3.2386 lr=2.72e-05 tok=21.316B dt=6176 wait=241 +06:53:18 step 161660 loss=3.3220 lr=2.72e-05 tok=21.319B dt=6239 wait=241 +06:54:22 step 161680 loss=3.2667 lr=2.72e-05 tok=21.321B dt=6303 wait=241 +06:55:26 step 161700 loss=3.3114 lr=2.72e-05 tok=21.324B dt=6367 wait=241 +06:55:46 pushed ckpt_w146.pt,log_w146.txt +06:56:49 waited 63s for global r=639 +06:56:58 global r=639 load 4.285 -> 4.287 +06:58:05 step 161720 loss=3.2761 lr=2.71e-05 tok=21.327B dt=6526 wait=242 +06:59:08 step 161740 loss=3.3114 lr=2.71e-05 tok=21.329B dt=6589 wait=242 +07:00:12 step 161760 loss=3.3497 lr=2.71e-05 tok=21.332B dt=6653 wait=242 +07:01:16 step 161780 loss=3.2258 lr=2.71e-05 tok=21.335B dt=6716 wait=242 +07:02:20 step 161800 loss=3.2396 lr=2.71e-05 tok=21.337B dt=6780 wait=242 +07:03:23 step 161820 loss=3.3296 lr=2.71e-05 tok=21.340B dt=6844 wait=242 +07:04:26 step 161840 loss=3.2440 lr=2.71e-05 tok=21.342B dt=6906 wait=243 +07:05:11 pushed ckpt_w146.pt,log_w146.txt +07:06:12 waited 61s for global r=640 +07:06:20 global r=640 load 4.273 -> 4.281 +07:06:53 step 161860 loss=3.4067 lr=2.71e-05 tok=21.345B dt=7054 wait=243 +07:07:56 step 161880 loss=3.3093 lr=2.70e-05 tok=21.348B dt=7117 wait=243 +07:08:59 step 161900 loss=3.2648 lr=2.70e-05 tok=21.350B dt=7180 wait=243 +07:10:03 step 161920 loss=3.3049 lr=2.70e-05 tok=21.353B dt=7244 wait=243 +07:11:06 step 161940 loss=3.1632 lr=2.70e-05 tok=21.356B dt=7307 wait=244 +07:12:10 step 161960 loss=3.2539 lr=2.70e-05 tok=21.358B dt=7370 wait=244 +07:13:13 step 161980 loss=3.2098 lr=2.70e-05 tok=21.361B dt=7434 wait=244 +07:14:16 step 162000 loss=3.2021 lr=2.70e-05 tok=21.363B dt=7497 wait=244 +07:14:36 pushed ckpt_w146.pt,log_w146.txt +07:15:52 waited 76s for global r=641 +07:16:03 global r=641 load 4.278 -> 4.281 +07:17:07 step 162020 loss=3.1799 lr=2.70e-05 tok=21.366B dt=7668 wait=244 +07:18:11 step 162040 loss=3.2020 lr=2.69e-05 tok=21.369B dt=7732 wait=245 +07:19:14 step 162060 loss=3.2965 lr=2.69e-05 tok=21.371B dt=7795 wait=245 +07:20:18 step 162080 loss=3.1262 lr=2.69e-05 tok=21.374B dt=7859 wait=245 +07:21:22 step 162100 loss=3.1098 lr=2.69e-05 tok=21.377B dt=7923 wait=245 +07:22:27 step 162120 loss=3.2447 lr=2.69e-05 tok=21.379B dt=7988 wait=245 +07:23:31 step 162140 loss=3.1791 lr=2.69e-05 tok=21.382B dt=8052 wait=246 +07:24:19 pushed ckpt_w146.pt,log_w146.txt +07:25:20 waited 61s for global r=642 +07:25:28 global r=642 load 4.290 -> 4.282 +07:26:01 step 162160 loss=3.3145 lr=2.69e-05 tok=21.384B dt=8202 wait=246 +07:27:05 step 162180 loss=3.2081 lr=2.69e-05 tok=21.387B dt=8266 wait=246 +07:28:10 step 162200 loss=3.2394 lr=2.69e-05 tok=21.390B dt=8330 wait=246 +07:29:13 step 162220 loss=3.2835 lr=2.68e-05 tok=21.392B dt=8394 wait=246 +07:30:17 step 162240 loss=3.1581 lr=2.68e-05 tok=21.395B dt=8457 wait=246 +07:31:21 step 162260 loss=3.2278 lr=2.68e-05 tok=21.398B dt=8522 wait=247 +07:32:24 step 162280 loss=3.2089 lr=2.68e-05 tok=21.400B dt=8585 wait=247 +07:33:27 step 162300 loss=3.1974 lr=2.68e-05 tok=21.403B dt=8648 wait=247 +07:33:49 pushed ckpt_w146.pt,log_w146.txt +07:34:50 waited 60s for global r=643 +07:34:55 global r=643 load 4.280 -> 4.274 +07:36:00 step 162320 loss=3.1829 lr=2.68e-05 tok=21.405B dt=8801 wait=247 +07:37:04 step 162340 loss=3.1236 lr=2.68e-05 tok=21.408B dt=8864 wait=247 +07:38:07 step 162360 loss=3.1609 lr=2.68e-05 tok=21.411B dt=8928 wait=248 +07:39:10 step 162380 loss=3.2766 lr=2.67e-05 tok=21.413B dt=8991 wait=248 +07:40:13 step 162400 loss=3.2416 lr=2.67e-05 tok=21.416B dt=9054 wait=248 +07:41:17 step 162420 loss=3.2264 lr=2.67e-05 tok=21.418B dt=9118 wait=248 +07:42:22 step 162440 loss=3.0778 lr=2.67e-05 tok=21.421B dt=9182 wait=248 +07:43:14 pushed ckpt_w146.pt,log_w146.txt +07:44:04 waited 51s for global r=644 +07:44:16 global r=644 load 4.281 -> 4.281 +07:44:50 step 162460 loss=3.1878 lr=2.67e-05 tok=21.424B dt=9331 wait=249 +07:45:53 step 162480 loss=3.1880 lr=2.67e-05 tok=21.426B dt=9394 wait=249 +07:46:57 step 162500 loss=3.1874 lr=2.67e-05 tok=21.429B dt=9457 wait=249 +07:48:00 step 162520 loss=3.1430 lr=2.67e-05 tok=21.432B dt=9521 wait=249 +07:49:04 step 162540 loss=3.0847 lr=2.66e-05 tok=21.434B dt=9585 wait=249 +07:50:07 step 162560 loss=3.2068 lr=2.66e-05 tok=21.437B dt=9648 wait=250 +07:51:11 step 162580 loss=3.1388 lr=2.66e-05 tok=21.439B dt=9712 wait=250 +07:52:16 step 162600 loss=3.0981 lr=2.66e-05 tok=21.442B dt=9777 wait=250 +07:52:40 pushed ckpt_w146.pt,log_w146.txt +07:53:27 waited 47s for global r=645 +07:53:33 global r=645 load 4.282 -> 4.280 +07:54:39 step 162620 loss=3.0418 lr=2.66e-05 tok=21.445B dt=9920 wait=250 +07:55:43 step 162640 loss=3.2047 lr=2.66e-05 tok=21.447B dt=9984 wait=250 +07:56:46 step 162660 loss=3.1564 lr=2.66e-05 tok=21.450B dt=10047 wait=251 +07:57:50 step 162680 loss=3.0875 lr=2.66e-05 tok=21.453B dt=10111 wait=251 +07:58:53 step 162700 loss=3.0808 lr=2.66e-05 tok=21.455B dt=10174 wait=251 +07:59:57 step 162720 loss=3.1752 lr=2.65e-05 tok=21.458B dt=10237 wait=251 +08:00:59 step 162740 loss=3.0925 lr=2.65e-05 tok=21.460B dt=10300 wait=251 diff --git a/log_w147.txt b/log_w147.txt new file mode 100644 index 0000000000000000000000000000000000000000..fb6e0a47a9d8e3efea2032db67847ca858b312c4 --- /dev/null +++ b/log_w147.txt @@ -0,0 +1,121 @@ +launch +launch +06:23:10 name=w147 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 94/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:23:16 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:23:16 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:23:26 params 1.08B +06:23:27 resumed step=161100 seen=21.24B round=635 +06:23:27 stream salt=161100 +06:25:09 blocks compiled +06:25:09 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +launch +06:25:47 name=w147 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 94/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:25:53 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:25:53 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:26:02 params 1.08B +06:26:04 resumed step=161100 seen=21.24B round=635 +06:26:04 stream salt=161100 +06:27:55 blocks compiled +06:27:55 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +06:29:15 step 161101 loss=3.0108 lr=2.75e-05 tok=21.244B dt=246 wait=216 +06:29:34 step 161102 loss=3.1539 lr=2.75e-05 tok=21.244B dt=265 wait=216 +06:30:14 global r=636 load 4.286 -> 4.285 +06:30:46 step 161260 loss=3.3884 lr=2.74e-05 tok=21.247B dt=337 wait=217 +06:31:51 step 161280 loss=3.2370 lr=2.74e-05 tok=21.249B dt=402 wait=217 +06:32:55 step 161300 loss=3.2384 lr=2.74e-05 tok=21.252B dt=466 wait=217 +06:33:59 step 161320 loss=3.3509 lr=2.74e-05 tok=21.255B dt=530 wait=217 +06:35:08 step 161340 loss=3.2646 lr=2.74e-05 tok=21.257B dt=599 wait=217 +06:36:12 step 161360 loss=3.2550 lr=2.74e-05 tok=21.260B dt=663 wait=218 +06:37:16 step 161380 loss=3.2598 lr=2.73e-05 tok=21.262B dt=727 wait=218 +06:37:56 global r=637 load 4.282 -> 4.283 +06:39:01 step 161420 loss=3.1742 lr=2.73e-05 tok=21.266B dt=831 wait=218 +06:40:05 step 161440 loss=3.2006 lr=2.73e-05 tok=21.269B dt=896 wait=218 +06:41:10 step 161460 loss=3.3120 lr=2.73e-05 tok=21.272B dt=960 wait=218 +06:42:14 step 161480 loss=3.3474 lr=2.73e-05 tok=21.274B dt=1025 wait=219 +06:43:19 step 161500 loss=3.2809 lr=2.73e-05 tok=21.277B dt=1089 wait=219 +06:44:23 step 161520 loss=3.3782 lr=2.73e-05 tok=21.280B dt=1154 wait=219 +06:45:27 step 161540 loss=3.1655 lr=2.73e-05 tok=21.282B dt=1218 wait=219 +06:46:15 pushed ckpt_w147.pt,log_w147.txt +06:47:15 waited 61s for global r=638 +06:47:24 global r=638 load 4.290 -> 4.279 +06:47:57 step 161560 loss=3.3529 lr=2.72e-05 tok=21.285B dt=1367 wait=219 +06:49:01 step 161580 loss=3.0319 lr=2.72e-05 tok=21.287B dt=1431 wait=220 +06:50:04 step 161600 loss=3.2645 lr=2.72e-05 tok=21.290B dt=1495 wait=220 +06:51:07 step 161620 loss=3.2435 lr=2.72e-05 tok=21.293B dt=1558 wait=220 +06:52:11 step 161640 loss=3.0723 lr=2.72e-05 tok=21.295B dt=1621 wait=220 +06:53:14 step 161660 loss=3.3031 lr=2.72e-05 tok=21.298B dt=1684 wait=220 +06:54:18 step 161680 loss=3.0150 lr=2.72e-05 tok=21.301B dt=1748 wait=220 +06:55:21 step 161700 loss=3.1535 lr=2.72e-05 tok=21.303B dt=1812 wait=221 +06:55:34 pushed ckpt_w147.pt,log_w147.txt +06:56:50 waited 76s for global r=639 +06:56:57 global r=639 load 4.304 -> 4.287 +06:58:00 step 161720 loss=3.2074 lr=2.71e-05 tok=21.306B dt=1971 wait=221 +06:59:04 step 161740 loss=3.1484 lr=2.71e-05 tok=21.308B dt=2034 wait=221 +07:00:08 step 161760 loss=3.2279 lr=2.71e-05 tok=21.311B dt=2099 wait=221 +07:01:13 step 161780 loss=3.1616 lr=2.71e-05 tok=21.314B dt=2163 wait=221 +07:02:16 step 161800 loss=3.2366 lr=2.71e-05 tok=21.316B dt=2227 wait=222 +07:03:20 step 161820 loss=3.3261 lr=2.71e-05 tok=21.319B dt=2291 wait=222 +07:04:24 step 161840 loss=3.2438 lr=2.71e-05 tok=21.321B dt=2355 wait=222 +07:05:08 pushed ckpt_w147.pt,log_w147.txt +07:06:24 waited 76s for global r=640 +07:06:31 global r=640 load 4.290 -> 4.281 +07:07:04 step 161860 loss=3.3017 lr=2.71e-05 tok=21.324B dt=2514 wait=222 +07:08:07 step 161880 loss=3.3219 lr=2.70e-05 tok=21.327B dt=2578 wait=222 +07:09:11 step 161900 loss=3.2303 lr=2.70e-05 tok=21.329B dt=2642 wait=223 +07:10:15 step 161920 loss=3.2892 lr=2.70e-05 tok=21.332B dt=2706 wait=223 +07:11:19 step 161940 loss=3.2106 lr=2.70e-05 tok=21.335B dt=2770 wait=223 +07:12:24 step 161960 loss=3.2849 lr=2.70e-05 tok=21.337B dt=2835 wait=223 +07:13:29 step 161980 loss=3.1931 lr=2.70e-05 tok=21.340B dt=2899 wait=223 +07:14:33 step 162000 loss=3.2842 lr=2.70e-05 tok=21.342B dt=2963 wait=224 +07:14:45 pushed ckpt_w147.pt,log_w147.txt +07:15:45 waited 61s for global r=641 +07:15:51 global r=641 load 4.283 -> 4.281 +07:16:56 step 162020 loss=3.2274 lr=2.70e-05 tok=21.345B dt=3107 wait=224 +07:18:00 step 162040 loss=3.2823 lr=2.69e-05 tok=21.348B dt=3171 wait=224 +07:19:05 step 162060 loss=3.3083 lr=2.69e-05 tok=21.350B dt=3236 wait=224 +07:20:10 step 162080 loss=3.2334 lr=2.69e-05 tok=21.353B dt=3300 wait=224 +07:21:14 step 162100 loss=3.3000 lr=2.69e-05 tok=21.356B dt=3365 wait=225 +07:22:19 step 162120 loss=3.2932 lr=2.69e-05 tok=21.358B dt=3429 wait=225 +07:23:23 step 162140 loss=3.1468 lr=2.69e-05 tok=21.361B dt=3494 wait=225 +07:24:07 pushed ckpt_w147.pt,log_w147.txt +07:25:23 waited 76s for global r=642 +07:25:32 global r=642 load 4.286 -> 4.282 +07:26:06 step 162160 loss=3.3338 lr=2.69e-05 tok=21.363B dt=3656 wait=225 +07:27:10 step 162180 loss=3.1754 lr=2.69e-05 tok=21.366B dt=3720 wait=226 +07:28:14 step 162200 loss=3.2085 lr=2.69e-05 tok=21.369B dt=3785 wait=226 +07:29:19 step 162220 loss=3.2945 lr=2.68e-05 tok=21.371B dt=3849 wait=226 +07:30:24 step 162240 loss=3.3716 lr=2.68e-05 tok=21.374B dt=3915 wait=226 +07:31:28 step 162260 loss=3.3621 lr=2.68e-05 tok=21.377B dt=3979 wait=226 +07:32:32 step 162280 loss=3.2555 lr=2.68e-05 tok=21.379B dt=4043 wait=227 +07:33:37 step 162300 loss=3.2167 lr=2.68e-05 tok=21.382B dt=4107 wait=227 +07:33:49 pushed ckpt_w147.pt,log_w147.txt +07:34:49 waited 61s for global r=643 +07:34:58 global r=643 load 4.277 -> 4.274 +07:36:04 step 162320 loss=3.2427 lr=2.68e-05 tok=21.384B dt=4254 wait=227 +07:37:08 step 162340 loss=3.2085 lr=2.68e-05 tok=21.387B dt=4318 wait=227 +07:38:12 step 162360 loss=3.3430 lr=2.68e-05 tok=21.390B dt=4382 wait=227 +07:39:16 step 162380 loss=3.2434 lr=2.67e-05 tok=21.392B dt=4447 wait=228 +07:40:21 step 162400 loss=3.3045 lr=2.67e-05 tok=21.395B dt=4512 wait=228 +07:41:24 step 162420 loss=3.3734 lr=2.67e-05 tok=21.398B dt=4575 wait=228 +07:42:29 step 162440 loss=3.1292 lr=2.67e-05 tok=21.400B dt=4639 wait=228 +07:43:14 pushed ckpt_w147.pt,log_w147.txt +07:43:59 waited 45s for global r=644 +07:44:05 global r=644 load 4.283 -> 4.281 +07:44:39 step 162460 loss=3.2205 lr=2.67e-05 tok=21.403B dt=4769 wait=228 +07:45:42 step 162480 loss=3.2197 lr=2.67e-05 tok=21.405B dt=4833 wait=229 +07:46:46 step 162500 loss=3.2180 lr=2.67e-05 tok=21.408B dt=4897 wait=229 +07:47:51 step 162520 loss=3.2261 lr=2.67e-05 tok=21.411B dt=4962 wait=229 +07:48:56 step 162540 loss=3.2071 lr=2.66e-05 tok=21.413B dt=5027 wait=229 +07:50:00 step 162560 loss=3.3517 lr=2.66e-05 tok=21.416B dt=5090 wait=229 +07:51:04 step 162580 loss=3.2427 lr=2.66e-05 tok=21.418B dt=5154 wait=230 +07:52:08 step 162600 loss=3.3004 lr=2.66e-05 tok=21.421B dt=5218 wait=230 +07:52:22 pushed ckpt_w147.pt,log_w147.txt +07:53:23 waited 61s for global r=645 +07:53:29 global r=645 load 4.283 -> 4.280 +07:54:34 step 162620 loss=3.2092 lr=2.66e-05 tok=21.424B dt=5365 wait=230 +07:55:39 step 162640 loss=3.3729 lr=2.66e-05 tok=21.426B dt=5430 wait=230 +07:56:44 step 162660 loss=3.3188 lr=2.66e-05 tok=21.429B dt=5494 wait=230 +07:57:48 step 162680 loss=3.2419 lr=2.66e-05 tok=21.432B dt=5559 wait=231 +07:58:52 step 162700 loss=3.3114 lr=2.66e-05 tok=21.434B dt=5623 wait=231 +07:59:56 step 162720 loss=3.3112 lr=2.65e-05 tok=21.437B dt=5687 wait=231 +08:00:59 step 162740 loss=3.1573 lr=2.65e-05 tok=21.439B dt=5750 wait=231 diff --git a/log_w148.txt b/log_w148.txt new file mode 100644 index 0000000000000000000000000000000000000000..33af404412013204d9bb59287dbd476cab4f8c09 --- /dev/null +++ b/log_w148.txt @@ -0,0 +1,154 @@ +launch +05:44:41 name=w148 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 95/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +05:44:47 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +05:44:47 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +05:44:57 params 1.08B +05:44:58 resumed step=160650 seen=21.18B round=632 +05:44:58 stream salt=160650 +05:46:46 blocks compiled +05:46:46 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:50:03 step 160651 loss=3.1224 lr=2.78e-05 tok=21.181B dt=197 wait=171 +05:50:18 step 160652 loss=3.1248 lr=2.78e-05 tok=21.181B dt=212 wait=171 +05:50:43 step 160660 loss=3.3787 lr=2.78e-05 tok=21.183B dt=237 wait=171 +05:51:02 global r=633 load 4.305 -> 4.300 +05:52:05 step 160820 loss=3.2068 lr=2.77e-05 tok=21.185B dt=319 wait=171 +05:53:10 step 160840 loss=3.2307 lr=2.77e-05 tok=21.188B dt=383 wait=171 +05:54:13 step 160860 loss=3.3851 lr=2.77e-05 tok=21.190B dt=447 wait=171 +05:55:16 step 160880 loss=3.3812 lr=2.77e-05 tok=21.193B dt=510 wait=172 +05:56:20 step 160900 loss=3.3672 lr=2.76e-05 tok=21.196B dt=574 wait=172 +05:57:24 step 160920 loss=3.5775 lr=2.76e-05 tok=21.198B dt=638 wait=172 +05:58:28 step 160940 loss=3.3514 lr=2.76e-05 tok=21.201B dt=702 wait=172 +05:59:18 pushed ckpt_w148.pt,log_w148.txt +06:01:34 waited 136s for global r=634 +06:03:07 global r=634 load 4.295 -> 4.292 +06:03:41 step 160960 loss=3.3883 lr=2.76e-05 tok=21.204B dt=1015 wait=172 +06:04:45 step 160980 loss=3.2933 lr=2.76e-05 tok=21.206B dt=1079 wait=173 +06:05:49 step 161000 loss=3.4817 lr=2.76e-05 tok=21.209B dt=1143 wait=173 +launch +06:05:57 name=w148 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 95/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:06:02 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:06:02 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:06:11 params 1.08B +06:06:13 resumed step=160950 seen=21.22B round=634 +06:06:13 stream salt=160950 +06:06:53 step 161020 loss=3.4571 lr=2.76e-05 tok=21.211B dt=1206 wait=173 +06:07:56 step 161040 loss=3.2916 lr=2.76e-05 tok=21.214B dt=1270 wait=173 +06:07:59 blocks compiled +06:07:59 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +06:08:59 step 161060 loss=3.4594 lr=2.75e-05 tok=21.217B dt=1333 wait=173 +06:10:02 step 161080 loss=3.4270 lr=2.75e-05 tok=21.219B dt=1396 wait=174 +06:11:05 step 161100 loss=3.3945 lr=2.75e-05 tok=21.222B dt=1459 wait=174 +06:11:18 pushed ckpt_w148.pt,log_w148.txt +06:15:04 global r=635 load 4.286 -> 4.284 +06:16:07 step 161120 loss=3.2698 lr=2.75e-05 tok=21.226B dt=1761 wait=174 +06:17:10 step 161140 loss=3.3574 lr=2.75e-05 tok=21.228B dt=1824 wait=174 +06:18:13 step 161160 loss=3.2642 lr=2.75e-05 tok=21.231B dt=1887 wait=174 +06:19:16 step 161180 loss=3.3218 lr=2.75e-05 tok=21.234B dt=1949 wait=175 +06:20:20 step 161200 loss=3.2842 lr=2.75e-05 tok=21.236B dt=2013 wait=175 +06:21:24 step 161220 loss=3.3812 lr=2.74e-05 tok=21.239B dt=2078 wait=175 +06:22:29 step 161240 loss=3.2056 lr=2.74e-05 tok=21.242B dt=2142 wait=175 +06:23:28 pushed ckpt_w148.pt,log_w148.txt +06:26:15 waited 166s for global r=636 +06:26:20 global r=636 load 4.284 -> 4.285 +06:26:53 step 161260 loss=3.3793 lr=2.74e-05 tok=21.244B dt=2407 wait=176 +06:27:58 step 161280 loss=3.2337 lr=2.74e-05 tok=21.247B dt=2472 wait=176 +06:29:23 step 161300 loss=3.2001 lr=2.74e-05 tok=21.249B dt=2557 wait=176 +06:30:28 step 161320 loss=3.3397 lr=2.74e-05 tok=21.252B dt=2621 wait=176 +06:31:32 step 161340 loss=3.2379 lr=2.74e-05 tok=21.255B dt=2685 wait=176 +06:32:35 step 161360 loss=3.3668 lr=2.74e-05 tok=21.257B dt=2749 wait=177 +06:33:39 step 161380 loss=3.2451 lr=2.73e-05 tok=21.260B dt=2813 wait=177 +06:34:42 step 161400 loss=3.2879 lr=2.73e-05 tok=21.262B dt=2876 wait=177 +06:34:55 pushed ckpt_w148.pt,log_w148.txt +06:37:56 waited 181s for global r=637 +06:38:01 global r=637 load 4.282 -> 4.283 +06:39:06 step 161420 loss=3.1985 lr=2.73e-05 tok=21.265B dt=3140 wait=177 +06:40:18 step 161440 loss=3.2966 lr=2.73e-05 tok=21.268B dt=3212 wait=177 +06:41:34 step 161460 loss=3.3615 lr=2.73e-05 tok=21.270B dt=3288 wait=178 +06:42:37 step 161480 loss=3.2974 lr=2.73e-05 tok=21.273B dt=3351 wait=178 +06:43:42 step 161500 loss=3.3536 lr=2.73e-05 tok=21.276B dt=3415 wait=178 +06:44:45 step 161520 loss=3.3562 lr=2.73e-05 tok=21.278B dt=3479 wait=178 +06:45:49 step 161540 loss=3.1986 lr=2.73e-05 tok=21.281B dt=3543 wait=178 +06:46:34 pushed ckpt_w148.pt,log_w148.txt +06:47:19 waited 45s for global r=638 +06:47:25 global r=638 load 4.271 -> 4.279 +06:47:59 step 161560 loss=3.2647 lr=2.72e-05 tok=21.283B dt=3672 wait=179 +06:49:03 step 161580 loss=3.2015 lr=2.72e-05 tok=21.286B dt=3737 wait=179 +06:50:23 step 161600 loss=3.1890 lr=2.72e-05 tok=21.289B dt=3817 wait=179 +06:51:30 step 161620 loss=3.3449 lr=2.72e-05 tok=21.291B dt=3884 wait=179 +06:52:34 step 161640 loss=3.2103 lr=2.72e-05 tok=21.294B dt=3948 wait=179 +06:53:38 step 161660 loss=3.2578 lr=2.72e-05 tok=21.297B dt=4012 wait=180 +06:54:42 step 161680 loss=3.1529 lr=2.72e-05 tok=21.299B dt=4076 wait=180 +06:55:47 step 161700 loss=3.2890 lr=2.72e-05 tok=21.302B dt=4141 wait=180 +06:56:07 pushed ckpt_w148.pt,log_w148.txt +06:56:53 waited 45s for global r=639 +06:56:58 global r=639 load 4.285 -> 4.287 +06:58:04 step 161720 loss=3.2998 lr=2.71e-05 tok=21.304B dt=4277 wait=180 +06:59:16 step 161740 loss=3.2666 lr=2.71e-05 tok=21.307B dt=4350 wait=181 +07:00:33 step 161760 loss=3.3301 lr=2.71e-05 tok=21.310B dt=4427 wait=181 +07:01:38 step 161780 loss=3.2954 lr=2.71e-05 tok=21.312B dt=4492 wait=181 +07:02:42 step 161800 loss=3.3318 lr=2.71e-05 tok=21.315B dt=4556 wait=181 +07:03:46 step 161820 loss=3.3539 lr=2.71e-05 tok=21.318B dt=4620 wait=181 +07:04:50 step 161840 loss=3.2439 lr=2.71e-05 tok=21.320B dt=4684 wait=182 +07:05:36 pushed ckpt_w148.pt,log_w148.txt +07:06:21 waited 45s for global r=640 +07:06:27 global r=640 load 4.278 -> 4.281 +07:07:09 step 161860 loss=3.2687 lr=2.71e-05 tok=21.323B dt=4822 wait=182 +07:08:25 step 161880 loss=3.2900 lr=2.70e-05 tok=21.325B dt=4899 wait=182 +07:09:29 step 161900 loss=3.2450 lr=2.70e-05 tok=21.328B dt=4963 wait=182 +07:10:34 step 161920 loss=3.4013 lr=2.70e-05 tok=21.331B dt=5028 wait=182 +07:11:38 step 161940 loss=3.3354 lr=2.70e-05 tok=21.333B dt=5092 wait=183 +07:12:42 step 161960 loss=3.3838 lr=2.70e-05 tok=21.336B dt=5156 wait=183 +07:13:45 step 161980 loss=3.3476 lr=2.70e-05 tok=21.339B dt=5219 wait=183 +07:14:50 step 162000 loss=3.3517 lr=2.70e-05 tok=21.341B dt=5284 wait=183 +07:15:03 pushed ckpt_w148.pt,log_w148.txt +07:15:49 waited 45s for global r=641 +07:15:54 global r=641 load 4.281 -> 4.281 +07:17:13 step 162020 loss=3.3268 lr=2.70e-05 tok=21.344B dt=5427 wait=183 +07:18:22 step 162040 loss=3.2910 lr=2.69e-05 tok=21.346B dt=5495 wait=184 +07:19:26 step 162060 loss=3.2788 lr=2.69e-05 tok=21.349B dt=5559 wait=184 +07:20:29 step 162080 loss=3.3528 lr=2.69e-05 tok=21.352B dt=5622 wait=184 +07:21:31 step 162100 loss=3.2286 lr=2.69e-05 tok=21.354B dt=5685 wait=184 +07:22:34 step 162120 loss=3.3257 lr=2.69e-05 tok=21.357B dt=5748 wait=184 +07:23:37 step 162140 loss=3.1709 lr=2.69e-05 tok=21.359B dt=5811 wait=185 +07:24:21 pushed ckpt_w148.pt,log_w148.txt +07:25:22 waited 60s for global r=642 +07:25:31 global r=642 load 4.283 -> 4.282 +07:26:12 step 162160 loss=3.2576 lr=2.69e-05 tok=21.362B dt=5966 wait=185 +07:27:28 step 162180 loss=3.2505 lr=2.69e-05 tok=21.365B dt=6042 wait=185 +07:28:31 step 162200 loss=3.1868 lr=2.69e-05 tok=21.367B dt=6105 wait=185 +07:29:35 step 162220 loss=3.3436 lr=2.68e-05 tok=21.370B dt=6168 wait=185 +07:30:38 step 162240 loss=3.2632 lr=2.68e-05 tok=21.373B dt=6232 wait=186 +07:31:41 step 162260 loss=3.3042 lr=2.68e-05 tok=21.375B dt=6295 wait=186 +07:32:44 step 162280 loss=3.3564 lr=2.68e-05 tok=21.378B dt=6358 wait=186 +07:33:48 step 162300 loss=3.2746 lr=2.68e-05 tok=21.380B dt=6422 wait=186 +07:34:00 pushed ckpt_w148.pt,log_w148.txt +07:34:45 waited 45s for global r=643 +07:34:53 global r=643 load 4.266 -> 4.274 +07:36:05 step 162320 loss=3.2160 lr=2.68e-05 tok=21.383B dt=6559 wait=186 +07:37:19 step 162340 loss=3.2422 lr=2.68e-05 tok=21.386B dt=6633 wait=187 +07:38:23 step 162360 loss=3.2612 lr=2.68e-05 tok=21.388B dt=6697 wait=187 +07:39:26 step 162380 loss=3.2253 lr=2.67e-05 tok=21.391B dt=6760 wait=187 +07:40:28 step 162400 loss=3.3393 lr=2.67e-05 tok=21.394B dt=6822 wait=187 +07:41:31 step 162420 loss=3.3143 lr=2.67e-05 tok=21.396B dt=6885 wait=187 +07:42:33 step 162440 loss=3.2199 lr=2.67e-05 tok=21.399B dt=6947 wait=187 +07:43:17 pushed ckpt_w148.pt,log_w148.txt +07:44:02 waited 45s for global r=644 +07:44:06 global r=644 load 4.289 -> 4.281 +07:44:39 step 162460 loss=3.2985 lr=2.67e-05 tok=21.401B dt=7073 wait=188 +07:45:57 step 162480 loss=3.2825 lr=2.67e-05 tok=21.404B dt=7150 wait=188 +07:47:05 step 162500 loss=3.2453 lr=2.67e-05 tok=21.407B dt=7219 wait=188 +07:48:08 step 162520 loss=3.3140 lr=2.67e-05 tok=21.409B dt=7282 wait=188 +07:49:11 step 162540 loss=3.3150 lr=2.66e-05 tok=21.412B dt=7345 wait=189 +07:50:14 step 162560 loss=3.3490 lr=2.66e-05 tok=21.415B dt=7408 wait=189 +07:51:17 step 162580 loss=3.1425 lr=2.66e-05 tok=21.417B dt=7471 wait=189 +07:52:21 step 162600 loss=3.3189 lr=2.66e-05 tok=21.420B dt=7535 wait=189 +07:52:33 pushed ckpt_w148.pt,log_w148.txt +07:53:34 waited 61s for global r=645 +07:53:39 global r=645 load 4.279 -> 4.280 +07:54:51 step 162620 loss=3.1879 lr=2.66e-05 tok=21.422B dt=7684 wait=189 +07:56:08 step 162640 loss=3.2000 lr=2.66e-05 tok=21.425B dt=7762 wait=189 +07:57:11 step 162660 loss=3.1834 lr=2.66e-05 tok=21.428B dt=7825 wait=190 +07:58:15 step 162680 loss=3.1392 lr=2.66e-05 tok=21.430B dt=7888 wait=190 +07:59:17 step 162700 loss=3.2342 lr=2.66e-05 tok=21.433B dt=7951 wait=190 +08:00:21 step 162720 loss=3.2468 lr=2.65e-05 tok=21.436B dt=8014 wait=190 +08:01:25 step 162740 loss=3.0745 lr=2.65e-05 tok=21.438B dt=8079 wait=190 diff --git a/log_w149.txt b/log_w149.txt new file mode 100644 index 0000000000000000000000000000000000000000..830e13bfb068e8c3ab63fddf2d7a693a6c2a26a6 --- /dev/null +++ b/log_w149.txt @@ -0,0 +1,126 @@ +launch +launch +06:15:38 name=w149 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 96/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:15:45 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:15:45 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:15:54 params 1.08B +06:15:56 resumed step=161100 seen=21.24B round=635 +06:15:56 stream salt=161100 +06:17:43 blocks compiled +06:17:43 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +06:22:12 step 161101 loss=3.1095 lr=2.75e-05 tok=21.244B dt=269 wait=239 +06:22:28 step 161102 loss=3.2614 lr=2.75e-05 tok=21.244B dt=285 wait=239 +06:23:25 step 161120 loss=3.4561 lr=2.75e-05 tok=21.247B dt=342 wait=240 +06:24:29 step 161140 loss=3.3864 lr=2.75e-05 tok=21.249B dt=406 wait=240 +06:25:33 step 161160 loss=3.3999 lr=2.75e-05 tok=21.252B dt=471 wait=240 +launch +06:25:39 name=w149 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 96/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:25:44 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:25:44 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:25:54 params 1.08B +06:25:55 resumed step=161100 seen=21.24B round=635 +06:25:55 stream salt=161100 +06:26:21 global r=636 load 4.276 -> 4.285 +06:26:53 step 161260 loss=3.4684 lr=2.74e-05 tok=21.255B dt=550 wait=240 +06:27:41 blocks compiled +06:27:41 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +06:27:57 step 161280 loss=3.4237 lr=2.74e-05 tok=21.257B dt=614 wait=241 +06:29:01 step 161300 loss=3.4367 lr=2.74e-05 tok=21.260B dt=678 wait=241 +06:30:06 step 161320 loss=3.4882 lr=2.74e-05 tok=21.262B dt=743 wait=241 +06:31:10 step 161340 loss=3.5630 lr=2.74e-05 tok=21.265B dt=807 wait=241 +06:32:15 step 161360 loss=3.4997 lr=2.74e-05 tok=21.268B dt=873 wait=241 +06:33:20 step 161380 loss=3.6122 lr=2.73e-05 tok=21.270B dt=937 wait=242 +06:34:24 step 161400 loss=3.4700 lr=2.73e-05 tok=21.273B dt=1001 wait=242 +06:34:37 pushed ckpt_w149.pt,log_w149.txt +06:38:20 global r=637 load 4.278 -> 4.283 +06:39:24 step 161420 loss=3.4730 lr=2.73e-05 tok=21.277B dt=1301 wait=242 +06:40:28 step 161440 loss=3.4776 lr=2.73e-05 tok=21.280B dt=1365 wait=242 +06:41:32 step 161460 loss=3.4026 lr=2.73e-05 tok=21.282B dt=1429 wait=242 +06:42:36 step 161480 loss=3.3043 lr=2.73e-05 tok=21.285B dt=1493 wait=243 +06:43:40 step 161500 loss=3.1936 lr=2.73e-05 tok=21.287B dt=1557 wait=243 +06:44:44 step 161520 loss=3.2198 lr=2.73e-05 tok=21.290B dt=1622 wait=243 +06:45:48 step 161540 loss=3.1806 lr=2.73e-05 tok=21.293B dt=1686 wait=243 +06:46:33 pushed ckpt_w149.pt,log_w149.txt +06:47:19 waited 45s for global r=638 +06:47:26 global r=638 load 4.282 -> 4.279 +06:47:59 step 161560 loss=3.3194 lr=2.72e-05 tok=21.295B dt=1816 wait=243 +06:49:03 step 161580 loss=3.2022 lr=2.72e-05 tok=21.298B dt=1880 wait=244 +06:50:08 step 161600 loss=3.1913 lr=2.72e-05 tok=21.301B dt=1945 wait=244 +06:51:12 step 161620 loss=3.2186 lr=2.72e-05 tok=21.303B dt=2010 wait=244 +06:52:16 step 161640 loss=3.1312 lr=2.72e-05 tok=21.306B dt=2074 wait=244 +06:53:21 step 161660 loss=3.2092 lr=2.72e-05 tok=21.308B dt=2139 wait=244 +06:54:27 step 161680 loss=3.0512 lr=2.72e-05 tok=21.311B dt=2204 wait=245 +06:55:32 step 161700 loss=3.1039 lr=2.72e-05 tok=21.314B dt=2270 wait=245 +06:55:44 pushed ckpt_w149.pt,log_w149.txt +06:56:45 waited 61s for global r=639 +06:56:54 global r=639 load 4.284 -> 4.287 +06:58:00 step 161720 loss=3.0789 lr=2.71e-05 tok=21.316B dt=2417 wait=245 +06:59:05 step 161740 loss=3.0591 lr=2.71e-05 tok=21.319B dt=2482 wait=245 +07:00:09 step 161760 loss=3.1823 lr=2.71e-05 tok=21.321B dt=2546 wait=245 +07:01:13 step 161780 loss=3.1075 lr=2.71e-05 tok=21.324B dt=2610 wait=246 +07:02:18 step 161800 loss=3.1770 lr=2.71e-05 tok=21.327B dt=2675 wait=246 +07:03:23 step 161820 loss=3.1589 lr=2.71e-05 tok=21.329B dt=2740 wait=246 +07:04:27 step 161840 loss=3.0212 lr=2.71e-05 tok=21.332B dt=2804 wait=246 +07:05:11 pushed ckpt_w149.pt,log_w149.txt +07:06:26 waited 76s for global r=640 +07:06:32 global r=640 load 4.295 -> 4.281 +07:07:05 step 161860 loss=3.1922 lr=2.71e-05 tok=21.335B dt=2962 wait=246 +07:08:09 step 161880 loss=3.1067 lr=2.70e-05 tok=21.337B dt=3026 wait=246 +07:09:14 step 161900 loss=3.0703 lr=2.70e-05 tok=21.340B dt=3091 wait=247 +07:10:18 step 161920 loss=3.1161 lr=2.70e-05 tok=21.342B dt=3155 wait=247 +07:11:21 step 161940 loss=3.0560 lr=2.70e-05 tok=21.345B dt=3219 wait=247 +07:12:25 step 161960 loss=3.1088 lr=2.70e-05 tok=21.348B dt=3283 wait=247 +07:13:31 step 161980 loss=2.9600 lr=2.70e-05 tok=21.350B dt=3348 wait=247 +07:14:35 step 162000 loss=3.0973 lr=2.70e-05 tok=21.353B dt=3412 wait=248 +07:14:47 pushed ckpt_w149.pt,log_w149.txt +07:15:48 waited 61s for global r=641 +07:15:52 global r=641 load 4.293 -> 4.281 +07:16:57 step 162020 loss=3.1776 lr=2.70e-05 tok=21.356B dt=3554 wait=248 +07:18:00 step 162040 loss=3.3003 lr=2.69e-05 tok=21.358B dt=3617 wait=248 +07:19:03 step 162060 loss=3.2339 lr=2.69e-05 tok=21.361B dt=3681 wait=248 +07:20:07 step 162080 loss=3.2695 lr=2.69e-05 tok=21.363B dt=3744 wait=248 +07:21:10 step 162100 loss=3.2884 lr=2.69e-05 tok=21.366B dt=3807 wait=248 +07:22:13 step 162120 loss=3.3219 lr=2.69e-05 tok=21.369B dt=3870 wait=249 +07:23:17 step 162140 loss=3.2582 lr=2.69e-05 tok=21.371B dt=3934 wait=249 +07:24:00 pushed ckpt_w149.pt,log_w149.txt +07:25:16 waited 76s for global r=642 +07:25:21 global r=642 load 4.278 -> 4.282 +07:25:54 step 162160 loss=3.3380 lr=2.69e-05 tok=21.374B dt=4091 wait=249 +07:26:57 step 162180 loss=3.2660 lr=2.69e-05 tok=21.377B dt=4154 wait=249 +07:28:00 step 162200 loss=3.1868 lr=2.69e-05 tok=21.379B dt=4217 wait=249 +07:29:04 step 162220 loss=3.1970 lr=2.68e-05 tok=21.382B dt=4281 wait=250 +07:30:07 step 162240 loss=3.1576 lr=2.68e-05 tok=21.384B dt=4344 wait=250 +07:31:10 step 162260 loss=3.2567 lr=2.68e-05 tok=21.387B dt=4407 wait=250 +07:32:13 step 162280 loss=3.2322 lr=2.68e-05 tok=21.390B dt=4470 wait=250 +07:33:16 step 162300 loss=3.2173 lr=2.68e-05 tok=21.392B dt=4533 wait=250 +07:33:28 pushed ckpt_w149.pt,log_w149.txt +07:34:43 waited 76s for global r=643 +07:34:49 global r=643 load 4.283 -> 4.274 +07:35:54 step 162320 loss=3.1311 lr=2.68e-05 tok=21.395B dt=4691 wait=250 +07:36:56 step 162340 loss=3.2458 lr=2.68e-05 tok=21.398B dt=4754 wait=251 +07:37:59 step 162360 loss=3.2194 lr=2.68e-05 tok=21.400B dt=4816 wait=251 +07:39:03 step 162380 loss=3.1450 lr=2.67e-05 tok=21.403B dt=4880 wait=251 +07:40:06 step 162400 loss=3.2549 lr=2.67e-05 tok=21.405B dt=4943 wait=251 +07:41:10 step 162420 loss=3.3747 lr=2.67e-05 tok=21.408B dt=5007 wait=252 +07:42:14 step 162440 loss=3.2390 lr=2.67e-05 tok=21.411B dt=5071 wait=252 +07:42:58 pushed ckpt_w149.pt,log_w149.txt +07:43:59 waited 61s for global r=644 +07:44:07 global r=644 load 4.279 -> 4.281 +07:44:40 step 162460 loss=3.3451 lr=2.67e-05 tok=21.413B dt=5217 wait=252 +07:45:43 step 162480 loss=3.2291 lr=2.67e-05 tok=21.416B dt=5280 wait=252 +07:46:46 step 162500 loss=3.1623 lr=2.67e-05 tok=21.418B dt=5343 wait=252 +07:47:49 step 162520 loss=3.3393 lr=2.67e-05 tok=21.421B dt=5406 wait=252 +07:48:52 step 162540 loss=3.1775 lr=2.66e-05 tok=21.424B dt=5469 wait=253 +07:49:55 step 162560 loss=3.1352 lr=2.66e-05 tok=21.426B dt=5532 wait=253 +07:50:58 step 162580 loss=3.0433 lr=2.66e-05 tok=21.429B dt=5595 wait=253 +07:52:00 step 162600 loss=3.1201 lr=2.66e-05 tok=21.432B dt=5657 wait=253 +07:52:13 pushed ckpt_w149.pt,log_w149.txt +07:53:29 waited 76s for global r=645 +07:53:34 global r=645 load 4.282 -> 4.280 +07:54:38 step 162620 loss=3.0309 lr=2.66e-05 tok=21.434B dt=5816 wait=253 +07:55:41 step 162640 loss=3.0422 lr=2.66e-05 tok=21.437B dt=5879 wait=254 +07:56:44 step 162660 loss=3.1397 lr=2.66e-05 tok=21.439B dt=5941 wait=254 +07:57:47 step 162680 loss=3.1816 lr=2.66e-05 tok=21.442B dt=6004 wait=254 +07:58:49 step 162700 loss=3.1638 lr=2.66e-05 tok=21.445B dt=6067 wait=254 +07:59:53 step 162720 loss=3.3223 lr=2.65e-05 tok=21.447B dt=6130 wait=254 +08:00:56 step 162740 loss=3.1164 lr=2.65e-05 tok=21.450B dt=6193 wait=255 diff --git a/log_w150.txt b/log_w150.txt new file mode 100644 index 0000000000000000000000000000000000000000..0ff0496ae12e1cf5adb25110f9c1e545e2cfefb2 --- /dev/null +++ b/log_w150.txt @@ -0,0 +1,167 @@ +launch +12:19:00 name=w150 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 97/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:19:07 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:19:07 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:19:16 params 1.08B +12:19:18 resumed step=162750 seen=21.46B round=646 +12:19:18 stream salt=162750 +12:21:17 blocks compiled +12:21:17 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:33:51 step 162751 loss=3.2919 lr=2.65e-05 tok=21.462B dt=754 wait=728 +12:34:08 step 162752 loss=3.1678 lr=2.65e-05 tok=21.462B dt=771 wait=728 +12:34:33 step 162760 loss=3.5004 lr=2.65e-05 tok=21.463B dt=796 wait=728 +12:35:37 step 162780 loss=3.2433 lr=2.65e-05 tok=21.466B dt=860 wait=728 +12:36:42 step 162800 loss=3.3126 lr=2.65e-05 tok=21.468B dt=925 wait=729 +12:37:45 step 162820 loss=3.2975 lr=2.65e-05 tok=21.471B dt=988 wait=729 +12:38:49 step 162840 loss=3.3483 lr=2.65e-05 tok=21.474B dt=1052 wait=729 +12:39:52 step 162860 loss=3.4144 lr=2.65e-05 tok=21.476B dt=1115 wait=729 +12:40:56 step 162880 loss=3.3714 lr=2.64e-05 tok=21.479B dt=1179 wait=729 +12:42:00 step 162900 loss=3.4100 lr=2.64e-05 tok=21.481B dt=1243 wait=730 +12:42:15 pushed ckpt_w150.pt,log_w150.txt +12:46:23 step 162920 loss=3.4781 lr=2.64e-05 tok=21.484B dt=1506 wait=730 +12:47:28 step 162940 loss=3.3873 lr=2.64e-05 tok=21.487B dt=1571 wait=730 +12:48:18 global r=647 load 4.292 -> 4.281 +12:49:22 step 162920 loss=3.2803 lr=2.64e-05 tok=21.491B dt=1685 wait=730 +12:50:26 step 162940 loss=3.3643 lr=2.64e-05 tok=21.493B dt=1749 wait=731 +12:51:31 step 162960 loss=3.4398 lr=2.64e-05 tok=21.496B dt=1814 wait=731 +12:52:35 step 162980 loss=3.2782 lr=2.64e-05 tok=21.498B dt=1878 wait=731 +12:53:38 step 163000 loss=3.3054 lr=2.64e-05 tok=21.501B dt=1941 wait=731 +12:54:41 step 163020 loss=3.4417 lr=2.64e-05 tok=21.504B dt=2004 wait=731 +12:55:46 step 163040 loss=3.2466 lr=2.64e-05 tok=21.506B dt=2069 wait=732 +12:56:32 pushed ckpt_w150.pt,log_w150.txt +12:57:32 waited 61s for global r=648 +12:57:42 global r=648 load 4.287 -> 4.281 +12:58:16 step 163060 loss=3.3488 lr=2.63e-05 tok=21.509B dt=2219 wait=732 +12:59:19 step 163080 loss=3.2923 lr=2.63e-05 tok=21.512B dt=2282 wait=732 +13:00:21 step 163100 loss=3.3087 lr=2.63e-05 tok=21.514B dt=2344 wait=732 +13:01:25 step 163120 loss=3.3103 lr=2.63e-05 tok=21.517B dt=2408 wait=732 +13:02:28 step 163140 loss=3.2033 lr=2.63e-05 tok=21.519B dt=2471 wait=733 +13:03:31 step 163160 loss=3.3073 lr=2.63e-05 tok=21.522B dt=2534 wait=733 +13:04:34 step 163180 loss=3.2032 lr=2.63e-05 tok=21.525B dt=2597 wait=733 +13:05:38 step 163200 loss=3.2797 lr=2.63e-05 tok=21.527B dt=2661 wait=733 +13:05:53 pushed ckpt_w150.pt,log_w150.txt +13:07:08 waited 76s for global r=649 +13:07:22 global r=649 load 4.281 -> 4.278 +13:08:27 step 163220 loss=3.3308 lr=2.62e-05 tok=21.530B dt=2830 wait=733 +13:09:31 step 163240 loss=3.3677 lr=2.62e-05 tok=21.533B dt=2894 wait=734 +13:10:35 step 163260 loss=3.3305 lr=2.62e-05 tok=21.535B dt=2958 wait=734 +13:11:39 step 163280 loss=3.2996 lr=2.62e-05 tok=21.538B dt=3022 wait=734 +13:12:42 step 163300 loss=3.4491 lr=2.62e-05 tok=21.540B dt=3085 wait=734 +13:13:45 step 163320 loss=3.4226 lr=2.62e-05 tok=21.543B dt=3148 wait=734 +13:14:49 step 163340 loss=3.2801 lr=2.62e-05 tok=21.546B dt=3212 wait=735 +13:15:38 pushed ckpt_w150.pt,log_w150.txt +13:16:39 waited 61s for global r=650 +13:16:52 global r=650 load 4.281 -> 4.282 +13:17:26 step 163360 loss=3.3341 lr=2.62e-05 tok=21.548B dt=3369 wait=735 +13:18:30 step 163380 loss=3.2607 lr=2.62e-05 tok=21.551B dt=3433 wait=735 +13:19:34 step 163400 loss=3.3530 lr=2.61e-05 tok=21.553B dt=3497 wait=735 +13:20:39 step 163420 loss=3.2542 lr=2.61e-05 tok=21.556B dt=3562 wait=735 +13:21:43 step 163440 loss=3.2835 lr=2.61e-05 tok=21.559B dt=3626 wait=735 +13:22:46 step 163460 loss=3.3592 lr=2.61e-05 tok=21.561B dt=3689 wait=736 +13:23:50 step 163480 loss=3.1615 lr=2.61e-05 tok=21.564B dt=3753 wait=736 +13:24:54 step 163500 loss=3.1423 lr=2.61e-05 tok=21.567B dt=3817 wait=736 +13:25:08 pushed ckpt_w150.pt,log_w150.txt +13:26:08 waited 61s for global r=651 +13:26:14 global r=651 load 4.285 -> 4.282 +13:27:20 step 163520 loss=3.0608 lr=2.61e-05 tok=21.569B dt=3963 wait=736 +13:28:23 step 163540 loss=3.1658 lr=2.61e-05 tok=21.572B dt=4026 wait=736 +13:29:26 step 163560 loss=3.0750 lr=2.60e-05 tok=21.574B dt=4089 wait=737 +13:30:30 step 163580 loss=3.1567 lr=2.60e-05 tok=21.577B dt=4153 wait=737 +13:31:33 step 163600 loss=3.2537 lr=2.60e-05 tok=21.580B dt=4216 wait=737 +13:32:37 step 163620 loss=3.2109 lr=2.60e-05 tok=21.582B dt=4280 wait=737 +13:33:40 step 163640 loss=3.0641 lr=2.60e-05 tok=21.585B dt=4343 wait=737 +13:34:26 pushed ckpt_w150.pt,log_w150.txt +13:35:27 waited 61s for global r=652 +13:35:37 global r=652 load 4.280 -> 4.278 +13:36:11 step 163660 loss=3.1473 lr=2.60e-05 tok=21.588B dt=4494 wait=738 +13:37:14 step 163680 loss=3.1439 lr=2.60e-05 tok=21.590B dt=4557 wait=738 +13:38:18 step 163700 loss=2.9909 lr=2.60e-05 tok=21.593B dt=4621 wait=738 +13:39:21 step 163720 loss=3.2190 lr=2.60e-05 tok=21.595B dt=4684 wait=738 +13:40:24 step 163740 loss=3.0715 lr=2.59e-05 tok=21.598B dt=4747 wait=738 +13:41:27 step 163760 loss=3.1759 lr=2.59e-05 tok=21.601B dt=4810 wait=738 +13:42:30 step 163780 loss=3.1153 lr=2.59e-05 tok=21.603B dt=4873 wait=739 +13:43:34 step 163800 loss=3.1237 lr=2.59e-05 tok=21.606B dt=4937 wait=739 +13:43:48 pushed ckpt_w150.pt,log_w150.txt +13:44:49 waited 61s for global r=653 +13:44:56 global r=653 load 4.277 -> 4.276 +13:46:02 step 163820 loss=3.0938 lr=2.59e-05 tok=21.609B dt=5085 wait=739 +13:47:05 step 163840 loss=3.0897 lr=2.59e-05 tok=21.611B dt=5148 wait=739 +13:48:09 step 163860 loss=3.1234 lr=2.59e-05 tok=21.614B dt=5212 wait=739 +13:49:13 step 163880 loss=3.0547 lr=2.59e-05 tok=21.616B dt=5276 wait=740 +13:50:16 step 163900 loss=3.1412 lr=2.59e-05 tok=21.619B dt=5339 wait=740 +13:51:19 step 163920 loss=3.1395 lr=2.58e-05 tok=21.622B dt=5402 wait=740 +13:52:22 step 163940 loss=3.1558 lr=2.58e-05 tok=21.624B dt=5465 wait=740 +13:53:08 pushed ckpt_w150.pt,log_w150.txt +13:56:09 waited 181s for global r=654 +13:56:19 global r=654 load 4.285 -> 4.277 +13:56:52 step 163960 loss=3.1584 lr=2.58e-05 tok=21.627B dt=5735 wait=740 +13:57:56 step 163980 loss=3.0122 lr=2.58e-05 tok=21.630B dt=5799 wait=741 +13:58:59 step 164000 loss=3.1466 lr=2.58e-05 tok=21.632B dt=5862 wait=741 +14:00:03 step 164020 loss=3.2424 lr=2.58e-05 tok=21.635B dt=5926 wait=741 +14:01:06 step 164040 loss=3.3168 lr=2.58e-05 tok=21.637B dt=5989 wait=741 +14:02:11 step 164060 loss=3.4124 lr=2.58e-05 tok=21.640B dt=6054 wait=741 +14:03:15 step 164080 loss=3.2231 lr=2.57e-05 tok=21.643B dt=6118 wait=742 +14:04:19 step 164100 loss=3.2750 lr=2.57e-05 tok=21.645B dt=6182 wait=742 +14:04:31 pushed ckpt_w150.pt,log_w150.txt +14:06:02 waited 91s for global r=655 +14:06:10 global r=655 load 4.286 -> 4.278 +14:07:15 step 164120 loss=3.1043 lr=2.57e-05 tok=21.648B dt=6358 wait=742 +14:08:18 step 164140 loss=3.2110 lr=2.57e-05 tok=21.650B dt=6421 wait=742 +14:09:21 step 164160 loss=3.1809 lr=2.57e-05 tok=21.653B dt=6484 wait=742 +14:10:23 step 164180 loss=3.2032 lr=2.57e-05 tok=21.656B dt=6546 wait=743 +14:11:26 step 164200 loss=3.1098 lr=2.57e-05 tok=21.658B dt=6609 wait=743 +14:12:29 step 164220 loss=3.1795 lr=2.57e-05 tok=21.661B dt=6672 wait=743 +14:13:32 step 164240 loss=3.0578 lr=2.57e-05 tok=21.664B dt=6735 wait=743 +14:14:18 pushed ckpt_w150.pt,log_w150.txt +14:15:51 waited 93s for global r=656 +14:16:06 global r=656 load 4.285 -> 4.279 +14:16:38 step 164260 loss=3.2551 lr=2.56e-05 tok=21.666B dt=6921 wait=743 +14:17:41 step 164280 loss=3.1493 lr=2.56e-05 tok=21.669B dt=6984 wait=744 +14:18:44 step 164300 loss=3.1335 lr=2.56e-05 tok=21.671B dt=7047 wait=744 +14:19:47 step 164320 loss=3.2782 lr=2.56e-05 tok=21.674B dt=7110 wait=744 +14:20:50 step 164340 loss=3.1059 lr=2.56e-05 tok=21.677B dt=7173 wait=744 +14:21:52 step 164360 loss=3.2772 lr=2.56e-05 tok=21.679B dt=7235 wait=744 +14:22:54 step 164380 loss=3.1924 lr=2.56e-05 tok=21.682B dt=7297 wait=744 +14:23:57 step 164400 loss=3.1974 lr=2.56e-05 tok=21.685B dt=7360 wait=745 +14:24:08 pushed ckpt_w150.pt,log_w150.txt +14:25:31 waited 83s for global r=657 +14:25:41 global r=657 load 4.287 -> 4.276 +14:26:45 step 164420 loss=3.1106 lr=2.56e-05 tok=21.687B dt=7528 wait=745 +14:27:48 step 164440 loss=3.2160 lr=2.55e-05 tok=21.690B dt=7591 wait=745 +14:28:51 step 164460 loss=3.1137 lr=2.55e-05 tok=21.692B dt=7654 wait=745 +14:29:53 step 164480 loss=3.1040 lr=2.55e-05 tok=21.695B dt=7716 wait=745 +14:30:56 step 164500 loss=3.0655 lr=2.55e-05 tok=21.698B dt=7779 wait=746 +14:31:59 step 164520 loss=3.0673 lr=2.55e-05 tok=21.700B dt=7842 wait=746 +14:33:02 step 164540 loss=3.0667 lr=2.55e-05 tok=21.703B dt=7905 wait=746 +14:33:46 pushed ckpt_w150.pt,log_w150.txt +14:35:02 waited 76s for global r=658 +14:35:10 global r=658 load 4.281 -> 4.278 +14:35:43 step 164560 loss=3.2618 lr=2.55e-05 tok=21.706B dt=8066 wait=746 +14:36:46 step 164580 loss=3.1901 lr=2.55e-05 tok=21.708B dt=8129 wait=746 +14:37:49 step 164600 loss=3.0411 lr=2.55e-05 tok=21.711B dt=8192 wait=747 +14:38:52 step 164620 loss=3.1892 lr=2.54e-05 tok=21.713B dt=8255 wait=747 +14:39:54 step 164640 loss=3.0828 lr=2.54e-05 tok=21.716B dt=8317 wait=747 +14:40:57 step 164660 loss=3.0775 lr=2.54e-05 tok=21.719B dt=8380 wait=747 +14:42:01 step 164680 loss=2.9371 lr=2.54e-05 tok=21.721B dt=8444 wait=747 +14:43:03 step 164700 loss=3.1714 lr=2.54e-05 tok=21.724B dt=8506 wait=747 +14:43:17 pushed ckpt_w150.pt,log_w150.txt +14:44:48 waited 91s for global r=659 +14:44:54 global r=659 load 4.274 -> 4.277 +14:45:58 step 164720 loss=3.1880 lr=2.54e-05 tok=21.726B dt=8681 wait=748 +14:47:01 step 164740 loss=3.1723 lr=2.54e-05 tok=21.729B dt=8744 wait=748 +14:48:04 step 164760 loss=3.2596 lr=2.54e-05 tok=21.732B dt=8807 wait=748 +14:49:07 step 164780 loss=3.2182 lr=2.53e-05 tok=21.734B dt=8870 wait=748 +14:50:10 step 164800 loss=3.1266 lr=2.53e-05 tok=21.737B dt=8933 wait=748 +14:51:14 step 164820 loss=3.1844 lr=2.53e-05 tok=21.740B dt=8997 wait=749 +14:52:17 step 164840 loss=3.1024 lr=2.53e-05 tok=21.742B dt=9060 wait=749 +14:53:10 pushed ckpt_w150.pt,log_w150.txt +14:54:42 waited 92s for global r=660 +14:54:51 global r=660 load 4.267 -> 4.275 +14:55:24 step 164860 loss=3.2889 lr=2.53e-05 tok=21.745B dt=9247 wait=749 +14:56:27 step 164880 loss=3.1543 lr=2.53e-05 tok=21.747B dt=9310 wait=749 +14:57:30 step 164900 loss=3.2463 lr=2.53e-05 tok=21.750B dt=9373 wait=749 +14:58:33 step 164920 loss=3.2146 lr=2.53e-05 tok=21.753B dt=9436 wait=749 +14:59:36 step 164940 loss=3.1469 lr=2.53e-05 tok=21.755B dt=9499 wait=750 +15:00:40 step 164960 loss=3.2201 lr=2.52e-05 tok=21.758B dt=9563 wait=750 +15:01:43 step 164980 loss=3.2226 lr=2.52e-05 tok=21.761B dt=9626 wait=750 +15:02:47 step 165000 loss=3.2291 lr=2.52e-05 tok=21.763B dt=9690 wait=750 diff --git a/log_w151.txt b/log_w151.txt new file mode 100644 index 0000000000000000000000000000000000000000..d921a92fc226fec8ffd01a717b36545ca1cca6de --- /dev/null +++ b/log_w151.txt @@ -0,0 +1,1111 @@ +launch +12:28:56 name=w151 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 98/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:29:03 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:29:03 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:29:12 params 1.08B +12:29:14 resumed step=162750 seen=21.46B round=646 +12:29:14 stream salt=162750 +12:33:45 blocks compiled +12:33:45 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:40:21 step 162751 loss=2.9268 lr=2.65e-05 tok=21.462B dt=396 wait=366 +12:40:36 step 162752 loss=3.0707 lr=2.65e-05 tok=21.462B dt=411 wait=366 +12:41:02 step 162760 loss=3.4344 lr=2.65e-05 tok=21.463B dt=437 wait=366 +12:42:06 step 162780 loss=3.1019 lr=2.65e-05 tok=21.466B dt=501 wait=366 +12:43:10 step 162800 loss=3.1545 lr=2.65e-05 tok=21.468B dt=565 wait=367 +12:44:14 step 162820 loss=3.1622 lr=2.65e-05 tok=21.471B dt=629 wait=367 +12:45:18 step 162840 loss=3.1917 lr=2.65e-05 tok=21.474B dt=692 wait=367 +12:46:22 step 162860 loss=3.2495 lr=2.65e-05 tok=21.476B dt=757 wait=367 +12:47:26 step 162880 loss=3.0869 lr=2.64e-05 tok=21.479B dt=821 wait=367 +12:48:17 global r=647 load 4.290 -> 4.281 +12:49:20 step 162920 loss=3.2187 lr=2.64e-05 tok=21.483B dt=935 wait=368 +12:50:25 step 162940 loss=3.1787 lr=2.64e-05 tok=21.485B dt=999 wait=368 +12:51:29 step 162960 loss=3.2552 lr=2.64e-05 tok=21.488B dt=1064 wait=368 +12:52:33 step 162980 loss=3.0467 lr=2.64e-05 tok=21.491B dt=1128 wait=368 +12:53:37 step 163000 loss=3.0201 lr=2.64e-05 tok=21.493B dt=1192 wait=369 +12:54:41 step 163020 loss=3.0912 lr=2.64e-05 tok=21.496B dt=1256 wait=369 +12:55:45 step 163040 loss=3.0122 lr=2.64e-05 tok=21.498B dt=1320 wait=369 +12:56:32 pushed ckpt_w151.pt,log_w151.txt +12:57:32 waited 60s for global r=648 +12:57:38 global r=648 load 4.287 -> 4.281 +12:58:12 step 163060 loss=3.3040 lr=2.63e-05 tok=21.501B dt=1466 wait=369 +12:59:15 step 163080 loss=3.1329 lr=2.63e-05 tok=21.504B dt=1530 wait=369 +13:00:18 step 163100 loss=3.2039 lr=2.63e-05 tok=21.506B dt=1593 wait=370 +13:01:22 step 163120 loss=3.1114 lr=2.63e-05 tok=21.509B dt=1657 wait=370 +13:02:26 step 163140 loss=3.1786 lr=2.63e-05 tok=21.512B dt=1720 wait=370 +13:03:28 step 163160 loss=3.2658 lr=2.63e-05 tok=21.514B dt=1783 wait=370 +13:04:32 step 163180 loss=3.2174 lr=2.63e-05 tok=21.517B dt=1847 wait=370 +13:05:36 step 163200 loss=3.1752 lr=2.63e-05 tok=21.519B dt=1910 wait=370 +13:05:49 pushed ckpt_w151.pt,log_w151.txt +13:07:05 waited 76s for global r=649 +13:07:12 global r=649 load 4.280 -> 4.278 +13:08:16 step 163220 loss=3.1845 lr=2.62e-05 tok=21.522B dt=2071 wait=371 +13:09:20 step 163240 loss=3.1480 lr=2.62e-05 tok=21.525B dt=2135 wait=371 +13:10:25 step 163260 loss=3.1268 lr=2.62e-05 tok=21.527B dt=2199 wait=371 +13:11:29 step 163280 loss=3.1734 lr=2.62e-05 tok=21.530B dt=2264 wait=371 +13:12:34 step 163300 loss=3.2718 lr=2.62e-05 tok=21.533B dt=2328 wait=371 +13:13:38 step 163320 loss=3.2513 lr=2.62e-05 tok=21.535B dt=2393 wait=372 +13:14:43 step 163340 loss=3.1265 lr=2.62e-05 tok=21.538B dt=2458 wait=372 +13:15:29 pushed ckpt_w151.pt,log_w151.txt +13:16:29 waited 61s for global r=650 +13:16:40 global r=650 load 4.284 -> 4.282 +13:17:14 step 163360 loss=3.3422 lr=2.62e-05 tok=21.540B dt=2608 wait=372 +13:18:18 step 163380 loss=3.3862 lr=2.62e-05 tok=21.543B dt=2673 wait=372 +13:19:22 step 163400 loss=3.2109 lr=2.61e-05 tok=21.546B dt=2737 wait=372 +13:20:26 step 163420 loss=3.3870 lr=2.61e-05 tok=21.548B dt=2801 wait=372 +13:21:30 step 163440 loss=3.2505 lr=2.61e-05 tok=21.551B dt=2864 wait=373 +13:22:34 step 163460 loss=3.4170 lr=2.61e-05 tok=21.553B dt=2929 wait=373 +13:23:38 step 163480 loss=3.2395 lr=2.61e-05 tok=21.556B dt=2993 wait=373 +13:24:42 step 163500 loss=3.2800 lr=2.61e-05 tok=21.559B dt=3057 wait=373 +13:24:58 pushed ckpt_w151.pt,log_w151.txt +13:25:58 waited 61s for global r=651 +13:26:07 global r=651 load 4.284 -> 4.282 +13:27:12 step 163520 loss=3.3000 lr=2.61e-05 tok=21.561B dt=3207 wait=373 +13:28:16 step 163540 loss=3.3292 lr=2.61e-05 tok=21.564B dt=3270 wait=374 +13:29:20 step 163560 loss=3.3012 lr=2.60e-05 tok=21.567B dt=3334 wait=374 +13:30:24 step 163580 loss=3.2072 lr=2.60e-05 tok=21.569B dt=3398 wait=374 +13:31:27 step 163600 loss=3.1693 lr=2.60e-05 tok=21.572B dt=3462 wait=374 +13:32:31 step 163620 loss=3.2853 lr=2.60e-05 tok=21.574B dt=3526 wait=374 +13:33:35 step 163640 loss=3.1102 lr=2.60e-05 tok=21.577B dt=3589 wait=375 +13:34:18 pushed ckpt_w151.pt,log_w151.txt +13:35:34 waited 76s for global r=652 +13:35:39 global r=652 load 4.287 -> 4.278 +13:36:13 step 163660 loss=3.2599 lr=2.60e-05 tok=21.580B dt=3747 wait=375 +13:37:16 step 163680 loss=3.0481 lr=2.60e-05 tok=21.582B dt=3811 wait=375 +13:38:19 step 163700 loss=3.2017 lr=2.60e-05 tok=21.585B dt=3874 wait=375 +13:39:24 step 163720 loss=3.0704 lr=2.60e-05 tok=21.588B dt=3938 wait=375 +13:40:28 step 163740 loss=3.2011 lr=2.59e-05 tok=21.590B dt=4002 wait=375 +13:41:32 step 163760 loss=3.3111 lr=2.59e-05 tok=21.593B dt=4067 wait=376 +13:42:36 step 163780 loss=3.3032 lr=2.59e-05 tok=21.595B dt=4130 wait=376 +13:43:40 step 163800 loss=3.2438 lr=2.59e-05 tok=21.598B dt=4195 wait=376 +13:43:51 pushed ckpt_w151.pt,log_w151.txt +13:44:52 waited 60s for global r=653 +13:45:00 global r=653 load 4.276 -> 4.276 +13:46:05 step 163820 loss=3.2126 lr=2.59e-05 tok=21.601B dt=4340 wait=376 +13:47:08 step 163840 loss=3.2008 lr=2.59e-05 tok=21.603B dt=4403 wait=376 +13:48:12 step 163860 loss=3.2492 lr=2.59e-05 tok=21.606B dt=4467 wait=377 +13:49:16 step 163880 loss=3.1736 lr=2.59e-05 tok=21.609B dt=4530 wait=377 +13:50:19 step 163900 loss=3.1720 lr=2.59e-05 tok=21.611B dt=4594 wait=377 +13:51:24 step 163920 loss=3.1856 lr=2.58e-05 tok=21.614B dt=4658 wait=377 +13:52:27 step 163940 loss=3.1858 lr=2.58e-05 tok=21.616B dt=4722 wait=378 +13:53:11 pushed ckpt_w151.pt,log_w151.txt +13:56:12 waited 181s for global r=654 +13:56:21 global r=654 load 4.275 -> 4.277 +13:56:54 step 163960 loss=3.2122 lr=2.58e-05 tok=21.619B dt=4988 wait=378 +13:57:58 step 163980 loss=3.1480 lr=2.58e-05 tok=21.622B dt=5053 wait=378 +13:59:01 step 164000 loss=3.1514 lr=2.58e-05 tok=21.624B dt=5115 wait=378 +14:00:04 step 164020 loss=3.1830 lr=2.58e-05 tok=21.627B dt=5179 wait=378 +14:01:08 step 164040 loss=3.0993 lr=2.58e-05 tok=21.630B dt=5243 wait=379 +14:02:11 step 164060 loss=3.1721 lr=2.58e-05 tok=21.632B dt=5306 wait=379 +14:03:14 step 164080 loss=3.1895 lr=2.57e-05 tok=21.635B dt=5369 wait=379 +14:04:17 step 164100 loss=3.1864 lr=2.57e-05 tok=21.637B dt=5432 wait=379 +14:04:29 pushed ckpt_w151.pt,log_w151.txt +14:06:15 waited 106s for global r=655 +14:06:21 global r=655 load 4.277 -> 4.278 +14:07:26 step 164120 loss=3.2296 lr=2.57e-05 tok=21.640B dt=5621 wait=379 +14:08:30 step 164140 loss=3.2063 lr=2.57e-05 tok=21.643B dt=5685 wait=379 +14:09:34 step 164160 loss=3.2711 lr=2.57e-05 tok=21.645B dt=5749 wait=380 +14:10:38 step 164180 loss=3.2017 lr=2.57e-05 tok=21.648B dt=5813 wait=380 +14:11:43 step 164200 loss=3.1926 lr=2.57e-05 tok=21.650B dt=5877 wait=380 +14:12:47 step 164220 loss=3.3308 lr=2.57e-05 tok=21.653B dt=5942 wait=380 +14:13:51 step 164240 loss=3.3420 lr=2.57e-05 tok=21.656B dt=6006 wait=381 +14:14:43 pushed ckpt_w151.pt,log_w151.txt +14:15:58 waited 76s for global r=656 +14:16:06 global r=656 load 4.277 -> 4.279 +14:16:40 step 164260 loss=3.3411 lr=2.56e-05 tok=21.658B dt=6174 wait=381 +14:17:44 step 164280 loss=3.3386 lr=2.56e-05 tok=21.661B dt=6239 wait=381 +14:18:48 step 164300 loss=3.2969 lr=2.56e-05 tok=21.664B dt=6303 wait=381 +14:19:52 step 164320 loss=3.3447 lr=2.56e-05 tok=21.666B dt=6367 wait=381 +14:20:57 step 164340 loss=3.2915 lr=2.56e-05 tok=21.669B dt=6432 wait=381 +14:22:02 step 164360 loss=3.2986 lr=2.56e-05 tok=21.671B dt=6496 wait=382 +14:23:05 step 164380 loss=3.2303 lr=2.56e-05 tok=21.674B dt=6560 wait=382 +14:24:10 step 164400 loss=3.1298 lr=2.56e-05 tok=21.677B dt=6625 wait=382 +14:24:25 pushed ckpt_w151.pt,log_w151.txt +14:25:25 waited 61s for global r=657 +14:25:33 global r=657 load 4.276 -> 4.276 +14:26:38 step 164420 loss=3.2785 lr=2.56e-05 tok=21.679B dt=6773 wait=382 +14:27:42 step 164440 loss=3.1687 lr=2.55e-05 tok=21.682B dt=6837 wait=382 +14:28:46 step 164460 loss=3.1952 lr=2.55e-05 tok=21.685B dt=6900 wait=383 +14:29:50 step 164480 loss=3.1949 lr=2.55e-05 tok=21.687B dt=6964 wait=383 +14:30:54 step 164500 loss=3.1738 lr=2.55e-05 tok=21.690B dt=7029 wait=383 +14:31:57 step 164520 loss=3.1975 lr=2.55e-05 tok=21.692B dt=7092 wait=383 +14:33:00 step 164540 loss=3.0831 lr=2.55e-05 tok=21.695B dt=7155 wait=383 +14:33:44 pushed ckpt_w151.pt,log_w151.txt +14:35:15 waited 91s for global r=658 +14:35:22 global r=658 load 4.276 -> 4.278 +14:35:55 step 164560 loss=3.1756 lr=2.55e-05 tok=21.698B dt=7329 wait=384 +14:36:58 step 164580 loss=3.0630 lr=2.55e-05 tok=21.700B dt=7393 wait=384 +14:38:01 step 164600 loss=3.0607 lr=2.55e-05 tok=21.703B dt=7456 wait=384 +14:39:05 step 164620 loss=3.1384 lr=2.54e-05 tok=21.706B dt=7519 wait=384 +14:40:08 step 164640 loss=3.1232 lr=2.54e-05 tok=21.708B dt=7583 wait=384 +14:41:12 step 164660 loss=3.0937 lr=2.54e-05 tok=21.711B dt=7646 wait=385 +14:42:15 step 164680 loss=3.2574 lr=2.54e-05 tok=21.713B dt=7709 wait=385 +14:43:18 step 164700 loss=3.2035 lr=2.54e-05 tok=21.716B dt=7772 wait=385 +14:43:34 pushed ckpt_w151.pt,log_w151.txt +14:44:50 waited 76s for global r=659 +14:44:59 global r=659 load 4.279 -> 4.277 +14:46:04 step 164720 loss=3.1091 lr=2.54e-05 tok=21.719B dt=7938 wait=385 +14:47:07 step 164740 loss=3.1904 lr=2.54e-05 tok=21.721B dt=8001 wait=385 +14:48:10 step 164760 loss=3.2573 lr=2.54e-05 tok=21.724B dt=8065 wait=386 +14:49:14 step 164780 loss=3.1578 lr=2.53e-05 tok=21.726B dt=8128 wait=386 +14:50:17 step 164800 loss=3.2272 lr=2.53e-05 tok=21.729B dt=8192 wait=386 +14:51:20 step 164820 loss=3.2964 lr=2.53e-05 tok=21.732B dt=8255 wait=386 +14:52:23 step 164840 loss=3.0891 lr=2.53e-05 tok=21.734B dt=8318 wait=386 +14:53:10 pushed ckpt_w151.pt,log_w151.txt +14:54:40 waited 91s for global r=660 +14:54:48 global r=660 load 4.276 -> 4.275 +14:55:21 step 164860 loss=3.1872 lr=2.53e-05 tok=21.737B dt=8496 wait=387 +14:56:25 step 164880 loss=3.0831 lr=2.53e-05 tok=21.740B dt=8559 wait=387 +14:57:28 step 164900 loss=3.1034 lr=2.53e-05 tok=21.742B dt=8623 wait=387 +14:58:32 step 164920 loss=3.1578 lr=2.53e-05 tok=21.745B dt=8686 wait=387 +14:59:35 step 164940 loss=3.0330 lr=2.53e-05 tok=21.747B dt=8750 wait=387 +15:00:39 step 164960 loss=3.2319 lr=2.52e-05 tok=21.750B dt=8813 wait=388 +15:01:42 step 164980 loss=3.0928 lr=2.52e-05 tok=21.753B dt=8876 wait=388 +15:02:45 step 165000 loss=3.1324 lr=2.52e-05 tok=21.755B dt=8940 wait=388 +15:02:57 pushed ckpt_w151.pt,log_w151.txt +15:04:43 waited 106s for global r=661 +15:04:51 global r=661 load 4.281 -> 4.275 +15:05:56 step 165020 loss=3.0794 lr=2.52e-05 tok=21.758B dt=9130 wait=388 +15:06:59 step 165040 loss=3.0848 lr=2.52e-05 tok=21.761B dt=9194 wait=388 +15:08:02 step 165060 loss=3.0725 lr=2.52e-05 tok=21.763B dt=9257 wait=389 +15:09:06 step 165080 loss=2.9531 lr=2.52e-05 tok=21.766B dt=9321 wait=389 +15:10:09 step 165100 loss=3.0929 lr=2.52e-05 tok=21.768B dt=9384 wait=389 +15:11:12 step 165120 loss=3.1822 lr=2.52e-05 tok=21.771B dt=9447 wait=389 +15:12:21 step 165140 loss=3.0462 lr=2.51e-05 tok=21.774B dt=9516 wait=389 +15:13:08 pushed ckpt_w151.pt,log_w151.txt +15:15:55 waited 167s for global r=662 +15:16:04 global r=662 load 4.280 -> 4.277 +15:16:37 step 165160 loss=3.1856 lr=2.51e-05 tok=21.776B dt=9772 wait=390 +15:17:44 step 165180 loss=3.1022 lr=2.51e-05 tok=21.779B dt=9838 wait=390 +15:18:47 step 165200 loss=3.0935 lr=2.51e-05 tok=21.782B dt=9901 wait=390 +15:19:51 step 165220 loss=3.1326 lr=2.51e-05 tok=21.784B dt=9965 wait=390 +15:20:54 step 165240 loss=3.0743 lr=2.51e-05 tok=21.787B dt=10029 wait=390 +15:21:58 step 165260 loss=3.1142 lr=2.51e-05 tok=21.789B dt=10092 wait=391 +15:23:01 step 165280 loss=3.0761 lr=2.51e-05 tok=21.792B dt=10156 wait=391 +15:24:05 step 165300 loss=3.1189 lr=2.51e-05 tok=21.795B dt=10219 wait=391 +15:24:17 pushed ckpt_w151.pt,log_w151.txt +15:25:33 waited 76s for global r=663 +15:25:42 global r=663 load 4.280 -> 4.276 +15:26:46 step 165320 loss=3.0525 lr=2.50e-05 tok=21.797B dt=10381 wait=391 +15:27:49 step 165340 loss=3.0838 lr=2.50e-05 tok=21.800B dt=10444 wait=391 +15:28:52 step 165360 loss=2.9882 lr=2.50e-05 tok=21.803B dt=10506 wait=392 +15:29:54 step 165380 loss=3.0464 lr=2.50e-05 tok=21.805B dt=10569 wait=392 +15:30:58 step 165400 loss=3.0051 lr=2.50e-05 tok=21.808B dt=10632 wait=392 +15:32:00 step 165420 loss=3.1091 lr=2.50e-05 tok=21.810B dt=10695 wait=392 +15:33:03 step 165440 loss=3.0079 lr=2.50e-05 tok=21.813B dt=10758 wait=392 +15:33:50 pushed ckpt_w151.pt,log_w151.txt +15:35:06 waited 76s for global r=664 +15:35:16 global r=664 load 4.279 -> 4.273 +15:35:49 step 165460 loss=3.0667 lr=2.50e-05 tok=21.816B dt=10924 wait=393 +15:36:52 step 165480 loss=3.0157 lr=2.50e-05 tok=21.818B dt=10987 wait=393 +15:37:56 step 165500 loss=2.9097 lr=2.49e-05 tok=21.821B dt=11050 wait=393 +15:38:59 step 165520 loss=3.0376 lr=2.49e-05 tok=21.823B dt=11114 wait=393 +15:40:02 step 165540 loss=2.9903 lr=2.49e-05 tok=21.826B dt=11176 wait=393 +15:41:05 step 165560 loss=3.0780 lr=2.49e-05 tok=21.829B dt=11240 wait=394 +15:42:08 step 165580 loss=3.0375 lr=2.49e-05 tok=21.831B dt=11302 wait=394 +15:43:11 step 165600 loss=3.0348 lr=2.49e-05 tok=21.834B dt=11366 wait=394 +15:43:29 pushed ckpt_w151.pt,log_w151.txt +15:44:45 waited 76s for global r=665 +15:44:51 global r=665 load 4.284 -> 4.275 +15:45:56 step 165620 loss=2.9783 lr=2.49e-05 tok=21.837B dt=11530 wait=394 +15:46:59 step 165640 loss=3.0750 lr=2.49e-05 tok=21.839B dt=11593 wait=394 +15:48:02 step 165660 loss=3.0028 lr=2.49e-05 tok=21.842B dt=11657 wait=395 +15:49:05 step 165680 loss=2.9717 lr=2.48e-05 tok=21.844B dt=11720 wait=395 +15:50:09 step 165700 loss=3.0177 lr=2.48e-05 tok=21.847B dt=11783 wait=395 +15:51:13 step 165720 loss=3.0486 lr=2.48e-05 tok=21.850B dt=11847 wait=395 +15:52:15 step 165740 loss=2.9044 lr=2.48e-05 tok=21.852B dt=11910 wait=395 +15:53:03 pushed ckpt_w151.pt,log_w151.txt +15:54:04 waited 61s for global r=666 +15:54:16 global r=666 load 4.280 -> 4.276 +15:54:49 step 165760 loss=2.9375 lr=2.48e-05 tok=21.855B dt=12064 wait=396 +15:55:53 step 165780 loss=2.9565 lr=2.48e-05 tok=21.858B dt=12127 wait=396 +15:56:56 step 165800 loss=2.9942 lr=2.48e-05 tok=21.860B dt=12190 wait=396 +15:57:59 step 165820 loss=3.0507 lr=2.48e-05 tok=21.863B dt=12253 wait=396 +15:59:02 step 165840 loss=2.9535 lr=2.48e-05 tok=21.865B dt=12317 wait=396 +16:00:05 step 165860 loss=3.1065 lr=2.47e-05 tok=21.868B dt=12379 wait=396 +16:01:08 step 165880 loss=2.9882 lr=2.47e-05 tok=21.871B dt=12443 wait=397 +16:02:11 step 165900 loss=3.0146 lr=2.47e-05 tok=21.873B dt=12506 wait=397 +16:02:24 pushed ckpt_w151.pt,log_w151.txt +16:03:41 waited 77s for global r=667 +16:03:51 global r=667 load 4.278 -> 4.273 +16:04:57 step 165920 loss=2.9001 lr=2.47e-05 tok=21.876B dt=12671 wait=397 +16:06:01 step 165940 loss=2.9440 lr=2.47e-05 tok=21.879B dt=12735 wait=397 +16:07:05 step 165960 loss=2.9661 lr=2.47e-05 tok=21.881B dt=12799 wait=398 +16:08:08 step 165980 loss=3.0117 lr=2.47e-05 tok=21.884B dt=12862 wait=398 +16:09:11 step 166000 loss=2.9316 lr=2.47e-05 tok=21.886B dt=12926 wait=398 +16:10:15 step 166020 loss=3.0111 lr=2.47e-05 tok=21.889B dt=12990 wait=398 +16:11:20 step 166040 loss=2.8815 lr=2.47e-05 tok=21.892B dt=13055 wait=398 +16:12:11 pushed ckpt_w151.pt,log_w151.txt +16:15:12 waited 181s for global r=668 +16:15:20 global r=668 load 4.282 -> 4.276 +16:15:54 step 166060 loss=2.9450 lr=2.46e-05 tok=21.894B dt=13329 wait=399 +16:16:58 step 166080 loss=2.9106 lr=2.46e-05 tok=21.897B dt=13393 wait=399 +16:18:02 step 166100 loss=2.9514 lr=2.46e-05 tok=21.900B dt=13457 wait=399 +16:19:07 step 166120 loss=3.0556 lr=2.46e-05 tok=21.902B dt=13521 wait=399 +16:20:11 step 166140 loss=2.9134 lr=2.46e-05 tok=21.905B dt=13586 wait=399 +16:21:16 step 166160 loss=2.9900 lr=2.46e-05 tok=21.907B dt=13651 wait=400 +16:22:21 step 166180 loss=2.8809 lr=2.46e-05 tok=21.910B dt=13715 wait=400 +16:23:25 step 166200 loss=3.0224 lr=2.46e-05 tok=21.913B dt=13780 wait=400 +16:23:40 pushed ckpt_w151.pt,log_w151.txt +16:27:28 global r=669 load 4.278 -> 4.276 +16:28:32 step 166220 loss=2.9121 lr=2.46e-05 tok=21.917B dt=14087 wait=400 +16:29:36 step 166240 loss=3.0407 lr=2.45e-05 tok=21.919B dt=14151 wait=401 +16:30:40 step 166260 loss=3.0753 lr=2.45e-05 tok=21.922B dt=14214 wait=401 +16:31:43 step 166280 loss=3.0223 lr=2.45e-05 tok=21.924B dt=14278 wait=401 +16:32:47 step 166300 loss=2.9498 lr=2.45e-05 tok=21.927B dt=14342 wait=401 +16:33:51 step 166320 loss=3.0825 lr=2.45e-05 tok=21.930B dt=14405 wait=402 +16:34:54 step 166340 loss=2.9353 lr=2.45e-05 tok=21.932B dt=14469 wait=402 +16:35:44 pushed ckpt_w151.pt,log_w151.txt +16:38:46 waited 182s for global r=670 +16:38:54 global r=670 load 4.284 -> 4.282 +16:39:28 step 166360 loss=2.9167 lr=2.45e-05 tok=21.935B dt=14743 wait=402 +16:40:32 step 166380 loss=2.9222 lr=2.45e-05 tok=21.938B dt=14806 wait=402 +16:41:36 step 166400 loss=2.9707 lr=2.45e-05 tok=21.940B dt=14870 wait=402 +16:42:39 step 166420 loss=3.0912 lr=2.44e-05 tok=21.943B dt=14934 wait=402 +16:43:44 step 166440 loss=2.8686 lr=2.44e-05 tok=21.945B dt=14999 wait=403 +16:44:48 step 166460 loss=3.1178 lr=2.44e-05 tok=21.948B dt=15063 wait=403 +16:45:53 step 166480 loss=3.1330 lr=2.44e-05 tok=21.951B dt=15128 wait=403 +16:46:57 step 166500 loss=2.9827 lr=2.44e-05 tok=21.953B dt=15192 wait=403 +16:47:11 pushed ckpt_w151.pt,log_w151.txt +launch +16:48:17 name=w151 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 98/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:48:22 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:48:22 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:48:31 params 1.08B +16:48:33 resumed step=166350 seen=21.93B round=670 +16:48:33 stream salt=166350 +16:50:32 blocks compiled +16:50:32 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:54:36 step 166351 loss=3.0172 lr=2.45e-05 tok=21.935B dt=245 wait=236 +16:54:44 step 166352 loss=3.1433 lr=2.45e-05 tok=21.935B dt=252 wait=236 +16:55:10 step 166360 loss=3.5278 lr=2.45e-05 tok=21.936B dt=278 wait=236 +16:55:22 global r=671 load 4.280 -> 4.284 +16:56:27 step 166520 loss=3.4162 lr=2.44e-05 tok=21.939B dt=355 wait=236 +16:57:32 step 166540 loss=3.3235 lr=2.44e-05 tok=21.941B dt=420 wait=236 +16:58:36 step 166560 loss=3.3672 lr=2.44e-05 tok=21.944B dt=484 wait=236 +16:59:40 step 166580 loss=3.3029 lr=2.44e-05 tok=21.947B dt=548 wait=236 +17:00:45 step 166600 loss=3.4005 lr=2.43e-05 tok=21.949B dt=614 wait=236 +17:01:50 step 166620 loss=3.4647 lr=2.43e-05 tok=21.952B dt=678 wait=237 +17:02:54 step 166640 loss=3.2583 lr=2.43e-05 tok=21.955B dt=743 wait=237 +17:03:41 pushed ckpt_w151.pt,log_w151.txt +17:04:42 waited 61s for global r=672 +17:04:53 global r=672 load 4.278 -> 4.280 +17:05:28 step 166660 loss=3.4166 lr=2.43e-05 tok=21.957B dt=896 wait=237 +17:06:33 step 166680 loss=3.2994 lr=2.43e-05 tok=21.960B dt=961 wait=237 +17:07:37 step 166700 loss=3.3403 lr=2.43e-05 tok=21.962B dt=1025 wait=237 +17:08:42 step 166720 loss=3.4838 lr=2.43e-05 tok=21.965B dt=1090 wait=237 +17:09:09 dolma file skipped data/ingredient1-common_crawl-high-quality_20_home_and_hobbies/shard_00000573.jsonl.zst: (Request ID: Root=1-6abbf0b5-0d1a942b1fe5a78452a8462f;5ff5f1cb-775f-4085-ac1c-27b1cbc0940b) + +429 Too Many Requests: you +17:09:10 dolma file skipped data/ingredient1-common_crawl-high-quality_20_politics/shard_00000491.jsonl.zst: (Request ID: Root=1-6abbf0b6-60e6c5022f787ac03cab9249;01f27c9c-24fb-4631-8b34-a11c77bd3528) + +429 Too Many Requests: you +17:09:10 dolma file skipped data/ingredient1-dolmino-math/dolmino_math_tinyGSM-MIND_problem-solving_tiny_gsm_inline_part061.000000.jsonl.jsonl.zst: (Request ID: Root=1-6abbf0b6-6eb82b1322e4effb1a014e06;70cc6b15-c1be-467f-a1ff-71e52f564606) + +429 Too Many Requests: you +17:09:16 dolma file skipped data/ingredient1-common_crawl-high-quality_19_transportation/shard_00000209.jsonl.zst: (Request ID: Root=1-6abbf0bc-7ace59ee0a20361a4b1e9762;c06e673a-287a-4feb-b71d-f35ec5f6756e) + +429 Too Many Requests: you +17:09:46 step 166740 loss=3.2982 lr=2.43e-05 tok=21.968B dt=1154 wait=238 +17:10:02 dolma file skipped data/ingredient1-tinymath-mind/output_188.jsonl.zst: (Request ID: Root=1-6abbf0ea-578c95f013a8d5e647869b33;b440bdd7-f9c2-4c9a-bced-d3cf20fede3b) + +429 Too Many Requests: you +17:10:20 dolma file skipped data/ingredient1-common_crawl-high-quality_19_fashion_and_beauty/shard_00000537.jsonl.zst: (Request ID: Root=1-6abbf0fc-29e4ec531d951a141571898b;108d2621-06a3-4ab1-b6fc-d138a9774cae) + +429 Too Many Requests: you +17:10:50 step 166760 loss=3.3855 lr=2.43e-05 tok=21.970B dt=1218 wait=238 +17:11:53 step 166780 loss=3.3647 lr=2.42e-05 tok=21.973B dt=1281 wait=238 +17:12:57 step 166800 loss=3.3780 lr=2.42e-05 tok=21.976B dt=1345 wait=238 +17:13:09 pushed ckpt_w151.pt,log_w151.txt +17:16:55 global r=673 load 4.265 -> 4.268 +17:17:59 step 166820 loss=3.3395 lr=2.42e-05 tok=21.979B dt=1647 wait=238 +17:19:03 step 166840 loss=3.2620 lr=2.42e-05 tok=21.982B dt=1711 wait=238 +launch +17:19:47 name=w151 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 98/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:19:52 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:19:52 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:20:01 params 1.08B +17:20:03 resumed step=166800 seen=21.98B round=673 +17:20:03 stream salt=166800 +17:21:53 blocks compiled +17:21:53 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:23:02 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00119-of-00167.parquet: (Request ID: Root=1-6abbf3f6-4ade4e0b12b8d72266c4a05b;5c409962-55ce-428c-9486-a22f3cd63816) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00061-of-00137.parquet: (Request ID: Root=1-6abbf3f6-2547a912638678855ea8885e;d20f782d-ec4a-4e9f-afbd-4342106f2288) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/cs/UltraData-Code-L3-cs-part-00015-of-00028.parquet: (Request ID: Root=1-6abbf3f6-49c52e806ad1d8466b7f233a;622ecd1e-e3b2-4edb-a43d-83add9699b27) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00146-of-00167.parquet: (Request ID: Root=1-6abbf3f6-2ca90a76776fbece5d9d27ea;4bfd6dab-0715-4a4a-8eff-7c2a4e9e605e) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/go/UltraData-Code-L3-go-part-00006-of-00012.parquet: (Request ID: Root=1-6abbf3f6-0961f08101380b92177c7c72;21ea1773-3e24-4ef8-bf7a-1eb877080580) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00123-of-00137.parquet: (Request ID: Root=1-6abbf3f6-3ba6c3191d848fdb5030f833;33c2fc0b-6474-4524-8134-975f4f9af52e) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00008-of-00147.parquet: (Request ID: Root=1-6abbf3f6-68f3d4d844c518aa6216aa59;b1f5b907-0000-45ba-b3b7-e0a869aa509f) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/js/UltraData-Code-L3-js-part-00014-of-00029.parquet: (Request ID: Root=1-6abbf3f6-7f60c08b6cf82e9b7594e78f;c60cd568-bad8-4c04-aae1-c11f1141d7dc) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00049-of-00147.parquet: (Request ID: Root=1-6abbf3f6-144bb3b0029078210018fbec;60f15865-4e98-4f0d-97e0-5df44dec286c) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00026-of-00147.parquet: (Request ID: Root=1-6abbf3f6-15a7c5b019cdfff76e1dbb7f;c2c252c4-6021-4893-9fa0-bd5029543c77) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00018-of-00137.parquet: (Request ID: Root=1-6abbf3f6-7b963912513ac754414b7144;be1ea154-22df-45c6-89cb-3253558eb1ce) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00009-of-00137.parquet: (Request ID: Root=1-6abbf3f6-5bf0ddb029e6c00a5dab0071;7355182c-41f1-4e8c-8763-8f110b1c6524) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00050-of-00167.parquet: (Request ID: Root=1-6abbf3f6-1fd7b6f8232864910c2ac4bf;4a32b9de-41ef-41c1-9463-a1628270ad9f) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00027-of-00167.parquet: (Request ID: Root=1-6abbf3f7-21c478af1b215d594d728be7;5b521537-358f-4528-8fae-6716924e7d37) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00092-of-00137.parquet: (Request ID: Root=1-6abbf3f7-3558fc1b6850889157ff7ea0;f34a941b-7166-4fec-ac31-7dd543ecd7a8) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00018-of-00167.parquet: (Request ID: Root=1-6abbf3f7-5a3524af305c370f4f2a6e08;3754e8cf-f8ac-4162-b1ce-6d96566944c1) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00161-of-00167.parquet: (Request ID: Root=1-6abbf3f7-4ccec844650b16f2004b2232;1f34eae9-0c47-4323-a7b1-8991926f3c53) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00121-of-00167.parquet: (Request ID: Root=1-6abbf3f7-55ad573a5c85df3b261e6dd4;758ff71d-45bb-4e54-b217-21bccd493cad) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00132-of-00137.parquet: (Request ID: Root=1-6abbf3f7-61df542d64c4fabd4964046d;031cdca2-5336-4fea-81cc-e2b7c9f045e4) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/php/UltraData-Code-L3-php-part-00013-of-00024.parquet: (Request ID: Root=1-6abbf3f7-168ec29125f7e37230e14df0;80d38041-09b0-45a3-8b7a-dfd82657a089) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00079-of-00167.parquet: (Request ID: Root=1-6abbf3f7-69c22b07690a198040ade0cd;56ccd080-2b3a-45ad-80aa-6a040317db79) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00110-of-00147.parquet: (Request ID: Root=1-6abbf3f7-00bf679c37e65b5744daaa41;5cbdc7ec-d4b2-420b-a530-ceddfc997730) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00150-of-00167.parquet: (Request ID: Root=1-6abbf3f7-29d7382c65c39fc12911bcdf;02f5b53b-f11b-4529-afe7-1c64cbe4bdef) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00095-of-00137.parquet: (Request ID: Root=1-6abbf3f7-03e527a87ec12a9c0f42db77;4f2968b6-d023-40c3-9517-f5b2ab12a861) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00149-of-00167.parquet: (Request ID: Root=1-6abbf3f7-62861bde3b4294093b35c9c1;d7972f75-6750-43a5-aaee-4b7dff6fbcb7) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00088-of-00137.parquet: (Request ID: Root=1-6abbf3f7-3f28176b283a7bc42690b2e5;47e7d3d3-8f2b-4e28-b3c0-b6e3fdee43e4) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00035-of-00147.parquet: (Request ID: Root=1-6abbf3f7-2415c5fa543215a328f68f87;95761b8c-372d-4777-8148-cbdfc56c6376) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00133-of-00147.parquet: (Request ID: Root=1-6abbf3f7-4fbbfe9134bda8966c046f2d;02c15aca-84e6-490b-ad2c-c59fb6d16239) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cs/UltraData-Code-L3-cs-part-00008-of-00028.parquet: (Request ID: Root=1-6abbf3f7-36b3080f0c64016a53d31a24;ce028f83-6032-4531-a624-c164439a863d) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00048-of-00167.parquet: (Request ID: Root=1-6abbf3f7-314f1619486526043eb449e5;fdf55f91-ae13-4648-89ef-b6c7f0e17e12) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00099-of-00167.parquet: (Request ID: Root=1-6abbf3f7-4e5a1095528b60e870e680b9;614c58dc-c798-4df2-856c-f60bde2b90e6) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00001-of-00167.parquet: (Request ID: Root=1-6abbf3f7-1e456ce652f976a95a6ffa5f;1b03b5f0-b235-4e64-958d-833014d25da1) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00078-of-00167.parquet: (Request ID: Root=1-6abbf3f7-7f3b825379d07f4d0022f891;fd5335d5-3b02-43d6-8c21-27b718461d34) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00116-of-00167.parquet: (Request ID: Root=1-6abbf3f7-0e8c8a1c4eb1afe86a9cb82b;0e870fb0-f35d-4b3e-9880-329c246315c9) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00066-of-00137.parquet: (Request ID: Root=1-6abbf3f7-310554d6550267b20dca3353;edc6ca9c-55a7-4c73-9e69-6722898a1deb) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00010-of-00147.parquet: (Request ID: Root=1-6abbf3f7-21fc297c6a0d2b5d5c7baac9;cfdb03e7-883f-485d-a9d1-58618136dbda) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00138-of-00167.parquet: (Request ID: Root=1-6abbf3f7-1d75ce723e2827a12a59600e;6b030834-d70a-49f6-a594-f395ee0b9603) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00031-of-00147.parquet: (Request ID: Root=1-6abbf3f7-728b9eb82b6ac54b75bf3be3;056e1c59-f4a7-47dd-9a7c-f305a283a4de) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00102-of-00137.parquet: (Request ID: Root=1-6abbf3f7-791d264d2a44d88f707b8003;43a7d510-ae90-4d30-9bc6-13e8688720d7) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00059-of-00137.parquet: (Request ID: Root=1-6abbf3f7-16a7df7037b28f3823a341ee;326b838b-016c-431e-bac1-821253049371) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00062-of-00147.parquet: (Request ID: Root=1-6abbf3f7-2a8ecc5b3538fe703d96cf79;2d0902b3-2bb4-4247-959d-51327630fa4a) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/r/UltraData-Code-L3-r-part-00003-of-00003.parquet: (Request ID: Root=1-6abbf3f7-5a9c53cd4247e247723f8bb7;5c967912-cc3b-4a8a-8594-40ff0753a9a2) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00155-of-00167.parquet: (Request ID: Root=1-6abbf3f7-70ed6d96173b0b786eae3c5c;47c4c158-3268-457b-bd70-dcf8f9dae410) + +429 Too Many Requests: you +17:23:03 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00134-of-00167.parquet: (Request ID: Root=1-6abbf3f7-7214c3c767168b2826b5151e;47d5d7b2-46b4-4fae-b7c2-dbf6e3d646af) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00044-of-00167.parquet: (Request ID: Root=1-6abbf3f7-23487f18308eafeb2e5f11a1;88369cb5-2dcd-458c-97f4-0211acbab0e4) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/js/UltraData-Code-L3-js-part-00010-of-00029.parquet: (Request ID: Root=1-6abbf3f8-22acbbf07d0d31493f3c4c05;d0e10de5-dc69-468b-aa4d-909ab879604d) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00034-of-00167.parquet: (Request ID: Root=1-6abbf3f8-7c8cc4bc5d78403a028c75ca;0979ab1f-ef16-40c2-a052-68a2364d602d) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00117-of-00137.parquet: (Request ID: Root=1-6abbf3f8-22a5b9d55ee0ebeb019f4a1f;e50de2a6-fa12-48c3-9e4f-310de061ba89) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00093-of-00167.parquet: (Request ID: Root=1-6abbf3f8-3a6e4dec2ef7de38712ec616;9fb12ea1-33c3-468a-8d4b-8b59d84ae603) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00045-of-00147.parquet: (Request ID: Root=1-6abbf3f8-45bb190d45f4e0eb7ea321eb;1fdc1685-0d11-4f6f-bdd1-6e536eb2b939) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00111-of-00167.parquet: (Request ID: Root=1-6abbf3f8-3c25dbc171ead5f04f315e0d;dba5a2d9-e1a7-4512-9b34-10b66e56bd2d) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/js/UltraData-Code-L3-js-part-00013-of-00029.parquet: (Request ID: Root=1-6abbf3f8-0f0774a82f4c873d0a580f7a;7d009548-4d3c-4bbd-b66e-0536a865a0da) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00100-of-00167.parquet: (Request ID: Root=1-6abbf3f8-31410706748bc53f4c0f749e;57b079bf-c89e-4d99-a5ae-a49dd620f155) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00080-of-00147.parquet: (Request ID: Root=1-6abbf3f8-20d1823c3ff6d0ae5a7c68dd;386b66f7-52d6-467e-89de-3db7d7c0c30e) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/js/UltraData-Code-L3-js-part-00022-of-00029.parquet: (Request ID: Root=1-6abbf3f8-6beefa6e4dba3ee93f6e7449;65fb21dd-30e1-46a1-b0ec-fee5acaf483f) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00136-of-00137.parquet: (Request ID: Root=1-6abbf3f8-0cf37941363edb171500cb4b;2a7927dd-e7e3-4733-acbc-49018a80a54f) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00016-of-00167.parquet: (Request ID: Root=1-6abbf3f8-64103bb8288280642084187f;0e5c4997-387c-44ef-91a7-013f46c607b6) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00004-of-00147.parquet: (Request ID: Root=1-6abbf3f8-1cdc823f68a03ffc71b49669;372e6abd-b612-4ef1-9b00-d93d617083b6) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cs/UltraData-Code-L3-cs-part-00003-of-00028.parquet: (Request ID: Root=1-6abbf3f8-3c4d94b0165919783aa22d25;2806e164-a299-4ffc-9892-f0bff9b4e8c2) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00109-of-00137.parquet: (Request ID: Root=1-6abbf3f8-3694f5e6799ac1db36aceee4;3f40a9f4-7be9-4d20-ad40-cc8ba1025cbe) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00121-of-00147.parquet: (Request ID: Root=1-6abbf3f8-61842d067332d8974dbff415;2696a851-3abb-4317-91a6-dfd1bb867a9c) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00057-of-00137.parquet: (Request ID: Root=1-6abbf3f8-3c34efab14d714d8176d7344;69d1764c-45af-40dc-9fd1-12d4f8945c0d) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00147-of-00167.parquet: (Request ID: Root=1-6abbf3f8-5e15adaf6a800b5e419bd4f7;2daa5fdd-9a77-4f31-b23f-4cc89ebefff2) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00052-of-00147.parquet: (Request ID: Root=1-6abbf3f8-2b93fa3049cfb4d3666486ff;5e84be01-cbea-44c9-9b54-5f0e96df5cb6) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00162-of-00167.parquet: (Request ID: Root=1-6abbf3f8-25a8cf7f67af61b2572364a1;f336f439-96ec-44e9-b816-7beb9f36df7f) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00037-of-00147.parquet: (Request ID: Root=1-6abbf3f8-7c2014e92892280f15ebcbdb;66bb52f2-5166-4c4c-996a-fbd171da80a4) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00008-of-00137.parquet: (Request ID: Root=1-6abbf3f8-170db48f78fa79c210ad93e2;9d627beb-d4bd-447e-8dc5-d1072fe7634d) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00144-of-00167.parquet: (Request ID: Root=1-6abbf3f8-11eaedf83bce7b33328fb860;d3563460-7f1e-4f59-b6bc-0801969e08d6) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00100-of-00137.parquet: (Request ID: Root=1-6abbf3f8-0a25d14e69562b5b2a5fe050;5528202b-41c8-4841-99a9-a0a59ed549d6) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00015-of-00147.parquet: (Request ID: Root=1-6abbf3f8-5483861a1acb60e7339efebb;c34650ff-23ff-4ef0-aa51-008af6eba926) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00025-of-00137.parquet: (Request ID: Root=1-6abbf3f8-01ff95503f838ec9509cb3fb;d4615dae-abeb-42d3-ad8e-17a40a4153d6) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00029-of-00137.parquet: (Request ID: Root=1-6abbf3f8-7df157682e103c055fe206c1;5ab30f52-9396-4c9d-8881-b4883e8b6e5a) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00106-of-00167.parquet: (Request ID: Root=1-6abbf3f8-4897c85928471b057c141f05;42fd10b0-8a6d-4664-9742-907db239606a) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/go/UltraData-Code-L3-go-part-00010-of-00012.parquet: (Request ID: Root=1-6abbf3f8-6f684de10f57816b444aa0c4;783a1478-0e66-45ab-95d7-6c7a03969d19) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00044-of-00147.parquet: (Request ID: Root=1-6abbf3f8-0ca3918d43b1e81f6effa761;ec31a726-ceeb-4230-9bde-192a75b72bbd) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00118-of-00137.parquet: (Request ID: Root=1-6abbf3f8-39fcf3cd14e5d3e16c2554af;f74538cc-a83a-47f0-b528-b8c7d718ca05) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00065-of-00147.parquet: (Request ID: Root=1-6abbf3f8-01b1e7390663bea075310aa6;e27813ac-efa9-4f4e-84da-144d1a18d3b2) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00048-of-00137.parquet: (Request ID: Root=1-6abbf3f8-52900bd1100c170932c854d0;24f58ed0-dc3f-40d5-adec-e082f056ddca) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00078-of-00137.parquet: (Request ID: Root=1-6abbf3f8-5b37f5f177820644169f6f73;c1efabac-a41a-43bd-9c90-1efa84cd5cb2) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00132-of-00167.parquet: (Request ID: Root=1-6abbf3f8-57a022b1118424ef31885fde;71650b8f-4789-4ea2-b8bc-7cec92138f57) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00087-of-00147.parquet: (Request ID: Root=1-6abbf3f8-06443d930b5b6e6e13a0ed12;0ca6da3b-6508-4bd5-b981-b966e342a72e) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00074-of-00137.parquet: (Request ID: Root=1-6abbf3f8-7490efec09b10a9a2b474684;45bab4d5-4429-4639-8380-9d8ebd337295) + +429 Too Many Requests: you +17:23:04 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00045-of-00167.parquet: (Request ID: Root=1-6abbf3f8-02762a730091836e28e401f6;650bef46-1f8a-41e5-90e2-63feaead41b1) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00082-of-00147.parquet: (Request ID: Root=1-6abbf3f8-112e19f0345fbdc0331a0aef;828dc241-3642-40c9-a6ff-0c4f16ac1171) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00163-of-00167.parquet: (Request ID: Root=1-6abbf3f9-1ec7bdfe092ce4cd2dcc956a;cb41e6d4-7c4c-4de3-b89b-0ee6f377fec4) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00011-of-00167.parquet: (Request ID: Root=1-6abbf3f9-645fb5615cdf0f087966c241;bcb4cf38-744b-432d-a430-59b6ddef17df) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00164-of-00167.parquet: (Request ID: Root=1-6abbf3f9-7d1624b3548c860d7d7cf463;7af6a2ce-9e15-4820-b74b-9b49c756dbcf) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/js/UltraData-Code-L3-js-part-00009-of-00029.parquet: (Request ID: Root=1-6abbf3f9-0248e9bb5f7b4f7b6d764e06;8ceebf3b-4303-4361-8fa8-e3304b5351a6) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00047-of-00147.parquet: (Request ID: Root=1-6abbf3f9-31e10d8e7fce6f2458d2d000;a3fcc2e6-64ce-4b1e-a630-709a90c82931) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00092-of-00147.parquet: (Request ID: Root=1-6abbf3f9-2536399d26e43d8430227444;5dd485ce-6f2d-44a2-822c-b5f525677175) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00013-of-00147.parquet: (Request ID: Root=1-6abbf3f9-0902963d3d7b91d0244120ca;1035a6aa-a9f8-499b-9458-13f9c97780ce) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00115-of-00147.parquet: (Request ID: Root=1-6abbf3f9-26507b1b46301c4d298a2e20;0d56270d-8716-4952-a016-d59fb65ba045) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00102-of-00147.parquet: (Request ID: Root=1-6abbf3f9-12a470b2071661f1360b2568;6bc788c4-283a-4c4f-8996-551cc029dc26) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00046-of-00167.parquet: (Request ID: Root=1-6abbf3f9-33f91ea5496d0d964859105f;a65f2099-d7ee-434c-9c38-f7b46324237e) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/js/UltraData-Code-L3-js-part-00004-of-00029.parquet: (Request ID: Root=1-6abbf3f9-493fcd19449c5dca17e67de8;d79717b2-2fda-44a6-a25d-9879802b0022) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00038-of-00137.parquet: (Request ID: Root=1-6abbf3f9-0159198b530c184404e03a5d;55decb33-2951-45da-9d92-3e6672cdf440) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00092-of-00167.parquet: (Request ID: Root=1-6abbf3f9-6fd1b90e664fd5a158ecd5cd;fbfc3ee5-0f71-4314-93d5-55b6eea1e07f) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00046-of-00147.parquet: (Request ID: Root=1-6abbf3f9-4b9bd76e515003aa424c85a4;b175c9f7-7bb2-4307-9fe9-2d1aaa3b2002) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00041-of-00147.parquet: (Request ID: Root=1-6abbf3f9-7d95c0ea043057ee5a65a096;2cf73003-dab5-47c9-ae71-3f29b163a2b7) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00085-of-00167.parquet: (Request ID: Root=1-6abbf3f9-3eaee7c148229f9c70694266;5b51f782-21b5-4ace-962f-744eebb51648) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/go/UltraData-Code-L3-go-part-00012-of-00012.parquet: (Request ID: Root=1-6abbf3f9-1b7b3c4b43b070bf1fa89002;5a5c7bac-6b3f-4f5e-84e1-00dcc088ad2e) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/js/UltraData-Code-L3-js-part-00008-of-00029.parquet: (Request ID: Root=1-6abbf3f9-62fb45de06bf34972885b09d;f7078b75-01a9-4b42-8291-999e259d2ea4) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00073-of-00167.parquet: (Request ID: Root=1-6abbf3f9-3a3cff882fac7fc4585648b2;a62f60d8-ad85-4cd4-9b7e-a74a4e5b98df) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00043-of-00167.parquet: (Request ID: Root=1-6abbf3f9-6ab6190a76cbdc19704a86ac;a35519ae-0fd7-44fb-83c3-f9466bdfd736) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00115-of-00167.parquet: (Request ID: Root=1-6abbf3f9-2ab5c59047c1301801ed5932;45d0ceda-7b7a-47f7-b618-d3a7bc3a7c04) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00056-of-00167.parquet: (Request ID: Root=1-6abbf3f9-7948346a39f204e62d505f51;809abe8b-58b1-47f5-a979-06b46f2bab6e) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00012-of-00137.parquet: (Request ID: Root=1-6abbf3f9-5423e4fc4e120d6859447dc3;be1fe89e-1690-4fbe-9905-07a76a23a5b7) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00113-of-00137.parquet: (Request ID: Root=1-6abbf3f9-509b329b3c5ac5ea346fd096;2212f3e8-8cf1-4d3c-8006-a9d6191e8c25) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cs/UltraData-Code-L3-cs-part-00009-of-00028.parquet: (Request ID: Root=1-6abbf3f9-3d627d1c157305114dc9fead;09a31cf4-1cdd-4144-a46e-b755c1773db5) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00032-of-00147.parquet: (Request ID: Root=1-6abbf3f9-51197e7772d14e752cafd1dc;e6a392b3-9850-46e1-82e2-c688317dd731) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00137-of-00167.parquet: (Request ID: Root=1-6abbf3f9-2acc09e609600267268a2e0f;be06d8ab-9a73-477d-b8e8-95b3a1d0c44a) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00090-of-00137.parquet: (Request ID: Root=1-6abbf3f9-24d1c89e2813d79d66e2838a;9c09d556-4527-4d71-914c-f66ffcbbd007) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00026-of-00137.parquet: (Request ID: Root=1-6abbf3f9-78a0813c358b4f314381f98a;5498c651-7642-4ae5-b010-1edf832490c7) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00125-of-00137.parquet: (Request ID: Root=1-6abbf3f9-3d11d748670cb0925ffb4c44;a1329b7e-d481-4dd0-8658-56f251989257) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00147-of-00147.parquet: (Request ID: Root=1-6abbf3f9-0b90d38b58797c3e15b7dce6;782b9d92-8d31-4f2c-b95a-b237304e4ca0) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00083-of-00147.parquet: (Request ID: Root=1-6abbf3f9-7dc9bcef3d7437bc25ec6254;c1c9e733-81a7-4929-a767-787f33157fb1) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00042-of-00167.parquet: (Request ID: Root=1-6abbf3f9-6b61258233abaf251830c35d;404379a8-2b96-42ed-a639-e3e5a3c18daa) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00108-of-00167.parquet: (Request ID: Root=1-6abbf3f9-7760319235de15351709f125;998e3463-4f31-48b2-8601-cad60fdfed46) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00025-of-00147.parquet: (Request ID: Root=1-6abbf3f9-31d3153e3fdbee3164f1b25a;a968b79d-06ac-422d-936c-ae5e0bd657df) + +429 Too Many Requests: you +17:23:05 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00002-of-00167.parquet: (Request ID: Root=1-6abbf3f9-29e7893445e3eb271637da35;38747740-8ad1-4e49-b908-258572bd242c) + +429 Too Many Requests: you +17:23:06 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00142-of-00147.parquet: (Request ID: Root=1-6abbf3fa-285fbfe5210d4f12342ed16c;bc786f11-7a5b-4361-8328-ba22ac1b7106) + +429 Too Many Requests: you +17:23:06 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00004-of-00167.parquet: (Request ID: Root=1-6abbf3fa-47653ec077b166b357899a76;3c123d1c-430f-4c6c-96ba-be9123a21be6) + +429 Too Many Requests: you +17:23:06 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00127-of-00147.parquet: (Request ID: Root=1-6abbf3fa-29b2da645279864570c8fc34;c201b46e-0050-42e2-9ba6-4147b7503f92) + +429 Too Many Requests: you +17:23:06 parquet file skipped data/UltraData-Code-L3/cs/UltraData-Code-L3-cs-part-00027-of-00028.parquet: (Request ID: Root=1-6abbf3fa-0224f328762f53b35a5eec2c;537ee2f0-f5f8-4452-9484-5e08ad7674ce) + +429 Too Many Requests: you +17:23:06 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00066-of-00147.parquet: (Request ID: Root=1-6abbf3fa-7ae9c4ce1f955b615c53c88c;17f6b265-fb96-4132-b75c-3ca67ea5051f) + +429 Too Many Requests: you +17:23:06 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00114-of-00167.parquet: (Request ID: Root=1-6abbf3fa-69ce34bb64a373857425aea2;74f04b8c-16e2-4f38-86d9-a97989204247) + +429 Too Many Requests: you +17:23:06 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00076-of-00167.parquet: (Request ID: Root=1-6abbf3fa-4363fabf32728b0438b17a10;d2ee1c4f-dbb2-46eb-bb7e-bf7e66f96f21) + +429 Too Many Requests: you +17:23:06 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00051-of-00147.parquet: (Request ID: Root=1-6abbf3fa-1ff0341a2d0dcd624f5e6ffa;8864cd58-7372-4dac-98c1-c0b5855dc950) + +429 Too Many Requests: you +17:23:06 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00131-of-00147.parquet: (Request ID: Root=1-6abbf3fa-0df362d273aa97c96cc60f4a;8a502929-e312-4a16-a18b-a148425f8f8b) + +429 Too Many Requests: you +17:23:57 mix source 2 paused 30 min ((Request ID: Root=1-6abbf42d-6254f2bd0d3826872ff80822;746c7f2b-b555-4877-be9d-f3) +17:23:57 mix source 1 paused 30 min ((Request ID: Root=1-6abbf42d-774c3a627e59b7c56bfbd567;3c33bc71-e1e3-48b1-abfa-07) +17:23:57 mix source 6 paused 30 min ((Request ID: Root=1-6abbf42d-2b58bbbe1ca0d3810517591e;9a887aa7-627e-43d1-b4f2-1a) +17:27:55 step 166801 loss=2.8582 lr=2.42e-05 tok=21.976B dt=362 wait=353 +17:28:03 step 166802 loss=2.9907 lr=2.42e-05 tok=21.976B dt=370 wait=353 +17:29:00 step 166820 loss=3.1607 lr=2.42e-05 tok=21.978B dt=427 wait=353 +17:30:04 step 166840 loss=3.0585 lr=2.42e-05 tok=21.981B dt=491 wait=353 +17:31:07 step 166860 loss=3.0301 lr=2.42e-05 tok=21.983B dt=554 wait=354 +launch +17:31:50 name=w151 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 98/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:31:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:31:54 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:32:04 params 1.08B +17:32:05 resumed step=166800 seen=21.98B round=673 +17:32:05 stream salt=166800 +17:34:07 blocks compiled +17:34:07 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:38:20 step 166801 loss=2.7407 lr=2.42e-05 tok=21.976B dt=253 wait=244 +17:38:27 step 166802 loss=2.9056 lr=2.42e-05 tok=21.976B dt=260 wait=244 +17:39:08 global r=674 load 4.270 -> 4.258 +17:39:41 step 166960 loss=3.2346 lr=2.42e-05 tok=21.978B dt=333 wait=244 +17:40:46 step 166980 loss=3.1417 lr=2.41e-05 tok=21.981B dt=399 wait=244 +17:41:50 step 167000 loss=3.0598 lr=2.41e-05 tok=21.983B dt=463 wait=244 +17:42:55 step 167020 loss=3.0812 lr=2.41e-05 tok=21.986B dt=528 wait=244 +17:43:59 step 167040 loss=3.1164 lr=2.41e-05 tok=21.989B dt=592 wait=244 +17:45:04 step 167060 loss=3.1357 lr=2.41e-05 tok=21.991B dt=657 wait=245 +17:46:08 step 167080 loss=2.9580 lr=2.41e-05 tok=21.994B dt=721 wait=245 +17:47:12 step 167100 loss=2.9662 lr=2.41e-05 tok=21.997B dt=785 wait=245 +17:47:56 pushed ckpt_w151.pt,log_w151.txt +17:47:56 waited 0s for global r=675 +17:48:03 global r=675 load 4.293 -> 4.260 +17:49:11 step 167120 loss=2.9045 lr=2.41e-05 tok=21.999B dt=903 wait=245 +17:50:14 step 167140 loss=3.0816 lr=2.41e-05 tok=22.002B dt=967 wait=245 +17:51:19 step 167160 loss=2.9382 lr=2.40e-05 tok=22.004B dt=1032 wait=245 +17:52:24 step 167180 loss=3.0782 lr=2.40e-05 tok=22.007B dt=1097 wait=245 +17:53:28 step 167200 loss=2.9820 lr=2.40e-05 tok=22.010B dt=1161 wait=246 +17:54:33 step 167220 loss=3.0019 lr=2.40e-05 tok=22.012B dt=1226 wait=246 +17:55:37 step 167240 loss=2.9676 lr=2.40e-05 tok=22.015B dt=1290 wait=246 +17:56:24 pushed ckpt_w151.pt,log_w151.txt +17:59:25 waited 181s for global r=676 +17:59:35 global r=676 load 4.283 -> 4.285 +18:00:09 step 167260 loss=3.0722 lr=2.40e-05 tok=22.017B dt=1561 wait=246 +18:01:13 step 167280 loss=2.9406 lr=2.40e-05 tok=22.020B dt=1626 wait=246 +18:02:17 step 167300 loss=2.9308 lr=2.40e-05 tok=22.023B dt=1690 wait=246 +18:03:22 step 167320 loss=3.0080 lr=2.40e-05 tok=22.025B dt=1755 wait=246 +18:04:26 step 167340 loss=2.9219 lr=2.40e-05 tok=22.028B dt=1818 wait=246 +18:05:30 step 167360 loss=2.9759 lr=2.39e-05 tok=22.031B dt=1883 wait=247 +18:06:34 step 167380 loss=2.9029 lr=2.39e-05 tok=22.033B dt=1947 wait=247 +18:07:38 step 167400 loss=2.8866 lr=2.39e-05 tok=22.036B dt=2011 wait=247 +18:07:51 pushed ckpt_w151.pt,log_w151.txt +18:09:23 waited 92s for global r=677 +18:09:32 global r=677 load 4.290 -> 4.280 +18:10:38 step 167420 loss=2.9131 lr=2.39e-05 tok=22.038B dt=2191 wait=247 +18:11:42 step 167440 loss=2.9206 lr=2.39e-05 tok=22.041B dt=2255 wait=247 +18:12:46 step 167460 loss=2.9356 lr=2.39e-05 tok=22.044B dt=2319 wait=247 +18:13:51 step 167480 loss=2.8682 lr=2.39e-05 tok=22.046B dt=2384 wait=247 +18:14:55 step 167500 loss=2.9085 lr=2.39e-05 tok=22.049B dt=2448 wait=248 +18:16:00 step 167520 loss=3.0009 lr=2.39e-05 tok=22.052B dt=2512 wait=248 +18:17:03 step 167540 loss=2.9008 lr=2.38e-05 tok=22.054B dt=2576 wait=248 +18:17:54 pushed ckpt_w151.pt,log_w151.txt +18:19:00 waited 66s for global r=678 +18:19:08 global r=678 load 4.286 -> 4.281 +18:19:41 step 167560 loss=3.0217 lr=2.38e-05 tok=22.057B dt=2734 wait=248 +18:20:46 step 167580 loss=2.8381 lr=2.38e-05 tok=22.059B dt=2799 wait=248 +18:21:50 step 167600 loss=2.8964 lr=2.38e-05 tok=22.062B dt=2863 wait=248 +18:22:54 step 167620 loss=2.9947 lr=2.38e-05 tok=22.065B dt=2927 wait=248 +18:23:58 step 167640 loss=2.9399 lr=2.38e-05 tok=22.067B dt=2991 wait=249 +18:25:03 step 167660 loss=2.9941 lr=2.38e-05 tok=22.070B dt=3056 wait=249 +18:26:07 step 167680 loss=2.9044 lr=2.38e-05 tok=22.073B dt=3120 wait=249 +18:27:11 step 167700 loss=2.9659 lr=2.38e-05 tok=22.075B dt=3184 wait=249 +18:27:23 pushed ckpt_w151.pt,log_w151.txt +18:28:54 waited 91s for global r=679 +18:29:03 global r=679 load 4.284 -> 4.283 +18:30:08 step 167720 loss=2.9276 lr=2.38e-05 tok=22.078B dt=3361 wait=249 +18:31:13 step 167740 loss=2.9502 lr=2.37e-05 tok=22.080B dt=3425 wait=249 +18:32:17 step 167760 loss=2.9422 lr=2.37e-05 tok=22.083B dt=3490 wait=249 +18:33:21 step 167780 loss=2.9135 lr=2.37e-05 tok=22.086B dt=3554 wait=249 +18:34:25 step 167800 loss=3.0227 lr=2.37e-05 tok=22.088B dt=3618 wait=250 +18:35:31 step 167820 loss=2.9358 lr=2.37e-05 tok=22.091B dt=3684 wait=250 +18:36:35 step 167840 loss=2.8975 lr=2.37e-05 tok=22.093B dt=3748 wait=250 +18:37:26 pushed ckpt_w151.pt,log_w151.txt +18:38:43 waited 77s for global r=680 +18:38:54 global r=680 load 4.285 -> 4.282 +18:39:28 step 167860 loss=3.0557 lr=2.37e-05 tok=22.096B dt=3921 wait=250 +18:40:32 step 167880 loss=2.9847 lr=2.37e-05 tok=22.099B dt=3985 wait=250 +18:41:37 step 167900 loss=2.8917 lr=2.37e-05 tok=22.101B dt=4050 wait=250 +18:42:41 step 167920 loss=3.0257 lr=2.36e-05 tok=22.104B dt=4114 wait=251 +18:43:46 step 167940 loss=2.9356 lr=2.36e-05 tok=22.107B dt=4179 wait=251 +18:44:50 step 167960 loss=2.9861 lr=2.36e-05 tok=22.109B dt=4243 wait=251 +18:45:55 step 167980 loss=2.9368 lr=2.36e-05 tok=22.112B dt=4308 wait=251 +18:47:00 step 168000 loss=2.8916 lr=2.36e-05 tok=22.114B dt=4373 wait=251 +18:47:19 pushed ckpt_w151.pt,log_w151.txt +18:48:36 waited 78s for global r=681 +18:48:45 global r=681 load 4.286 -> 4.285 +18:49:51 step 168020 loss=2.9365 lr=2.36e-05 tok=22.117B dt=4544 wait=251 +18:50:55 step 168040 loss=2.8792 lr=2.36e-05 tok=22.120B dt=4607 wait=251 +18:51:59 step 168060 loss=2.9106 lr=2.36e-05 tok=22.122B dt=4672 wait=251 +18:53:03 step 168080 loss=2.8555 lr=2.36e-05 tok=22.125B dt=4736 wait=252 +18:54:07 step 168100 loss=2.8487 lr=2.36e-05 tok=22.128B dt=4800 wait=252 +18:55:12 step 168120 loss=2.9783 lr=2.35e-05 tok=22.130B dt=4865 wait=252 +18:56:16 step 168140 loss=2.9592 lr=2.35e-05 tok=22.133B dt=4929 wait=252 +18:57:05 pushed ckpt_w151.pt,log_w151.txt +18:58:36 waited 91s for global r=682 +18:58:45 global r=682 load 4.292 -> 4.283 +18:59:19 step 168160 loss=3.0415 lr=2.35e-05 tok=22.135B dt=5112 wait=252 +19:00:23 step 168180 loss=3.0235 lr=2.35e-05 tok=22.138B dt=5176 wait=252 +19:01:28 step 168200 loss=2.9479 lr=2.35e-05 tok=22.141B dt=5241 wait=252 +19:02:34 step 168220 loss=3.0337 lr=2.35e-05 tok=22.143B dt=5307 wait=253 +19:03:39 step 168240 loss=3.0436 lr=2.35e-05 tok=22.146B dt=5372 wait=253 +19:04:44 step 168260 loss=3.1104 lr=2.35e-05 tok=22.149B dt=5436 wait=253 +19:05:48 step 168280 loss=3.0712 lr=2.35e-05 tok=22.151B dt=5500 wait=253 +19:06:53 step 168300 loss=3.0592 lr=2.35e-05 tok=22.154B dt=5566 wait=253 +19:07:13 pushed ckpt_w151.pt,log_w151.txt +19:08:19 waited 66s for global r=683 +19:08:28 global r=683 load 4.285 -> 4.281 +19:09:34 step 168320 loss=2.9856 lr=2.34e-05 tok=22.156B dt=5727 wait=253 +19:10:39 step 168340 loss=3.1016 lr=2.34e-05 tok=22.159B dt=5792 wait=254 +19:11:43 step 168360 loss=3.0887 lr=2.34e-05 tok=22.162B dt=5856 wait=254 +19:12:48 step 168380 loss=3.0025 lr=2.34e-05 tok=22.164B dt=5921 wait=254 +19:13:53 step 168400 loss=3.0089 lr=2.34e-05 tok=22.167B dt=5986 wait=254 +19:14:58 step 168420 loss=3.0762 lr=2.34e-05 tok=22.170B dt=6051 wait=254 +19:16:02 step 168440 loss=2.9115 lr=2.34e-05 tok=22.172B dt=6115 wait=254 +19:16:47 pushed ckpt_w151.pt,log_w151.txt +19:18:19 waited 93s for global r=684 +19:18:34 global r=684 load 4.286 -> 4.282 +19:19:07 step 168460 loss=3.0852 lr=2.34e-05 tok=22.175B dt=6300 wait=254 +19:20:11 step 168480 loss=2.9604 lr=2.34e-05 tok=22.177B dt=6364 wait=254 +19:21:14 step 168500 loss=2.9293 lr=2.33e-05 tok=22.180B dt=6427 wait=255 +19:22:18 step 168520 loss=3.0208 lr=2.33e-05 tok=22.183B dt=6491 wait=255 +19:23:23 step 168540 loss=2.9370 lr=2.33e-05 tok=22.185B dt=6556 wait=255 +19:24:27 step 168560 loss=2.8945 lr=2.33e-05 tok=22.188B dt=6619 wait=255 +19:25:30 step 168580 loss=2.8648 lr=2.33e-05 tok=22.190B dt=6683 wait=255 +19:26:33 step 168600 loss=2.8325 lr=2.33e-05 tok=22.193B dt=6746 wait=255 +19:26:49 pushed ckpt_w151.pt,log_w151.txt +19:27:50 waited 61s for global r=685 +19:27:59 global r=685 load 4.289 -> 4.283 +19:29:04 step 168620 loss=2.8657 lr=2.33e-05 tok=22.196B dt=6897 wait=256 +19:30:08 step 168640 loss=2.8373 lr=2.33e-05 tok=22.198B dt=6961 wait=256 +19:31:12 step 168660 loss=2.7955 lr=2.33e-05 tok=22.201B dt=7025 wait=256 +19:32:16 step 168680 loss=2.7990 lr=2.33e-05 tok=22.204B dt=7089 wait=256 +19:33:20 step 168700 loss=2.8305 lr=2.32e-05 tok=22.206B dt=7152 wait=256 +19:34:23 step 168720 loss=2.9182 lr=2.32e-05 tok=22.209B dt=7216 wait=256 +19:35:28 step 168740 loss=2.7626 lr=2.32e-05 tok=22.211B dt=7280 wait=257 +19:36:12 pushed ckpt_w151.pt,log_w151.txt +19:37:43 waited 91s for global r=686 +19:37:51 global r=686 load 4.291 -> 4.286 +19:38:24 step 168760 loss=2.9224 lr=2.32e-05 tok=22.214B dt=7457 wait=257 +19:39:28 step 168780 loss=2.8282 lr=2.32e-05 tok=22.217B dt=7521 wait=257 +19:40:31 step 168800 loss=2.8784 lr=2.32e-05 tok=22.219B dt=7584 wait=257 +19:41:36 step 168820 loss=2.7727 lr=2.32e-05 tok=22.222B dt=7648 wait=257 +19:42:39 step 168840 loss=2.7593 lr=2.32e-05 tok=22.225B dt=7712 wait=257 +19:43:43 step 168860 loss=2.9280 lr=2.32e-05 tok=22.227B dt=7776 wait=257 +19:44:46 step 168880 loss=2.9270 lr=2.32e-05 tok=22.230B dt=7839 wait=258 +19:45:51 step 168900 loss=2.8065 lr=2.31e-05 tok=22.232B dt=7904 wait=258 +19:46:03 pushed ckpt_w151.pt,log_w151.txt +19:47:36 waited 93s for global r=687 +19:47:42 global r=687 load 4.286 -> 4.284 +19:48:46 step 168920 loss=2.8114 lr=2.31e-05 tok=22.235B dt=8079 wait=258 +19:49:50 step 168940 loss=2.9493 lr=2.31e-05 tok=22.238B dt=8143 wait=258 +19:50:54 step 168960 loss=2.9253 lr=2.31e-05 tok=22.240B dt=8207 wait=258 +19:51:58 step 168980 loss=2.8523 lr=2.31e-05 tok=22.243B dt=8271 wait=258 +19:53:02 step 169000 loss=2.9439 lr=2.31e-05 tok=22.246B dt=8335 wait=258 +19:54:06 step 169020 loss=3.0135 lr=2.31e-05 tok=22.248B dt=8399 wait=259 +19:55:10 step 169040 loss=2.9265 lr=2.31e-05 tok=22.251B dt=8463 wait=259 +19:55:56 pushed ckpt_w151.pt,log_w151.txt +19:57:14 waited 78s for global r=688 +19:57:24 global r=688 load 4.291 -> 4.284 +19:57:58 step 169060 loss=3.0324 lr=2.31e-05 tok=22.253B dt=8631 wait=259 +19:59:03 step 169080 loss=2.9437 lr=2.31e-05 tok=22.256B dt=8695 wait=259 +20:00:07 step 169100 loss=2.9018 lr=2.30e-05 tok=22.259B dt=8760 wait=259 +20:01:11 step 169120 loss=2.9989 lr=2.30e-05 tok=22.261B dt=8824 wait=259 +20:02:16 step 169140 loss=2.9317 lr=2.30e-05 tok=22.264B dt=8889 wait=259 +20:03:20 step 169160 loss=3.0133 lr=2.30e-05 tok=22.267B dt=8953 wait=260 +20:04:25 step 169180 loss=2.9705 lr=2.30e-05 tok=22.269B dt=9018 wait=260 +20:05:29 step 169200 loss=2.9425 lr=2.30e-05 tok=22.272B dt=9082 wait=260 +20:05:47 pushed ckpt_w151.pt,log_w151.txt +20:09:29 global r=689 load 4.285 -> 4.280 +20:10:33 step 169220 loss=2.9511 lr=2.30e-05 tok=22.276B dt=9386 wait=260 +20:11:37 step 169240 loss=2.9650 lr=2.30e-05 tok=22.278B dt=9450 wait=260 +20:12:41 step 169260 loss=3.0368 lr=2.30e-05 tok=22.281B dt=9514 wait=260 +20:13:46 step 169280 loss=2.9702 lr=2.30e-05 tok=22.284B dt=9578 wait=260 +20:14:50 step 169300 loss=2.9599 lr=2.29e-05 tok=22.286B dt=9643 wait=261 +20:15:55 step 169320 loss=2.9806 lr=2.29e-05 tok=22.289B dt=9708 wait=261 +20:17:00 step 169340 loss=2.9702 lr=2.29e-05 tok=22.291B dt=9773 wait=261 +20:17:50 pushed ckpt_w151.pt,log_w151.txt +20:18:55 waited 65s for global r=690 +20:19:02 global r=690 load 4.289 -> 4.284 +20:19:35 step 169360 loss=3.2022 lr=2.29e-05 tok=22.294B dt=9928 wait=261 +20:20:40 step 169380 loss=3.0127 lr=2.29e-05 tok=22.297B dt=9993 wait=261 +20:21:46 step 169400 loss=2.8600 lr=2.29e-05 tok=22.299B dt=10059 wait=261 +20:22:50 step 169420 loss=3.0455 lr=2.29e-05 tok=22.302B dt=10123 wait=261 +20:23:55 step 169440 loss=2.9098 lr=2.29e-05 tok=22.305B dt=10188 wait=262 +20:25:00 step 169460 loss=2.9561 lr=2.29e-05 tok=22.307B dt=10253 wait=262 +20:26:05 step 169480 loss=2.8873 lr=2.29e-05 tok=22.310B dt=10318 wait=262 +20:27:10 step 169500 loss=2.9340 lr=2.28e-05 tok=22.312B dt=10383 wait=262 +20:27:24 pushed ckpt_w151.pt,log_w151.txt +20:28:25 waited 61s for global r=691 +20:28:33 global r=691 load 4.299 -> 4.287 +20:29:40 step 169520 loss=2.9487 lr=2.28e-05 tok=22.315B dt=10532 wait=262 +20:30:44 step 169540 loss=2.9880 lr=2.28e-05 tok=22.318B dt=10597 wait=262 +20:31:49 step 169560 loss=2.9352 lr=2.28e-05 tok=22.320B dt=10661 wait=262 +20:32:53 step 169580 loss=2.9452 lr=2.28e-05 tok=22.323B dt=10726 wait=263 +20:33:58 step 169600 loss=2.9028 lr=2.28e-05 tok=22.325B dt=10790 wait=263 +20:35:02 step 169620 loss=2.9649 lr=2.28e-05 tok=22.328B dt=10855 wait=263 +20:36:07 step 169640 loss=2.9139 lr=2.28e-05 tok=22.331B dt=10920 wait=263 +20:36:52 pushed ckpt_w151.pt,log_w151.txt +20:38:08 waited 76s for global r=692 +20:38:14 global r=692 load 4.284 -> 4.284 +20:38:48 step 169660 loss=2.9743 lr=2.28e-05 tok=22.333B dt=11081 wait=263 +20:39:53 step 169680 loss=2.8826 lr=2.28e-05 tok=22.336B dt=11146 wait=263 +20:40:58 step 169700 loss=2.9620 lr=2.27e-05 tok=22.339B dt=11211 wait=263 +20:42:03 step 169720 loss=3.0353 lr=2.27e-05 tok=22.341B dt=11276 wait=264 +20:43:08 step 169740 loss=2.9013 lr=2.27e-05 tok=22.344B dt=11341 wait=264 +20:44:13 step 169760 loss=2.9445 lr=2.27e-05 tok=22.346B dt=11406 wait=264 +20:45:17 step 169780 loss=2.9419 lr=2.27e-05 tok=22.349B dt=11470 wait=264 +20:46:22 step 169800 loss=3.0395 lr=2.27e-05 tok=22.352B dt=11535 wait=264 +20:46:45 pushed ckpt_w151.pt,log_w151.txt +20:47:46 waited 61s for global r=693 +20:47:56 global r=693 load 4.281 -> 4.282 +20:49:02 step 169820 loss=3.0384 lr=2.27e-05 tok=22.354B dt=11695 wait=264 +20:50:07 step 169840 loss=2.9904 lr=2.27e-05 tok=22.357B dt=11760 wait=264 +20:51:12 step 169860 loss=2.9034 lr=2.27e-05 tok=22.360B dt=11825 wait=265 +20:52:18 step 169880 loss=2.9203 lr=2.27e-05 tok=22.362B dt=11890 wait=265 +20:53:22 step 169900 loss=2.9313 lr=2.26e-05 tok=22.365B dt=11955 wait=265 +20:54:27 step 169920 loss=3.0404 lr=2.26e-05 tok=22.367B dt=12019 wait=265 +20:55:31 step 169940 loss=2.9814 lr=2.26e-05 tok=22.370B dt=12084 wait=265 +20:56:23 pushed ckpt_w151.pt,log_w151.txt +20:57:24 waited 61s for global r=694 +20:57:34 global r=694 load 4.289 -> 4.282 +20:58:07 step 169960 loss=3.0370 lr=2.26e-05 tok=22.373B dt=12240 wait=265 +20:59:12 step 169980 loss=2.9473 lr=2.26e-05 tok=22.375B dt=12305 wait=265 +21:00:17 step 170000 loss=2.9411 lr=2.26e-05 tok=22.378B dt=12370 wait=266 +21:01:22 step 170020 loss=3.0087 lr=2.26e-05 tok=22.381B dt=12435 wait=266 +21:02:26 step 170040 loss=2.9919 lr=2.26e-05 tok=22.383B dt=12499 wait=266 +21:03:31 step 170060 loss=3.0703 lr=2.26e-05 tok=22.386B dt=12564 wait=266 +21:04:35 step 170080 loss=3.0109 lr=2.26e-05 tok=22.388B dt=12628 wait=266 +21:05:41 step 170100 loss=2.7145 lr=2.26e-05 tok=22.391B dt=12694 wait=266 +21:05:59 pushed ckpt_w151.pt,log_w151.txt +21:07:04 waited 66s for global r=695 +21:07:15 global r=695 load 4.286 -> 4.282 +21:08:22 step 170120 loss=2.9992 lr=2.25e-05 tok=22.394B dt=12854 wait=266 +21:09:26 step 170140 loss=3.1367 lr=2.25e-05 tok=22.396B dt=12919 wait=267 +21:10:32 step 170160 loss=2.9703 lr=2.25e-05 tok=22.399B dt=12985 wait=267 +21:11:37 step 170180 loss=3.0181 lr=2.25e-05 tok=22.402B dt=13049 wait=267 +21:12:41 step 170200 loss=2.9841 lr=2.25e-05 tok=22.404B dt=13114 wait=267 +21:13:46 step 170220 loss=3.0494 lr=2.25e-05 tok=22.407B dt=13179 wait=267 +21:14:52 step 170240 loss=3.0259 lr=2.25e-05 tok=22.409B dt=13244 wait=267 +21:15:38 pushed ckpt_w151.pt,log_w151.txt +21:19:13 step 170260 loss=2.9796 lr=2.25e-05 tok=22.412B dt=13506 wait=268 +21:19:21 global r=696 load 4.279 -> 4.281 +21:19:54 step 170260 loss=3.0444 lr=2.25e-05 tok=22.413B dt=13547 wait=268 +21:20:58 step 170280 loss=2.8888 lr=2.25e-05 tok=22.416B dt=13611 wait=268 +21:22:03 step 170300 loss=2.9734 lr=2.25e-05 tok=22.419B dt=13676 wait=268 +21:23:08 step 170320 loss=2.9909 lr=2.24e-05 tok=22.421B dt=13741 wait=268 +21:24:13 step 170340 loss=2.9164 lr=2.24e-05 tok=22.424B dt=13806 wait=268 +21:25:18 step 170360 loss=2.9255 lr=2.24e-05 tok=22.426B dt=13871 wait=268 +21:26:22 step 170380 loss=2.9150 lr=2.24e-05 tok=22.429B dt=13935 wait=268 +21:27:26 step 170400 loss=2.9682 lr=2.24e-05 tok=22.432B dt=13999 wait=268 +21:27:42 pushed ckpt_w151.pt,log_w151.txt +21:30:43 waited 181s for global r=697 +21:30:52 global r=697 load 4.283 -> 4.282 +21:31:57 step 170420 loss=2.9840 lr=2.24e-05 tok=22.434B dt=14270 wait=269 +21:33:02 step 170440 loss=2.9040 lr=2.24e-05 tok=22.437B dt=14335 wait=269 +21:34:07 step 170460 loss=2.9337 lr=2.24e-05 tok=22.440B dt=14399 wait=269 +21:35:11 step 170480 loss=2.9681 lr=2.24e-05 tok=22.442B dt=14464 wait=269 +21:36:15 step 170500 loss=2.9778 lr=2.24e-05 tok=22.445B dt=14528 wait=269 +21:37:20 step 170520 loss=3.1599 lr=2.23e-05 tok=22.447B dt=14593 wait=269 +21:38:25 step 170540 loss=2.9760 lr=2.23e-05 tok=22.450B dt=14658 wait=269 +21:39:11 pushed ckpt_w151.pt,log_w151.txt +21:42:46 step 170560 loss=3.2213 lr=2.23e-05 tok=22.453B dt=14919 wait=270 +21:42:53 global r=698 load 4.283 -> 4.285 +21:43:25 step 170560 loss=3.2185 lr=2.23e-05 tok=22.454B dt=14958 wait=270 +21:44:29 step 170580 loss=3.0800 lr=2.23e-05 tok=22.457B dt=15022 wait=270 +21:45:35 step 170600 loss=3.0456 lr=2.23e-05 tok=22.459B dt=15088 wait=270 +21:46:40 step 170620 loss=3.0168 lr=2.23e-05 tok=22.462B dt=15153 wait=270 +21:47:46 step 170640 loss=2.8891 lr=2.23e-05 tok=22.464B dt=15218 wait=270 +21:48:50 step 170660 loss=2.9927 lr=2.23e-05 tok=22.467B dt=15283 wait=270 +21:49:55 step 170680 loss=2.9721 lr=2.23e-05 tok=22.470B dt=15347 wait=271 +21:51:00 step 170700 loss=3.0018 lr=2.23e-05 tok=22.472B dt=15413 wait=271 +21:51:22 pushed ckpt_w151.pt,log_w151.txt +21:54:09 waited 168s for global r=699 +21:54:17 global r=699 load 4.287 -> 4.282 +21:55:23 step 170720 loss=2.9205 lr=2.23e-05 tok=22.475B dt=15676 wait=271 +21:56:34 step 170740 loss=2.9212 lr=2.22e-05 tok=22.478B dt=15746 wait=271 +21:57:39 step 170760 loss=2.8662 lr=2.22e-05 tok=22.480B dt=15812 wait=271 +21:58:44 step 170780 loss=2.8395 lr=2.22e-05 tok=22.483B dt=15877 wait=271 +21:59:49 step 170800 loss=2.8417 lr=2.22e-05 tok=22.485B dt=15942 wait=271 +22:00:53 step 170820 loss=2.9963 lr=2.22e-05 tok=22.488B dt=16006 wait=272 +22:01:58 step 170840 loss=2.7966 lr=2.22e-05 tok=22.491B dt=16071 wait=272 +22:02:46 pushed ckpt_w151.pt,log_w151.txt +22:05:47 waited 182s for global r=700 +22:05:56 global r=700 load 4.288 -> 4.283 +22:06:29 step 170860 loss=2.9823 lr=2.22e-05 tok=22.493B dt=16342 wait=272 +22:07:33 step 170880 loss=2.9596 lr=2.22e-05 tok=22.496B dt=16406 wait=272 +22:08:37 step 170900 loss=2.9158 lr=2.22e-05 tok=22.499B dt=16470 wait=272 +22:09:42 step 170920 loss=2.9991 lr=2.22e-05 tok=22.501B dt=16535 wait=272 +22:10:47 step 170940 loss=2.8670 lr=2.21e-05 tok=22.504B dt=16600 wait=273 +22:11:51 step 170960 loss=2.9695 lr=2.21e-05 tok=22.506B dt=16664 wait=273 +22:12:55 step 170980 loss=2.9456 lr=2.21e-05 tok=22.509B dt=16728 wait=273 +22:14:00 step 171000 loss=2.9106 lr=2.21e-05 tok=22.512B dt=16793 wait=273 +22:14:14 pushed ckpt_w151.pt,log_w151.txt +22:17:16 waited 182s for global r=701 +22:17:26 global r=701 load 4.292 -> 4.287 +22:18:32 step 171020 loss=2.9137 lr=2.21e-05 tok=22.514B dt=17065 wait=273 +22:19:36 step 171040 loss=2.9399 lr=2.21e-05 tok=22.517B dt=17129 wait=273 +22:20:40 step 171060 loss=2.9632 lr=2.21e-05 tok=22.519B dt=17193 wait=273 +22:21:45 step 171080 loss=2.9248 lr=2.21e-05 tok=22.522B dt=17258 wait=274 +22:22:50 step 171100 loss=2.9131 lr=2.21e-05 tok=22.525B dt=17323 wait=274 +22:23:55 step 171120 loss=3.0365 lr=2.21e-05 tok=22.527B dt=17388 wait=274 +22:25:00 step 171140 loss=2.8166 lr=2.21e-05 tok=22.530B dt=17452 wait=274 +22:25:45 pushed ckpt_w151.pt,log_w151.txt +22:29:21 step 171160 loss=3.0252 lr=2.20e-05 tok=22.533B dt=17714 wait=274 +22:29:32 global r=702 load 4.286 -> 4.284 +22:30:04 step 171160 loss=2.9196 lr=2.20e-05 tok=22.534B dt=17757 wait=274 +22:31:08 step 171180 loss=2.8482 lr=2.20e-05 tok=22.537B dt=17821 wait=274 +22:32:13 step 171200 loss=2.8203 lr=2.20e-05 tok=22.539B dt=17886 wait=275 +22:33:17 step 171220 loss=2.9416 lr=2.20e-05 tok=22.542B dt=17950 wait=275 +22:34:21 step 171240 loss=2.8051 lr=2.20e-05 tok=22.544B dt=18014 wait=275 +22:35:25 step 171260 loss=2.8745 lr=2.20e-05 tok=22.547B dt=18078 wait=275 +22:36:29 step 171280 loss=2.8592 lr=2.20e-05 tok=22.550B dt=18142 wait=275 +22:37:34 step 171300 loss=2.8296 lr=2.20e-05 tok=22.552B dt=18207 wait=275 +22:37:59 pushed ckpt_w151.pt,log_w151.txt +22:40:46 waited 167s for global r=703 +22:40:54 global r=703 load 4.281 -> 4.279 +22:41:59 step 171320 loss=2.8153 lr=2.20e-05 tok=22.555B dt=18472 wait=275 +22:43:04 step 171340 loss=2.8044 lr=2.20e-05 tok=22.557B dt=18537 wait=275 +22:44:08 step 171360 loss=2.9488 lr=2.19e-05 tok=22.560B dt=18601 wait=276 +22:45:12 step 171380 loss=2.8839 lr=2.19e-05 tok=22.563B dt=18665 wait=276 +22:46:16 step 171400 loss=2.8933 lr=2.19e-05 tok=22.565B dt=18729 wait=276 +22:47:22 step 171420 loss=2.8498 lr=2.19e-05 tok=22.568B dt=18795 wait=276 +22:48:26 step 171440 loss=2.7789 lr=2.19e-05 tok=22.571B dt=18859 wait=276 +22:49:11 pushed ckpt_w151.pt,log_w151.txt +22:52:13 waited 182s for global r=704 +22:52:21 global r=704 load 4.278 -> 4.282 +22:52:54 step 171460 loss=2.8749 lr=2.19e-05 tok=22.573B dt=19127 wait=276 +22:53:59 step 171480 loss=2.8043 lr=2.19e-05 tok=22.576B dt=19192 wait=276 +22:55:03 step 171500 loss=2.8395 lr=2.19e-05 tok=22.578B dt=19256 wait=277 +22:56:07 step 171520 loss=2.8824 lr=2.19e-05 tok=22.581B dt=19320 wait=277 +22:57:12 step 171540 loss=2.6755 lr=2.19e-05 tok=22.584B dt=19385 wait=277 +22:58:16 step 171560 loss=2.8932 lr=2.19e-05 tok=22.586B dt=19449 wait=277 +22:59:21 step 171580 loss=2.7799 lr=2.18e-05 tok=22.589B dt=19514 wait=277 +23:00:24 step 171600 loss=2.8892 lr=2.18e-05 tok=22.592B dt=19577 wait=277 +23:00:39 pushed ckpt_w151.pt,log_w151.txt +23:03:40 waited 181s for global r=705 +23:03:49 global r=705 load 4.280 -> 4.282 +23:04:55 step 171620 loss=2.8608 lr=2.18e-05 tok=22.594B dt=19847 wait=277 +23:05:59 step 171640 loss=2.8553 lr=2.18e-05 tok=22.597B dt=19912 wait=278 +23:07:03 step 171660 loss=2.8026 lr=2.18e-05 tok=22.599B dt=19976 wait=278 +23:08:08 step 171680 loss=2.7479 lr=2.18e-05 tok=22.602B dt=20041 wait=278 +23:09:12 step 171700 loss=2.7210 lr=2.18e-05 tok=22.605B dt=20105 wait=278 +23:10:17 step 171720 loss=2.7727 lr=2.18e-05 tok=22.607B dt=20170 wait=278 +23:11:21 step 171740 loss=2.7667 lr=2.18e-05 tok=22.610B dt=20234 wait=278 +23:12:09 pushed ckpt_w151.pt,log_w151.txt +23:15:45 step 171760 loss=2.8842 lr=2.18e-05 tok=22.613B dt=20498 wait=279 +23:15:55 global r=706 load 4.289 -> 4.279 +23:16:27 step 171760 loss=2.5892 lr=2.18e-05 tok=22.614B dt=20540 wait=279 +23:17:31 step 171780 loss=2.7654 lr=2.18e-05 tok=22.616B dt=20604 wait=279 +23:18:35 step 171800 loss=2.8682 lr=2.17e-05 tok=22.619B dt=20668 wait=279 +23:19:41 step 171820 loss=2.9133 lr=2.17e-05 tok=22.622B dt=20733 wait=279 +23:20:45 step 171840 loss=2.8830 lr=2.17e-05 tok=22.624B dt=20798 wait=279 +23:21:49 step 171860 loss=2.9078 lr=2.17e-05 tok=22.627B dt=20862 wait=279 +23:22:54 step 171880 loss=2.8746 lr=2.17e-05 tok=22.630B dt=20926 wait=279 +23:23:58 step 171900 loss=2.8840 lr=2.17e-05 tok=22.632B dt=20991 wait=280 +23:24:13 pushed ckpt_w151.pt,log_w151.txt +23:27:00 waited 166s for global r=707 +23:27:11 global r=707 load 4.275 -> 4.277 +23:28:16 step 171920 loss=2.8554 lr=2.17e-05 tok=22.635B dt=21249 wait=280 +23:29:20 step 171940 loss=2.8458 lr=2.17e-05 tok=22.637B dt=21313 wait=280 +23:30:25 step 171960 loss=2.8070 lr=2.17e-05 tok=22.640B dt=21378 wait=280 +23:31:28 step 171980 loss=2.8555 lr=2.17e-05 tok=22.643B dt=21441 wait=280 +23:32:32 step 172000 loss=2.7034 lr=2.17e-05 tok=22.645B dt=21505 wait=280 +23:33:36 step 172020 loss=2.9602 lr=2.16e-05 tok=22.648B dt=21569 wait=280 +23:34:41 step 172040 loss=2.7838 lr=2.16e-05 tok=22.651B dt=21634 wait=281 +23:35:32 pushed ckpt_w151.pt,log_w151.txt +23:38:35 waited 183s for global r=708 +23:38:42 global r=708 load 4.293 -> 4.281 +23:39:16 step 172060 loss=2.7925 lr=2.16e-05 tok=22.653B dt=21909 wait=281 +23:40:20 step 172080 loss=2.8002 lr=2.16e-05 tok=22.656B dt=21973 wait=281 +23:41:24 step 172100 loss=2.8792 lr=2.16e-05 tok=22.658B dt=22037 wait=281 +23:42:28 step 172120 loss=2.8258 lr=2.16e-05 tok=22.661B dt=22101 wait=281 +23:43:32 step 172140 loss=2.7508 lr=2.16e-05 tok=22.664B dt=22165 wait=281 +23:44:37 step 172160 loss=2.8062 lr=2.16e-05 tok=22.666B dt=22230 wait=281 +23:45:40 step 172180 loss=2.8347 lr=2.16e-05 tok=22.669B dt=22293 wait=282 +23:46:44 step 172200 loss=2.8243 lr=2.16e-05 tok=22.672B dt=22357 wait=282 +23:47:00 pushed ckpt_w151.pt,log_w151.txt +23:50:02 waited 181s for global r=709 +23:50:12 global r=709 load 4.285 -> 4.280 +23:51:17 step 172220 loss=2.8285 lr=2.16e-05 tok=22.674B dt=22630 wait=282 +23:52:20 step 172240 loss=2.8692 lr=2.15e-05 tok=22.677B dt=22693 wait=282 +23:53:23 step 172260 loss=2.8787 lr=2.15e-05 tok=22.679B dt=22756 wait=282 +23:54:27 step 172280 loss=2.8453 lr=2.15e-05 tok=22.682B dt=22820 wait=282 +23:55:30 step 172300 loss=2.9164 lr=2.15e-05 tok=22.685B dt=22883 wait=283 +23:56:34 step 172320 loss=2.9292 lr=2.15e-05 tok=22.687B dt=22947 wait=283 +23:57:38 step 172340 loss=2.7526 lr=2.15e-05 tok=22.690B dt=23011 wait=283 diff --git a/log_w152.txt b/log_w152.txt new file mode 100644 index 0000000000000000000000000000000000000000..d7530e9301e0ff8613d906b5be8d28da0888ce39 --- /dev/null +++ b/log_w152.txt @@ -0,0 +1,166 @@ +launch +12:25:48 name=w152 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 99/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:25:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:25:54 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:26:04 params 1.08B +12:26:05 resumed step=162750 seen=21.46B round=646 +12:26:05 stream salt=162750 +12:30:07 blocks compiled +12:30:07 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:38:42 step 162751 loss=2.8352 lr=2.65e-05 tok=21.462B dt=515 wait=486 +12:38:58 step 162752 loss=2.8270 lr=2.65e-05 tok=21.462B dt=531 wait=486 +12:39:23 step 162760 loss=3.2761 lr=2.65e-05 tok=21.463B dt=556 wait=486 +12:40:28 step 162780 loss=3.0026 lr=2.65e-05 tok=21.466B dt=621 wait=486 +12:41:34 step 162800 loss=3.0802 lr=2.65e-05 tok=21.468B dt=687 wait=486 +12:42:38 step 162820 loss=3.0859 lr=2.65e-05 tok=21.471B dt=751 wait=486 +12:43:43 step 162840 loss=3.2063 lr=2.65e-05 tok=21.474B dt=816 wait=486 +12:44:48 step 162860 loss=3.2713 lr=2.65e-05 tok=21.476B dt=881 wait=487 +12:45:52 step 162880 loss=3.1632 lr=2.64e-05 tok=21.479B dt=945 wait=487 +12:46:57 step 162900 loss=3.2110 lr=2.64e-05 tok=21.481B dt=1010 wait=487 +12:47:17 pushed ckpt_w152.pt,log_w152.txt +12:48:02 waited 45s for global r=647 +12:48:16 global r=647 load 4.280 -> 4.281 +12:49:25 step 162920 loss=3.1300 lr=2.64e-05 tok=21.484B dt=1158 wait=487 +12:50:30 step 162940 loss=3.2081 lr=2.64e-05 tok=21.487B dt=1223 wait=487 +12:51:36 step 162960 loss=3.3448 lr=2.64e-05 tok=21.489B dt=1289 wait=488 +12:52:42 step 162980 loss=3.1601 lr=2.64e-05 tok=21.492B dt=1355 wait=488 +12:53:47 step 163000 loss=3.1630 lr=2.64e-05 tok=21.494B dt=1420 wait=488 +12:54:53 step 163020 loss=3.2244 lr=2.64e-05 tok=21.497B dt=1486 wait=488 +12:55:57 step 163040 loss=3.1253 lr=2.64e-05 tok=21.500B dt=1550 wait=488 +12:56:44 pushed ckpt_w152.pt,log_w152.txt +12:57:29 waited 45s for global r=648 +12:57:36 global r=648 load 4.271 -> 4.281 +12:58:11 step 163060 loss=3.3101 lr=2.63e-05 tok=21.502B dt=1684 wait=489 +12:59:15 step 163080 loss=3.0958 lr=2.63e-05 tok=21.505B dt=1748 wait=489 +13:00:20 step 163100 loss=3.0304 lr=2.63e-05 tok=21.508B dt=1813 wait=489 +13:01:25 step 163120 loss=3.1998 lr=2.63e-05 tok=21.510B dt=1878 wait=489 +13:02:29 step 163140 loss=3.0354 lr=2.63e-05 tok=21.513B dt=1942 wait=489 +13:03:33 step 163160 loss=3.2351 lr=2.63e-05 tok=21.515B dt=2006 wait=489 +13:04:38 step 163180 loss=3.1376 lr=2.63e-05 tok=21.518B dt=2071 wait=490 +13:05:43 step 163200 loss=3.0945 lr=2.63e-05 tok=21.521B dt=2136 wait=490 +13:05:56 pushed ckpt_w152.pt,log_w152.txt +13:07:12 waited 76s for global r=649 +13:07:20 global r=649 load 4.283 -> 4.278 +13:08:26 step 163220 loss=3.0514 lr=2.62e-05 tok=21.523B dt=2300 wait=490 +13:09:31 step 163240 loss=3.1504 lr=2.62e-05 tok=21.526B dt=2364 wait=490 +13:10:35 step 163260 loss=3.1016 lr=2.62e-05 tok=21.529B dt=2428 wait=490 +13:11:40 step 163280 loss=3.0399 lr=2.62e-05 tok=21.531B dt=2493 wait=490 +13:12:44 step 163300 loss=3.0822 lr=2.62e-05 tok=21.534B dt=2557 wait=491 +13:13:48 step 163320 loss=3.2676 lr=2.62e-05 tok=21.536B dt=2621 wait=491 +13:14:52 step 163340 loss=3.0495 lr=2.62e-05 tok=21.539B dt=2685 wait=491 +13:15:38 pushed ckpt_w152.pt,log_w152.txt +13:16:38 waited 60s for global r=650 +13:16:45 global r=650 load 4.284 -> 4.282 +13:17:18 step 163360 loss=3.1101 lr=2.62e-05 tok=21.542B dt=2831 wait=491 +13:18:22 step 163380 loss=3.1451 lr=2.62e-05 tok=21.544B dt=2895 wait=491 +13:19:27 step 163400 loss=2.9919 lr=2.61e-05 tok=21.547B dt=2961 wait=492 +13:20:31 step 163420 loss=3.2479 lr=2.61e-05 tok=21.550B dt=3024 wait=492 +13:21:36 step 163440 loss=3.1230 lr=2.61e-05 tok=21.552B dt=3089 wait=492 +13:22:40 step 163460 loss=3.0941 lr=2.61e-05 tok=21.555B dt=3153 wait=492 +13:23:45 step 163480 loss=3.1057 lr=2.61e-05 tok=21.557B dt=3218 wait=492 +13:24:49 step 163500 loss=3.1188 lr=2.61e-05 tok=21.560B dt=3282 wait=493 +13:25:01 pushed ckpt_w152.pt,log_w152.txt +13:26:02 waited 61s for global r=651 +13:26:10 global r=651 load 4.284 -> 4.282 +13:27:15 step 163520 loss=3.0885 lr=2.61e-05 tok=21.563B dt=3428 wait=493 +13:28:19 step 163540 loss=3.0993 lr=2.61e-05 tok=21.565B dt=3492 wait=493 +13:29:24 step 163560 loss=3.1458 lr=2.60e-05 tok=21.568B dt=3557 wait=493 +13:30:28 step 163580 loss=3.1806 lr=2.60e-05 tok=21.571B dt=3621 wait=493 +13:31:32 step 163600 loss=3.2161 lr=2.60e-05 tok=21.573B dt=3685 wait=494 +13:32:38 step 163620 loss=2.8911 lr=2.60e-05 tok=21.576B dt=3751 wait=494 +13:33:43 step 163640 loss=3.0184 lr=2.60e-05 tok=21.578B dt=3816 wait=494 +13:34:28 pushed ckpt_w152.pt,log_w152.txt +13:35:29 waited 60s for global r=652 +13:35:34 global r=652 load 4.279 -> 4.278 +13:36:08 step 163660 loss=3.1230 lr=2.60e-05 tok=21.581B dt=3961 wait=494 +13:37:12 step 163680 loss=3.0450 lr=2.60e-05 tok=21.584B dt=4025 wait=495 +13:38:16 step 163700 loss=3.1864 lr=2.60e-05 tok=21.586B dt=4090 wait=495 +13:39:22 step 163720 loss=3.2032 lr=2.60e-05 tok=21.589B dt=4155 wait=495 +13:40:27 step 163740 loss=3.0664 lr=2.59e-05 tok=21.591B dt=4220 wait=495 +13:41:31 step 163760 loss=3.2879 lr=2.59e-05 tok=21.594B dt=4284 wait=495 +13:42:37 step 163780 loss=3.0754 lr=2.59e-05 tok=21.597B dt=4350 wait=495 +13:43:41 step 163800 loss=3.2076 lr=2.59e-05 tok=21.599B dt=4414 wait=496 +13:43:54 pushed ckpt_w152.pt,log_w152.txt +13:44:55 waited 61s for global r=653 +13:45:00 global r=653 load 4.282 -> 4.276 +13:46:05 step 163820 loss=3.1030 lr=2.59e-05 tok=21.602B dt=4558 wait=496 +13:47:09 step 163840 loss=3.2352 lr=2.59e-05 tok=21.605B dt=4622 wait=496 +13:48:13 step 163860 loss=3.2065 lr=2.59e-05 tok=21.607B dt=4686 wait=496 +13:49:18 step 163880 loss=3.0350 lr=2.59e-05 tok=21.610B dt=4751 wait=496 +13:50:22 step 163900 loss=3.0033 lr=2.59e-05 tok=21.612B dt=4815 wait=496 +13:51:26 step 163920 loss=3.1494 lr=2.58e-05 tok=21.615B dt=4879 wait=497 +13:52:30 step 163940 loss=2.9946 lr=2.58e-05 tok=21.618B dt=4943 wait=497 +13:53:15 pushed ckpt_w152.pt,log_w152.txt +13:56:16 waited 181s for global r=654 +13:56:22 global r=654 load 4.281 -> 4.277 +13:56:55 step 163960 loss=3.1044 lr=2.58e-05 tok=21.620B dt=5208 wait=497 +13:58:05 step 163980 loss=1.3085 lr=2.58e-05 tok=21.623B dt=5278 wait=498 +13:59:13 step 164000 loss=1.6162 lr=2.58e-05 tok=21.626B dt=5346 wait=498 +14:00:17 step 164020 loss=3.1144 lr=2.58e-05 tok=21.628B dt=5410 wait=498 +14:01:21 step 164040 loss=3.0439 lr=2.58e-05 tok=21.631B dt=5474 wait=498 +14:02:25 step 164060 loss=3.1690 lr=2.58e-05 tok=21.633B dt=5538 wait=499 +14:03:28 step 164080 loss=3.0559 lr=2.57e-05 tok=21.636B dt=5601 wait=499 +14:04:38 step 164100 loss=2.0927 lr=2.57e-05 tok=21.639B dt=5671 wait=500 +14:04:50 pushed ckpt_w152.pt,log_w152.txt +14:06:05 waited 76s for global r=655 +14:06:12 global r=655 load 4.286 -> 4.278 +14:07:24 step 164120 loss=1.4701 lr=2.57e-05 tok=21.641B dt=5837 wait=500 +14:08:31 step 164140 loss=2.2780 lr=2.57e-05 tok=21.644B dt=5904 wait=501 +14:09:35 step 164160 loss=3.1119 lr=2.57e-05 tok=21.647B dt=5968 wait=501 +14:10:39 step 164180 loss=3.0930 lr=2.57e-05 tok=21.649B dt=6032 wait=501 +14:11:43 step 164200 loss=3.1944 lr=2.57e-05 tok=21.652B dt=6096 wait=501 +14:12:49 step 164220 loss=3.2585 lr=2.57e-05 tok=21.654B dt=6162 wait=501 +14:13:53 step 164240 loss=3.1701 lr=2.57e-05 tok=21.657B dt=6226 wait=502 +14:14:37 pushed ckpt_w152.pt,log_w152.txt +14:15:53 waited 76s for global r=656 +14:16:04 global r=656 load 4.285 -> 4.279 +14:16:38 step 164260 loss=3.2916 lr=2.56e-05 tok=21.660B dt=6391 wait=502 +14:17:42 step 164280 loss=3.2280 lr=2.56e-05 tok=21.662B dt=6455 wait=502 +14:18:46 step 164300 loss=3.2577 lr=2.56e-05 tok=21.665B dt=6519 wait=502 +14:19:49 step 164320 loss=3.3263 lr=2.56e-05 tok=21.668B dt=6582 wait=502 +14:20:54 step 164340 loss=3.0383 lr=2.56e-05 tok=21.670B dt=6647 wait=503 +14:21:58 step 164360 loss=3.3083 lr=2.56e-05 tok=21.673B dt=6711 wait=503 +14:23:02 step 164380 loss=3.1757 lr=2.56e-05 tok=21.675B dt=6775 wait=503 +14:24:06 step 164400 loss=3.3946 lr=2.56e-05 tok=21.678B dt=6839 wait=503 +14:24:18 pushed ckpt_w152.pt,log_w152.txt +14:25:34 waited 76s for global r=657 +14:25:41 global r=657 load 4.272 -> 4.276 +14:26:48 step 164420 loss=3.2635 lr=2.56e-05 tok=21.681B dt=7001 wait=503 +14:27:51 step 164440 loss=3.2788 lr=2.55e-05 tok=21.683B dt=7064 wait=504 +14:28:54 step 164460 loss=3.2638 lr=2.55e-05 tok=21.686B dt=7127 wait=504 +14:29:58 step 164480 loss=3.1622 lr=2.55e-05 tok=21.688B dt=7191 wait=504 +14:31:02 step 164500 loss=3.2416 lr=2.55e-05 tok=21.691B dt=7255 wait=504 +14:32:08 step 164520 loss=2.8949 lr=2.55e-05 tok=21.694B dt=7321 wait=504 +14:33:15 step 164540 loss=2.5944 lr=2.55e-05 tok=21.696B dt=7388 wait=504 +14:34:00 pushed ckpt_w152.pt,log_w152.txt +14:35:15 waited 76s for global r=658 +14:35:20 global r=658 load 4.284 -> 4.278 +14:35:54 step 164560 loss=3.2142 lr=2.55e-05 tok=21.699B dt=7547 wait=505 +14:36:57 step 164580 loss=3.2005 lr=2.55e-05 tok=21.702B dt=7610 wait=505 +14:38:01 step 164600 loss=3.2582 lr=2.55e-05 tok=21.704B dt=7674 wait=505 +14:39:04 step 164620 loss=3.3243 lr=2.54e-05 tok=21.707B dt=7737 wait=505 +14:40:09 step 164640 loss=3.0720 lr=2.54e-05 tok=21.709B dt=7802 wait=505 +14:41:13 step 164660 loss=3.2297 lr=2.54e-05 tok=21.712B dt=7866 wait=506 +14:42:16 step 164680 loss=3.2779 lr=2.54e-05 tok=21.715B dt=7929 wait=506 +14:43:20 step 164700 loss=3.2749 lr=2.54e-05 tok=21.717B dt=7993 wait=506 +14:43:32 pushed ckpt_w152.pt,log_w152.txt +14:44:47 waited 76s for global r=659 +14:44:54 global r=659 load 4.277 -> 4.277 +14:45:59 step 164720 loss=3.3487 lr=2.54e-05 tok=21.720B dt=8152 wait=506 +14:47:03 step 164740 loss=3.1785 lr=2.54e-05 tok=21.723B dt=8216 wait=506 +14:48:06 step 164760 loss=3.1625 lr=2.54e-05 tok=21.725B dt=8279 wait=507 +14:49:10 step 164780 loss=3.0888 lr=2.53e-05 tok=21.728B dt=8343 wait=507 +14:50:12 step 164800 loss=3.2458 lr=2.53e-05 tok=21.730B dt=8405 wait=507 +14:51:16 step 164820 loss=3.1901 lr=2.53e-05 tok=21.733B dt=8469 wait=507 +14:52:19 step 164840 loss=3.0835 lr=2.53e-05 tok=21.736B dt=8532 wait=507 +14:53:06 pushed ckpt_w152.pt,log_w152.txt +14:54:37 waited 91s for global r=660 +14:54:47 global r=660 load 4.278 -> 4.275 +14:55:24 step 164860 loss=2.2429 lr=2.53e-05 tok=21.738B dt=8717 wait=508 +14:56:31 step 164880 loss=2.3008 lr=2.53e-05 tok=21.741B dt=8784 wait=508 +14:57:35 step 164900 loss=3.1285 lr=2.53e-05 tok=21.744B dt=8848 wait=508 +14:58:38 step 164920 loss=3.1066 lr=2.53e-05 tok=21.746B dt=8911 wait=509 +14:59:42 step 164940 loss=3.0155 lr=2.53e-05 tok=21.749B dt=8975 wait=509 +15:00:45 step 164960 loss=3.1919 lr=2.52e-05 tok=21.751B dt=9038 wait=509 +15:01:48 step 164980 loss=3.0127 lr=2.52e-05 tok=21.754B dt=9101 wait=509 +15:02:52 step 165000 loss=3.1702 lr=2.52e-05 tok=21.757B dt=9165 wait=509 diff --git a/log_w153.txt b/log_w153.txt new file mode 100644 index 0000000000000000000000000000000000000000..8dfe737d9e059e3524288507f42952ce2e18b837 --- /dev/null +++ b/log_w153.txt @@ -0,0 +1,575 @@ +launch +12:28:34 name=w153 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 100/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:28:40 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:28:40 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:28:49 params 1.08B +12:28:51 resumed step=162750 seen=21.46B round=646 +12:28:51 stream salt=162750 +12:33:42 blocks compiled +12:33:42 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:39:27 step 162751 loss=3.1273 lr=2.65e-05 tok=21.462B dt=345 wait=318 +12:39:43 step 162752 loss=3.3741 lr=2.65e-05 tok=21.462B dt=361 wait=318 +12:40:09 step 162760 loss=3.7669 lr=2.65e-05 tok=21.463B dt=387 wait=318 +12:41:13 step 162780 loss=3.4846 lr=2.65e-05 tok=21.466B dt=451 wait=318 +12:42:18 step 162800 loss=3.3950 lr=2.65e-05 tok=21.468B dt=515 wait=318 +12:43:21 step 162820 loss=3.4313 lr=2.65e-05 tok=21.471B dt=578 wait=319 +12:44:23 step 162840 loss=3.4299 lr=2.65e-05 tok=21.474B dt=641 wait=319 +12:45:27 step 162860 loss=3.5153 lr=2.65e-05 tok=21.476B dt=705 wait=319 +12:46:31 step 162880 loss=3.4005 lr=2.64e-05 tok=21.479B dt=769 wait=319 +12:47:34 step 162900 loss=3.4583 lr=2.64e-05 tok=21.481B dt=832 wait=319 +12:47:48 pushed ckpt_w153.pt,log_w153.txt +12:48:03 waited 15s for global r=647 +12:48:19 global r=647 load 4.278 -> 4.281 +12:49:25 step 162920 loss=3.3171 lr=2.64e-05 tok=21.484B dt=943 wait=320 +12:50:37 step 162940 loss=3.2737 lr=2.64e-05 tok=21.487B dt=1015 wait=320 +12:51:52 step 162960 loss=3.2962 lr=2.64e-05 tok=21.489B dt=1090 wait=320 +12:52:55 step 162980 loss=3.3006 lr=2.64e-05 tok=21.492B dt=1153 wait=320 +12:53:58 step 163000 loss=3.2535 lr=2.64e-05 tok=21.494B dt=1216 wait=320 +12:55:02 step 163020 loss=3.3864 lr=2.64e-05 tok=21.497B dt=1280 wait=321 +12:56:06 step 163040 loss=3.1998 lr=2.64e-05 tok=21.500B dt=1344 wait=321 +12:56:50 pushed ckpt_w153.pt,log_w153.txt +12:57:36 waited 45s for global r=648 +12:57:44 global r=648 load 4.292 -> 4.281 +12:58:18 step 163060 loss=3.4123 lr=2.63e-05 tok=21.502B dt=1476 wait=321 +12:59:22 step 163080 loss=3.2394 lr=2.63e-05 tok=21.505B dt=1540 wait=321 +13:00:42 step 163100 loss=3.3089 lr=2.63e-05 tok=21.508B dt=1620 wait=322 +13:01:51 step 163120 loss=3.4193 lr=2.63e-05 tok=21.510B dt=1689 wait=322 +13:02:55 step 163140 loss=3.3185 lr=2.63e-05 tok=21.513B dt=1753 wait=322 +13:03:58 step 163160 loss=3.4396 lr=2.63e-05 tok=21.515B dt=1816 wait=322 +13:05:03 step 163180 loss=3.2551 lr=2.63e-05 tok=21.518B dt=1880 wait=322 +13:06:06 step 163200 loss=3.3209 lr=2.63e-05 tok=21.521B dt=1944 wait=323 +13:06:19 pushed ckpt_w153.pt,log_w153.txt +13:07:04 waited 45s for global r=649 +13:07:14 global r=649 load 4.278 -> 4.278 +13:08:19 step 163220 loss=3.2227 lr=2.62e-05 tok=21.523B dt=2077 wait=323 +13:09:29 step 163240 loss=3.2711 lr=2.62e-05 tok=21.526B dt=2147 wait=323 +13:10:47 step 163260 loss=3.2640 lr=2.62e-05 tok=21.529B dt=2225 wait=323 +13:11:51 step 163280 loss=3.1561 lr=2.62e-05 tok=21.531B dt=2289 wait=323 +13:12:54 step 163300 loss=3.1831 lr=2.62e-05 tok=21.534B dt=2351 wait=324 +13:13:57 step 163320 loss=3.3298 lr=2.62e-05 tok=21.536B dt=2414 wait=324 +13:14:59 step 163340 loss=3.1952 lr=2.62e-05 tok=21.539B dt=2477 wait=324 +13:15:43 pushed ckpt_w153.pt,log_w153.txt +13:16:43 waited 60s for global r=650 +13:16:50 global r=650 load 4.288 -> 4.282 +13:17:31 step 163360 loss=3.2785 lr=2.62e-05 tok=21.542B dt=2629 wait=324 +13:18:46 step 163380 loss=3.3019 lr=2.62e-05 tok=21.544B dt=2704 wait=324 +13:19:49 step 163400 loss=3.1978 lr=2.61e-05 tok=21.547B dt=2767 wait=324 +13:20:53 step 163420 loss=3.1985 lr=2.61e-05 tok=21.550B dt=2830 wait=325 +13:21:56 step 163440 loss=3.2143 lr=2.61e-05 tok=21.552B dt=2894 wait=325 +13:22:59 step 163460 loss=3.2700 lr=2.61e-05 tok=21.555B dt=2957 wait=325 +13:24:03 step 163480 loss=3.1585 lr=2.61e-05 tok=21.557B dt=3021 wait=325 +13:25:06 step 163500 loss=3.3243 lr=2.61e-05 tok=21.560B dt=3084 wait=325 +13:25:19 pushed ckpt_w153.pt,log_w153.txt +13:26:04 waited 45s for global r=651 +13:26:10 global r=651 load 4.285 -> 4.282 +13:27:29 step 163520 loss=3.3029 lr=2.61e-05 tok=21.563B dt=3226 wait=326 +13:28:38 step 163540 loss=3.3112 lr=2.61e-05 tok=21.565B dt=3296 wait=326 +13:29:42 step 163560 loss=3.2054 lr=2.60e-05 tok=21.568B dt=3360 wait=326 +13:30:46 step 163580 loss=3.1023 lr=2.60e-05 tok=21.571B dt=3424 wait=326 +13:31:50 step 163600 loss=3.2241 lr=2.60e-05 tok=21.573B dt=3488 wait=326 +13:32:54 step 163620 loss=3.1775 lr=2.60e-05 tok=21.576B dt=3552 wait=327 +13:33:58 step 163640 loss=3.0815 lr=2.60e-05 tok=21.578B dt=3615 wait=327 +13:34:41 pushed ckpt_w153.pt,log_w153.txt +13:35:27 waited 45s for global r=652 +13:35:33 global r=652 load 4.275 -> 4.278 +13:36:06 step 163660 loss=3.2141 lr=2.60e-05 tok=21.581B dt=3744 wait=327 +13:37:30 step 163680 loss=3.1888 lr=2.60e-05 tok=21.584B dt=3828 wait=327 +13:38:34 step 163700 loss=3.1213 lr=2.60e-05 tok=21.586B dt=3892 wait=327 +13:39:38 step 163720 loss=3.1577 lr=2.60e-05 tok=21.589B dt=3956 wait=328 +13:40:42 step 163740 loss=3.0938 lr=2.59e-05 tok=21.591B dt=4020 wait=328 +13:41:46 step 163760 loss=3.1540 lr=2.59e-05 tok=21.594B dt=4084 wait=328 +13:42:50 step 163780 loss=3.0436 lr=2.59e-05 tok=21.597B dt=4148 wait=328 +13:43:54 step 163800 loss=3.1458 lr=2.59e-05 tok=21.599B dt=4212 wait=328 +13:44:08 pushed ckpt_w153.pt,log_w153.txt +13:44:53 waited 45s for global r=653 +13:44:59 global r=653 load 4.276 -> 4.276 +13:46:13 step 163820 loss=3.1884 lr=2.59e-05 tok=21.602B dt=4350 wait=329 +13:47:29 step 163840 loss=3.1830 lr=2.59e-05 tok=21.605B dt=4427 wait=329 +13:48:32 step 163860 loss=3.2142 lr=2.59e-05 tok=21.607B dt=4490 wait=329 +13:49:36 step 163880 loss=3.0609 lr=2.59e-05 tok=21.610B dt=4554 wait=329 +13:50:40 step 163900 loss=3.1053 lr=2.59e-05 tok=21.612B dt=4618 wait=329 +13:51:44 step 163920 loss=3.1647 lr=2.58e-05 tok=21.615B dt=4682 wait=330 +13:52:48 step 163940 loss=3.0560 lr=2.58e-05 tok=21.618B dt=4745 wait=330 +13:53:31 pushed ckpt_w153.pt,log_w153.txt +13:56:17 waited 166s for global r=654 +13:56:23 global r=654 load 4.287 -> 4.277 +13:56:56 step 163960 loss=3.2229 lr=2.58e-05 tok=21.620B dt=4994 wait=330 +13:58:08 step 163980 loss=3.1030 lr=2.58e-05 tok=21.623B dt=5065 wait=330 +13:59:24 step 164000 loss=3.1708 lr=2.58e-05 tok=21.626B dt=5142 wait=330 +14:00:28 step 164020 loss=3.2140 lr=2.58e-05 tok=21.628B dt=5206 wait=331 +14:01:33 step 164040 loss=3.0935 lr=2.58e-05 tok=21.631B dt=5270 wait=331 +14:02:36 step 164060 loss=3.2036 lr=2.58e-05 tok=21.633B dt=5334 wait=331 +14:03:39 step 164080 loss=3.0468 lr=2.57e-05 tok=21.636B dt=5397 wait=331 +14:04:42 step 164100 loss=3.0576 lr=2.57e-05 tok=21.639B dt=5460 wait=331 +14:04:54 pushed ckpt_w153.pt,log_w153.txt +14:06:10 waited 76s for global r=655 +14:06:16 global r=655 load 4.280 -> 4.278 +14:07:28 step 164120 loss=3.0527 lr=2.57e-05 tok=21.641B dt=5626 wait=332 +14:08:44 step 164140 loss=3.1024 lr=2.57e-05 tok=21.644B dt=5702 wait=332 +14:09:47 step 164160 loss=3.0785 lr=2.57e-05 tok=21.647B dt=5765 wait=332 +14:10:51 step 164180 loss=3.0017 lr=2.57e-05 tok=21.649B dt=5829 wait=332 +14:11:54 step 164200 loss=3.0913 lr=2.57e-05 tok=21.652B dt=5892 wait=332 +14:12:57 step 164220 loss=3.1749 lr=2.57e-05 tok=21.654B dt=5955 wait=333 +14:14:01 step 164240 loss=3.0439 lr=2.57e-05 tok=21.657B dt=6018 wait=333 +14:14:46 pushed ckpt_w153.pt,log_w153.txt +14:15:47 waited 61s for global r=656 +14:15:54 global r=656 load 4.275 -> 4.279 +14:16:28 step 164260 loss=3.2339 lr=2.56e-05 tok=21.660B dt=6165 wait=333 +14:17:45 step 164280 loss=3.0830 lr=2.56e-05 tok=21.662B dt=6243 wait=333 +14:18:55 step 164300 loss=3.0580 lr=2.56e-05 tok=21.665B dt=6313 wait=333 +14:19:58 step 164320 loss=3.2524 lr=2.56e-05 tok=21.668B dt=6375 wait=334 +14:21:01 step 164340 loss=3.0498 lr=2.56e-05 tok=21.670B dt=6439 wait=334 +14:22:04 step 164360 loss=3.2051 lr=2.56e-05 tok=21.673B dt=6502 wait=334 +14:23:08 step 164380 loss=3.0491 lr=2.56e-05 tok=21.675B dt=6566 wait=334 +14:24:11 step 164400 loss=3.1170 lr=2.56e-05 tok=21.678B dt=6629 wait=334 +14:24:23 pushed ckpt_w153.pt,log_w153.txt +14:25:39 waited 76s for global r=657 +14:25:46 global r=657 load 4.278 -> 4.276 +14:26:58 step 164420 loss=3.1399 lr=2.56e-05 tok=21.681B dt=6796 wait=335 +14:28:14 step 164440 loss=3.0720 lr=2.55e-05 tok=21.683B dt=6872 wait=335 +14:29:18 step 164460 loss=3.0509 lr=2.55e-05 tok=21.686B dt=6936 wait=335 +14:30:23 step 164480 loss=2.9332 lr=2.55e-05 tok=21.688B dt=7000 wait=335 +14:31:26 step 164500 loss=2.9646 lr=2.55e-05 tok=21.691B dt=7064 wait=335 +14:32:30 step 164520 loss=3.0156 lr=2.55e-05 tok=21.694B dt=7128 wait=336 +14:33:34 step 164540 loss=2.9389 lr=2.55e-05 tok=21.696B dt=7192 wait=336 +14:34:18 pushed ckpt_w153.pt,log_w153.txt +14:35:03 waited 45s for global r=658 +14:35:11 global r=658 load 4.286 -> 4.278 +14:35:45 step 164560 loss=3.1167 lr=2.55e-05 tok=21.699B dt=7323 wait=336 +14:36:56 step 164580 loss=3.0506 lr=2.55e-05 tok=21.702B dt=7394 wait=336 +14:38:00 step 164600 loss=2.9739 lr=2.55e-05 tok=21.704B dt=7458 wait=336 +14:39:03 step 164620 loss=3.0796 lr=2.54e-05 tok=21.707B dt=7521 wait=337 +14:40:06 step 164640 loss=3.0655 lr=2.54e-05 tok=21.709B dt=7584 wait=337 +14:41:10 step 164660 loss=3.0524 lr=2.54e-05 tok=21.712B dt=7648 wait=337 +14:42:13 step 164680 loss=2.9141 lr=2.54e-05 tok=21.715B dt=7711 wait=337 +14:43:17 step 164700 loss=3.0672 lr=2.54e-05 tok=21.717B dt=7775 wait=337 +14:43:30 pushed ckpt_w153.pt,log_w153.txt +14:44:45 waited 76s for global r=659 +14:44:53 global r=659 load 4.282 -> 4.277 +14:45:57 step 164720 loss=2.9967 lr=2.54e-05 tok=21.720B dt=7935 wait=337 +14:47:07 step 164740 loss=3.0809 lr=2.54e-05 tok=21.723B dt=8004 wait=338 +14:48:24 step 164760 loss=3.0321 lr=2.54e-05 tok=21.725B dt=8082 wait=338 +14:49:27 step 164780 loss=2.9774 lr=2.53e-05 tok=21.728B dt=8145 wait=338 +14:50:30 step 164800 loss=3.0260 lr=2.53e-05 tok=21.730B dt=8208 wait=338 +14:51:33 step 164820 loss=3.0613 lr=2.53e-05 tok=21.733B dt=8271 wait=339 +14:52:36 step 164840 loss=2.9254 lr=2.53e-05 tok=21.736B dt=8334 wait=339 +14:53:22 pushed ckpt_w153.pt,log_w153.txt +14:54:38 waited 76s for global r=660 +14:54:49 global r=660 load 4.278 -> 4.275 +14:55:23 step 164860 loss=3.0636 lr=2.53e-05 tok=21.738B dt=8501 wait=339 +14:56:25 step 164880 loss=2.9985 lr=2.53e-05 tok=21.741B dt=8563 wait=339 +14:57:49 step 164900 loss=2.9295 lr=2.53e-05 tok=21.744B dt=8647 wait=339 +14:58:54 step 164920 loss=3.1114 lr=2.53e-05 tok=21.746B dt=8711 wait=340 +14:59:57 step 164940 loss=2.9306 lr=2.53e-05 tok=21.749B dt=8775 wait=340 +15:01:01 step 164960 loss=3.0520 lr=2.52e-05 tok=21.751B dt=8838 wait=340 +15:02:04 step 164980 loss=2.9339 lr=2.52e-05 tok=21.754B dt=8902 wait=340 +15:03:07 step 165000 loss=3.0220 lr=2.52e-05 tok=21.757B dt=8965 wait=340 +15:03:24 pushed ckpt_w153.pt,log_w153.txt +15:04:39 waited 76s for global r=661 +15:04:47 global r=661 load 4.281 -> 4.275 +15:05:51 step 165020 loss=2.9618 lr=2.52e-05 tok=21.759B dt=9129 wait=341 +15:07:02 step 165040 loss=3.0232 lr=2.52e-05 tok=21.762B dt=9200 wait=341 +15:08:18 step 165060 loss=3.0918 lr=2.52e-05 tok=21.765B dt=9276 wait=341 +15:09:21 step 165080 loss=3.0995 lr=2.52e-05 tok=21.767B dt=9338 wait=341 +15:10:24 step 165100 loss=3.0524 lr=2.52e-05 tok=21.770B dt=9401 wait=342 +15:11:27 step 165120 loss=3.0524 lr=2.52e-05 tok=21.772B dt=9465 wait=342 +15:12:30 step 165140 loss=3.0261 lr=2.51e-05 tok=21.775B dt=9528 wait=342 +15:13:14 pushed ckpt_w153.pt,log_w153.txt +15:16:00 waited 166s for global r=662 +15:16:08 global r=662 load 4.275 -> 4.277 +15:16:41 step 165160 loss=3.0772 lr=2.51e-05 tok=21.778B dt=9779 wait=342 +15:17:43 step 165180 loss=3.0683 lr=2.51e-05 tok=21.780B dt=9841 wait=342 +15:19:02 step 165200 loss=3.0726 lr=2.51e-05 tok=21.783B dt=9920 wait=342 +15:20:10 step 165220 loss=3.1619 lr=2.51e-05 tok=21.785B dt=9988 wait=343 +15:21:13 step 165240 loss=3.0610 lr=2.51e-05 tok=21.788B dt=10051 wait=343 +15:22:17 step 165260 loss=2.9927 lr=2.51e-05 tok=21.791B dt=10115 wait=343 +15:23:21 step 165280 loss=3.0447 lr=2.51e-05 tok=21.793B dt=10179 wait=343 +15:24:25 step 165300 loss=3.0277 lr=2.51e-05 tok=21.796B dt=10243 wait=343 +15:24:44 pushed ckpt_w153.pt,log_w153.txt +15:25:30 waited 45s for global r=663 +15:25:41 global r=663 load 4.285 -> 4.276 +15:26:46 step 165320 loss=3.0157 lr=2.50e-05 tok=21.799B dt=10384 wait=344 +15:27:56 step 165340 loss=3.0469 lr=2.50e-05 tok=21.801B dt=10453 wait=344 +15:29:14 step 165360 loss=2.9971 lr=2.50e-05 tok=21.804B dt=10531 wait=344 +15:30:17 step 165380 loss=2.9740 lr=2.50e-05 tok=21.806B dt=10595 wait=344 +15:31:22 step 165400 loss=2.9080 lr=2.50e-05 tok=21.809B dt=10660 wait=344 +15:32:26 step 165420 loss=3.0163 lr=2.50e-05 tok=21.812B dt=10723 wait=345 +15:33:30 step 165440 loss=2.8806 lr=2.50e-05 tok=21.814B dt=10788 wait=345 +15:34:16 pushed ckpt_w153.pt,log_w153.txt +15:35:02 waited 46s for global r=664 +15:35:11 global r=664 load 4.284 -> 4.273 +15:35:53 step 165460 loss=2.9880 lr=2.50e-05 tok=21.817B dt=10931 wait=345 +15:37:10 step 165480 loss=2.8684 lr=2.50e-05 tok=21.820B dt=11008 wait=345 +15:38:14 step 165500 loss=2.8519 lr=2.49e-05 tok=21.822B dt=11072 wait=345 +15:39:18 step 165520 loss=3.0893 lr=2.49e-05 tok=21.825B dt=11136 wait=346 +15:40:20 step 165540 loss=2.9738 lr=2.49e-05 tok=21.827B dt=11198 wait=346 +15:41:24 step 165560 loss=3.0661 lr=2.49e-05 tok=21.830B dt=11261 wait=346 +15:42:27 step 165580 loss=2.9982 lr=2.49e-05 tok=21.833B dt=11325 wait=346 +15:43:30 step 165600 loss=2.9953 lr=2.49e-05 tok=21.835B dt=11388 wait=346 +15:43:43 pushed ckpt_w153.pt,log_w153.txt +15:44:43 waited 61s for global r=665 +15:44:55 global r=665 load 4.279 -> 4.275 +15:46:17 step 165620 loss=3.0111 lr=2.49e-05 tok=21.838B dt=11554 wait=347 +15:47:22 step 165640 loss=3.0940 lr=2.49e-05 tok=21.841B dt=11620 wait=347 +15:48:25 step 165660 loss=3.1271 lr=2.49e-05 tok=21.843B dt=11683 wait=347 +15:49:28 step 165680 loss=3.0237 lr=2.48e-05 tok=21.846B dt=11746 wait=347 +15:50:31 step 165700 loss=3.0560 lr=2.48e-05 tok=21.848B dt=11808 wait=347 +15:51:33 step 165720 loss=3.1994 lr=2.48e-05 tok=21.851B dt=11871 wait=348 +15:52:36 step 165740 loss=2.9968 lr=2.48e-05 tok=21.854B dt=11934 wait=348 +15:53:21 pushed ckpt_w153.pt,log_w153.txt +15:54:06 waited 45s for global r=666 +15:54:13 global r=666 load 4.277 -> 4.276 +15:54:46 step 165760 loss=3.0776 lr=2.48e-05 tok=21.856B dt=12064 wait=348 +15:56:09 step 165780 loss=3.0690 lr=2.48e-05 tok=21.859B dt=12147 wait=348 +15:57:12 step 165800 loss=3.0103 lr=2.48e-05 tok=21.861B dt=12209 wait=349 +15:58:14 step 165820 loss=3.1211 lr=2.48e-05 tok=21.864B dt=12272 wait=349 +15:59:17 step 165840 loss=3.0090 lr=2.48e-05 tok=21.867B dt=12335 wait=349 +16:00:19 step 165860 loss=3.1745 lr=2.47e-05 tok=21.869B dt=12397 wait=349 +16:01:21 step 165880 loss=3.1152 lr=2.47e-05 tok=21.872B dt=12459 wait=349 +16:02:23 step 165900 loss=3.1085 lr=2.47e-05 tok=21.875B dt=12521 wait=349 +16:02:35 pushed ckpt_w153.pt,log_w153.txt +16:03:51 waited 76s for global r=667 +16:03:58 global r=667 load 4.276 -> 4.273 +16:05:16 step 165920 loss=2.9178 lr=2.47e-05 tok=21.877B dt=12694 wait=350 +16:06:25 step 165940 loss=3.0066 lr=2.47e-05 tok=21.880B dt=12763 wait=350 +16:07:28 step 165960 loss=3.0590 lr=2.47e-05 tok=21.882B dt=12826 wait=350 +16:08:32 step 165980 loss=3.0568 lr=2.47e-05 tok=21.885B dt=12889 wait=350 +16:09:34 step 166000 loss=2.9990 lr=2.47e-05 tok=21.888B dt=12952 wait=350 +16:10:37 step 166020 loss=3.1366 lr=2.47e-05 tok=21.890B dt=13015 wait=351 +16:11:41 step 166040 loss=3.0053 lr=2.47e-05 tok=21.893B dt=13078 wait=351 +16:12:24 pushed ckpt_w153.pt,log_w153.txt +16:15:10 waited 166s for global r=668 +16:15:18 global r=668 load 4.274 -> 4.276 +16:15:51 step 166060 loss=3.0976 lr=2.46e-05 tok=21.896B dt=13328 wait=351 +16:17:14 step 166080 loss=2.9852 lr=2.46e-05 tok=21.898B dt=13411 wait=351 +16:18:18 step 166100 loss=2.9624 lr=2.46e-05 tok=21.901B dt=13475 wait=352 +16:19:21 step 166120 loss=3.0373 lr=2.46e-05 tok=21.903B dt=13538 wait=352 +16:20:24 step 166140 loss=2.9860 lr=2.46e-05 tok=21.906B dt=13602 wait=352 +16:21:27 step 166160 loss=3.0411 lr=2.46e-05 tok=21.909B dt=13665 wait=352 +16:22:30 step 166180 loss=2.9792 lr=2.46e-05 tok=21.911B dt=13728 wait=352 +16:23:33 step 166200 loss=3.0275 lr=2.46e-05 tok=21.914B dt=13791 wait=352 +16:23:47 pushed ckpt_w153.pt,log_w153.txt +16:27:30 global r=669 load 4.282 -> 4.276 +16:28:52 step 166220 loss=2.9508 lr=2.46e-05 tok=21.918B dt=14110 wait=353 +16:29:57 step 166240 loss=2.9794 lr=2.45e-05 tok=21.920B dt=14174 wait=353 +16:30:59 step 166260 loss=3.0056 lr=2.45e-05 tok=21.923B dt=14237 wait=353 +16:32:02 step 166280 loss=2.9473 lr=2.45e-05 tok=21.926B dt=14300 wait=353 +16:33:05 step 166300 loss=2.9612 lr=2.45e-05 tok=21.928B dt=14363 wait=354 +16:34:08 step 166320 loss=3.0232 lr=2.45e-05 tok=21.931B dt=14426 wait=354 +16:35:11 step 166340 loss=2.9982 lr=2.45e-05 tok=21.934B dt=14488 wait=354 +16:35:55 pushed ckpt_w153.pt,log_w153.txt +16:38:42 waited 166s for global r=670 +16:38:48 global r=670 load 4.288 -> 4.282 +16:39:27 step 166360 loss=3.0203 lr=2.45e-05 tok=21.936B dt=14744 wait=354 +16:40:44 step 166380 loss=2.9443 lr=2.45e-05 tok=21.939B dt=14822 wait=354 +16:41:48 step 166400 loss=2.9745 lr=2.45e-05 tok=21.941B dt=14885 wait=355 +16:42:51 step 166420 loss=3.0261 lr=2.44e-05 tok=21.944B dt=14949 wait=355 +16:43:55 step 166440 loss=3.0011 lr=2.44e-05 tok=21.947B dt=15013 wait=355 +16:45:00 step 166460 loss=3.0873 lr=2.44e-05 tok=21.949B dt=15077 wait=355 +16:46:04 step 166480 loss=3.0304 lr=2.44e-05 tok=21.952B dt=15141 wait=355 +16:47:07 step 166500 loss=3.0541 lr=2.44e-05 tok=21.955B dt=15205 wait=356 +16:47:20 pushed ckpt_w153.pt,log_w153.txt +launch +16:48:20 name=w153 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 100/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:48:25 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:48:26 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:48:35 params 1.08B +16:48:36 resumed step=166350 seen=21.93B round=670 +16:48:36 stream salt=166350 +16:50:28 blocks compiled +16:50:28 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:54:58 step 166351 loss=2.9844 lr=2.45e-05 tok=21.935B dt=270 wait=261 +16:55:05 step 166352 loss=3.2453 lr=2.45e-05 tok=21.935B dt=277 wait=261 +16:55:30 step 166360 loss=3.3561 lr=2.45e-05 tok=21.936B dt=303 wait=261 +16:55:45 global r=671 load 4.282 -> 4.284 +16:56:49 step 166520 loss=3.2289 lr=2.44e-05 tok=21.939B dt=382 wait=261 +16:57:54 step 166540 loss=3.3544 lr=2.44e-05 tok=21.941B dt=446 wait=261 +16:58:58 step 166560 loss=3.3764 lr=2.44e-05 tok=21.944B dt=510 wait=261 +17:00:01 step 166580 loss=3.2819 lr=2.44e-05 tok=21.947B dt=574 wait=261 +17:01:06 step 166600 loss=3.2637 lr=2.43e-05 tok=21.949B dt=638 wait=262 +17:02:10 step 166620 loss=3.2567 lr=2.43e-05 tok=21.952B dt=702 wait=262 +17:03:14 step 166640 loss=3.1823 lr=2.43e-05 tok=21.955B dt=767 wait=262 +17:04:00 pushed ckpt_w153.pt,log_w153.txt +17:04:46 waited 45s for global r=672 +17:04:56 global r=672 load 4.280 -> 4.280 +17:05:30 step 166660 loss=3.3283 lr=2.43e-05 tok=21.957B dt=903 wait=262 +17:06:50 step 166680 loss=3.2152 lr=2.43e-05 tok=21.960B dt=982 wait=262 +17:08:00 step 166700 loss=3.3120 lr=2.43e-05 tok=21.962B dt=1052 wait=262 +17:09:05 step 166720 loss=3.2679 lr=2.43e-05 tok=21.965B dt=1117 wait=263 +17:09:13 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-health-2e12/health_p070_shard_00003626.jsonl.zst: (Request ID: Root=1-6abbf0b9-33be2ea20827d4662640f540;3ccd80b3-74cb-4619-9318-5587400c0421) + +429 Too Many Requests: you +17:10:04 dolma file skipped data/ingredient1-cranecode/shard_00000167.jsonl.zst: (Request ID: Root=1-6abbf0ec-2c0db4c25b73b4c972800585;d64afa1c-3261-408c-bc78-715e8da18811) + +429 Too Many Requests: you +17:10:10 step 166740 loss=3.0848 lr=2.43e-05 tok=21.968B dt=1182 wait=263 +17:11:14 step 166760 loss=3.3043 lr=2.43e-05 tok=21.970B dt=1246 wait=263 +17:12:18 step 166780 loss=3.2895 lr=2.42e-05 tok=21.973B dt=1311 wait=263 +17:13:22 step 166800 loss=3.1910 lr=2.42e-05 tok=21.976B dt=1374 wait=263 +17:13:34 pushed ckpt_w153.pt,log_w153.txt +17:16:36 waited 181s for global r=673 +17:16:45 global r=673 load 4.270 -> 4.268 +17:17:59 step 166820 loss=3.2220 lr=2.42e-05 tok=21.978B dt=1651 wait=264 +17:19:20 step 166840 loss=3.1850 lr=2.42e-05 tok=21.981B dt=1732 wait=264 +launch +17:19:55 name=w153 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 100/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:20:00 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:20:00 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:20:10 params 1.08B +17:20:11 resumed step=166800 seen=21.98B round=673 +17:20:11 stream salt=166800 +17:21:58 blocks compiled +17:21:58 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:23:46 mix source 5 paused 30 min ((Request ID: Root=1-6abbf422-637eaef05b4a88ff52677682;d7808b9e-91a3-4bb3-bcdb-bd) +17:23:46 mix source 1 paused 30 min ((Request ID: Root=1-6abbf422-638ee59a5608c4b01c7ca6ae;7df9f720-4b88-4c62-a2e9-f9) +17:23:46 mix source 6 paused 30 min ((Request ID: Root=1-6abbf422-16096e2f464a53bd117171cf;074182a6-5884-4303-a6cd-60) +17:23:46 mix source 8 paused 30 min ((Request ID: Root=1-6abbf422-6d23033b5625dcb909b648f6;9b4c02e8-864a-4413-82d8-2a) +17:23:46 mix source 7 paused 30 min ((Request ID: Root=1-6abbf422-0fceb504369186830682e595;c6bc901f-60e5-41e5-a13e-cc) +17:23:47 dolma file skipped data/ingredient1-common_crawl-high-quality_20_finance_and_business/shard_00000274.jsonl.zst: (Request ID: Root=1-6abbf423-195e6c1a465dbb7029ec9b15;c879bf87-d5dc-4b86-a322-8bf59f5875e4) + +429 Too Many Requests: you +17:23:47 dolma file skipped data/ingredient1-common_crawl-high-quality_20_entertainment/shard_00000655.jsonl.zst: (Request ID: Root=1-6abbf423-1d29ee09471cb80b495580ed;d90491c9-33fa-4cab-aa90-740d42676bed) + +429 Too Many Requests: you +17:23:47 dolma file skipped data/ingredient1-reddit_to_flashcards/merged_qa_prefilter_densesubs_lowthresh-0074_f4.jsonl.zst: (Request ID: Root=1-6abbf423-1f3db3203be5970c33bb027c;725d2360-fd70-4626-a7d5-fea9d939a5a9) + +429 Too Many Requests: you +17:23:47 dolma file skipped data/ingredient1-common_crawl-high-quality_19_finance_and_business/shard_00000650.jsonl.zst: (Request ID: Root=1-6abbf423-3330615767f9b2e337d6031d;b7bfae58-db90-408d-8651-5674fbf90592) + +429 Too Many Requests: you +17:24:05 step 166801 loss=2.9516 lr=2.42e-05 tok=21.976B dt=127 wait=107 +17:24:17 step 166802 loss=3.1115 lr=2.42e-05 tok=21.976B dt=139 wait=107 +17:25:19 step 166820 loss=3.2899 lr=2.42e-05 tok=21.978B dt=201 wait=108 +17:26:25 step 166840 loss=3.2496 lr=2.42e-05 tok=21.981B dt=267 wait=109 +17:27:29 step 166860 loss=3.3540 lr=2.42e-05 tok=21.983B dt=331 wait=109 +17:28:32 step 166880 loss=3.2339 lr=2.42e-05 tok=21.986B dt=394 wait=109 +17:29:37 step 166900 loss=3.3546 lr=2.42e-05 tok=21.989B dt=459 wait=109 +17:30:41 step 166920 loss=3.2792 lr=2.42e-05 tok=21.991B dt=523 wait=110 +17:31:45 step 166940 loss=3.3294 lr=2.42e-05 tok=21.994B dt=587 wait=110 +17:32:31 pushed ckpt_w153.pt,log_w153.txt +launch +17:34:41 name=w153 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 100/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:34:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:34:46 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:34:56 params 1.08B +17:34:58 resumed step=166800 seen=21.98B round=673 +17:34:58 stream salt=166800 +17:36:46 blocks compiled +17:36:46 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:40:54 step 166801 loss=2.7487 lr=2.42e-05 tok=21.976B dt=249 wait=233 +17:41:02 step 166802 loss=2.9373 lr=2.42e-05 tok=21.976B dt=257 wait=233 +17:41:44 global r=674 load 4.273 -> 4.258 +17:42:17 step 166960 loss=3.2454 lr=2.42e-05 tok=21.978B dt=331 wait=233 +17:43:22 step 166980 loss=2.9932 lr=2.41e-05 tok=21.981B dt=397 wait=234 +17:44:27 step 167000 loss=3.0671 lr=2.41e-05 tok=21.983B dt=461 wait=234 +17:45:31 step 167020 loss=3.1948 lr=2.41e-05 tok=21.986B dt=526 wait=234 +17:46:36 step 167040 loss=3.1278 lr=2.41e-05 tok=21.989B dt=590 wait=234 +17:47:40 step 167060 loss=3.1713 lr=2.41e-05 tok=21.991B dt=655 wait=234 +17:47:53 global r=675 load 4.284 -> 4.260 +17:49:16 step 167120 loss=3.1032 lr=2.41e-05 tok=21.994B dt=751 wait=234 +17:50:22 step 167140 loss=3.2180 lr=2.41e-05 tok=21.997B dt=817 wait=234 +17:51:27 step 167160 loss=3.1149 lr=2.40e-05 tok=21.999B dt=882 wait=235 +17:52:31 step 167180 loss=3.2136 lr=2.40e-05 tok=22.002B dt=945 wait=235 +17:53:35 step 167200 loss=3.1034 lr=2.40e-05 tok=22.004B dt=1010 wait=235 +17:54:40 step 167220 loss=3.2093 lr=2.40e-05 tok=22.007B dt=1074 wait=235 +17:55:45 step 167240 loss=3.0444 lr=2.40e-05 tok=22.010B dt=1139 wait=235 +17:56:30 pushed ckpt_w153.pt,log_w153.txt +17:59:16 waited 166s for global r=676 +17:59:23 global r=676 load 4.293 -> 4.285 +18:00:04 step 167260 loss=3.1957 lr=2.40e-05 tok=22.012B dt=1399 wait=235 +18:01:21 step 167280 loss=3.1124 lr=2.40e-05 tok=22.015B dt=1476 wait=236 +18:02:25 step 167300 loss=3.0801 lr=2.40e-05 tok=22.017B dt=1539 wait=236 +18:03:28 step 167320 loss=3.1997 lr=2.40e-05 tok=22.020B dt=1603 wait=236 +18:04:32 step 167340 loss=3.0492 lr=2.40e-05 tok=22.023B dt=1667 wait=236 +18:05:36 step 167360 loss=3.2287 lr=2.39e-05 tok=22.025B dt=1730 wait=236 +18:06:39 step 167380 loss=3.1396 lr=2.39e-05 tok=22.028B dt=1793 wait=236 +18:07:43 step 167400 loss=3.0587 lr=2.39e-05 tok=22.031B dt=1857 wait=237 +18:07:57 pushed ckpt_w153.pt,log_w153.txt +18:09:13 waited 76s for global r=677 +18:09:19 global r=677 load 4.277 -> 4.280 +18:10:24 step 167420 loss=2.5837 lr=2.39e-05 tok=22.033B dt=2019 wait=237 +18:11:34 step 167440 loss=3.0536 lr=2.39e-05 tok=22.036B dt=2088 wait=237 +18:12:52 step 167460 loss=3.0810 lr=2.39e-05 tok=22.038B dt=2166 wait=237 +18:13:55 step 167480 loss=3.0039 lr=2.39e-05 tok=22.041B dt=2230 wait=237 +18:14:58 step 167500 loss=2.9775 lr=2.39e-05 tok=22.044B dt=2293 wait=237 +18:16:02 step 167520 loss=2.7714 lr=2.39e-05 tok=22.046B dt=2357 wait=238 +18:17:06 step 167540 loss=3.0054 lr=2.38e-05 tok=22.049B dt=2421 wait=238 +18:17:50 pushed ckpt_w153.pt,log_w153.txt +18:19:06 waited 76s for global r=678 +18:19:13 global r=678 load 4.283 -> 4.281 +18:19:46 step 167560 loss=3.2288 lr=2.38e-05 tok=22.052B dt=2580 wait=238 +18:20:50 step 167580 loss=3.0729 lr=2.38e-05 tok=22.054B dt=2644 wait=238 +18:22:10 step 167600 loss=3.1385 lr=2.38e-05 tok=22.057B dt=2724 wait=238 +18:23:19 step 167620 loss=3.1051 lr=2.38e-05 tok=22.059B dt=2794 wait=238 +18:24:23 step 167640 loss=3.0423 lr=2.38e-05 tok=22.062B dt=2857 wait=239 +18:25:26 step 167660 loss=3.0368 lr=2.38e-05 tok=22.065B dt=2921 wait=239 +18:26:30 step 167680 loss=3.0118 lr=2.38e-05 tok=22.067B dt=2984 wait=239 +18:27:34 step 167700 loss=2.9812 lr=2.38e-05 tok=22.070B dt=3049 wait=239 +18:27:47 pushed ckpt_w153.pt,log_w153.txt +18:28:47 waited 60s for global r=679 +18:28:55 global r=679 load 4.279 -> 4.283 +18:30:00 step 167720 loss=2.9691 lr=2.38e-05 tok=22.073B dt=3194 wait=239 +18:31:10 step 167740 loss=3.0396 lr=2.37e-05 tok=22.075B dt=3265 wait=239 +18:32:26 step 167760 loss=2.9872 lr=2.37e-05 tok=22.078B dt=3341 wait=239 +18:33:29 step 167780 loss=2.8850 lr=2.37e-05 tok=22.080B dt=3404 wait=240 +18:34:32 step 167800 loss=2.9566 lr=2.37e-05 tok=22.083B dt=3467 wait=240 +18:35:35 step 167820 loss=2.9764 lr=2.37e-05 tok=22.086B dt=3530 wait=240 +18:36:38 step 167840 loss=2.7616 lr=2.37e-05 tok=22.088B dt=3593 wait=240 +18:37:24 pushed ckpt_w153.pt,log_w153.txt +18:38:40 waited 76s for global r=680 +18:38:49 global r=680 load 4.279 -> 4.282 +18:39:22 step 167860 loss=2.9680 lr=2.37e-05 tok=22.091B dt=3756 wait=240 +18:40:25 step 167880 loss=2.8936 lr=2.37e-05 tok=22.093B dt=3819 wait=240 +18:41:43 step 167900 loss=2.8976 lr=2.37e-05 tok=22.096B dt=3897 wait=241 +18:42:50 step 167920 loss=3.0077 lr=2.36e-05 tok=22.099B dt=3964 wait=241 +18:43:53 step 167940 loss=2.8944 lr=2.36e-05 tok=22.101B dt=4027 wait=241 +18:44:55 step 167960 loss=2.8979 lr=2.36e-05 tok=22.104B dt=4090 wait=241 +18:45:59 step 167980 loss=2.6104 lr=2.36e-05 tok=22.107B dt=4153 wait=241 +18:47:01 step 168000 loss=2.8899 lr=2.36e-05 tok=22.109B dt=4216 wait=241 +18:47:15 pushed ckpt_w153.pt,log_w153.txt +18:48:46 waited 91s for global r=681 +18:48:53 global r=681 load 4.284 -> 4.285 +18:49:57 step 168020 loss=2.5912 lr=2.36e-05 tok=22.112B dt=4392 wait=241 +18:51:09 step 168040 loss=2.9365 lr=2.36e-05 tok=22.114B dt=4463 wait=242 +18:52:23 step 168060 loss=2.9920 lr=2.36e-05 tok=22.117B dt=4538 wait=242 +18:53:26 step 168080 loss=2.6359 lr=2.36e-05 tok=22.120B dt=4600 wait=242 +18:54:29 step 168100 loss=2.7327 lr=2.36e-05 tok=22.122B dt=4663 wait=242 +18:55:31 step 168120 loss=2.9322 lr=2.35e-05 tok=22.125B dt=4726 wait=242 +18:56:35 step 168140 loss=2.6946 lr=2.35e-05 tok=22.128B dt=4789 wait=242 +18:57:18 pushed ckpt_w153.pt,log_w153.txt +18:58:34 waited 76s for global r=682 +18:58:44 global r=682 load 4.295 -> 4.283 +18:59:17 step 168160 loss=2.9746 lr=2.35e-05 tok=22.130B dt=4952 wait=242 +19:00:32 step 168180 loss=2.8851 lr=2.35e-05 tok=22.133B dt=5026 wait=243 +19:01:48 step 168200 loss=2.8287 lr=2.35e-05 tok=22.135B dt=5102 wait=243 +19:02:50 step 168220 loss=2.9840 lr=2.35e-05 tok=22.138B dt=5164 wait=243 +19:03:53 step 168240 loss=2.9195 lr=2.35e-05 tok=22.141B dt=5227 wait=243 +19:04:56 step 168260 loss=3.0412 lr=2.35e-05 tok=22.143B dt=5291 wait=243 +19:06:00 step 168280 loss=3.0194 lr=2.35e-05 tok=22.146B dt=5355 wait=243 +19:07:04 step 168300 loss=3.1882 lr=2.35e-05 tok=22.149B dt=5419 wait=244 +19:07:16 pushed ckpt_w153.pt,log_w153.txt +19:08:17 waited 61s for global r=683 +19:08:23 global r=683 load 4.282 -> 4.281 +19:09:28 step 168320 loss=3.1025 lr=2.34e-05 tok=22.151B dt=5562 wait=244 +19:10:43 step 168340 loss=3.1157 lr=2.34e-05 tok=22.154B dt=5638 wait=244 +19:11:58 step 168360 loss=3.1133 lr=2.34e-05 tok=22.156B dt=5712 wait=244 +19:13:02 step 168380 loss=3.0275 lr=2.34e-05 tok=22.159B dt=5777 wait=244 +19:14:07 step 168400 loss=3.0978 lr=2.34e-05 tok=22.162B dt=5841 wait=244 +19:15:11 step 168420 loss=3.1453 lr=2.34e-05 tok=22.164B dt=5905 wait=244 +19:16:14 step 168440 loss=3.0437 lr=2.34e-05 tok=22.167B dt=5969 wait=245 +19:17:01 pushed ckpt_w153.pt,log_w153.txt +19:18:16 waited 76s for global r=684 +19:18:24 global r=684 load 4.278 -> 4.282 +19:18:57 step 168460 loss=3.1039 lr=2.34e-05 tok=22.170B dt=6132 wait=245 +19:20:09 step 168480 loss=2.9765 lr=2.34e-05 tok=22.172B dt=6204 wait=245 +19:21:27 step 168500 loss=3.0045 lr=2.33e-05 tok=22.175B dt=6281 wait=245 +19:22:31 step 168520 loss=3.0152 lr=2.33e-05 tok=22.177B dt=6345 wait=245 +19:23:35 step 168540 loss=3.0551 lr=2.33e-05 tok=22.180B dt=6409 wait=245 +19:24:39 step 168560 loss=3.0370 lr=2.33e-05 tok=22.183B dt=6473 wait=246 +19:25:44 step 168580 loss=2.9468 lr=2.33e-05 tok=22.185B dt=6538 wait=246 +19:26:47 step 168600 loss=3.0133 lr=2.33e-05 tok=22.188B dt=6602 wait=246 +19:27:01 pushed ckpt_w153.pt,log_w153.txt +19:28:01 waited 60s for global r=685 +19:28:08 global r=685 load 4.280 -> 4.283 +19:29:12 step 168620 loss=2.8979 lr=2.33e-05 tok=22.190B dt=6747 wait=246 +19:30:37 step 168640 loss=2.7324 lr=2.33e-05 tok=22.193B dt=6831 wait=246 +19:31:40 step 168660 loss=2.8955 lr=2.33e-05 tok=22.196B dt=6894 wait=246 +19:32:43 step 168680 loss=2.9056 lr=2.33e-05 tok=22.198B dt=6957 wait=247 +19:33:47 step 168700 loss=2.5101 lr=2.32e-05 tok=22.201B dt=7021 wait=247 +19:34:50 step 168720 loss=3.0234 lr=2.32e-05 tok=22.204B dt=7084 wait=247 +19:35:53 step 168740 loss=2.8338 lr=2.32e-05 tok=22.206B dt=7147 wait=247 +19:36:40 pushed ckpt_w153.pt,log_w153.txt +19:37:41 waited 60s for global r=686 +19:37:47 global r=686 load 4.285 -> 4.286 +19:38:20 step 168760 loss=2.9699 lr=2.32e-05 tok=22.209B dt=7294 wait=247 +19:39:31 step 168780 loss=2.8896 lr=2.32e-05 tok=22.211B dt=7366 wait=247 +19:40:47 step 168800 loss=2.6219 lr=2.32e-05 tok=22.214B dt=7442 wait=248 +19:41:51 step 168820 loss=2.8645 lr=2.32e-05 tok=22.217B dt=7505 wait=248 +19:42:54 step 168840 loss=2.9132 lr=2.32e-05 tok=22.219B dt=7568 wait=248 +19:43:57 step 168860 loss=3.0271 lr=2.32e-05 tok=22.222B dt=7631 wait=248 +19:45:00 step 168880 loss=2.9923 lr=2.32e-05 tok=22.225B dt=7695 wait=248 +19:46:04 step 168900 loss=2.9916 lr=2.31e-05 tok=22.227B dt=7758 wait=248 +19:46:16 pushed ckpt_w153.pt,log_w153.txt +19:47:31 waited 76s for global r=687 +19:47:38 global r=687 load 4.281 -> 4.284 +19:48:50 step 168920 loss=2.9312 lr=2.31e-05 tok=22.230B dt=7924 wait=249 +19:50:08 step 168940 loss=2.9140 lr=2.31e-05 tok=22.232B dt=8002 wait=249 +19:51:10 step 168960 loss=3.0523 lr=2.31e-05 tok=22.235B dt=8065 wait=249 +19:52:14 step 168980 loss=2.9612 lr=2.31e-05 tok=22.238B dt=8128 wait=249 +19:53:17 step 169000 loss=2.9768 lr=2.31e-05 tok=22.240B dt=8191 wait=249 +19:54:21 step 169020 loss=3.0388 lr=2.31e-05 tok=22.243B dt=8256 wait=249 +19:55:25 step 169040 loss=2.9252 lr=2.31e-05 tok=22.246B dt=8319 wait=249 +19:56:37 pushed ckpt_w153.pt,log_w153.txt +19:57:22 waited 45s for global r=688 +19:57:29 global r=688 load 4.286 -> 4.284 +19:58:02 step 169060 loss=2.9977 lr=2.31e-05 tok=22.248B dt=8476 wait=250 +19:59:25 step 169080 loss=3.0218 lr=2.31e-05 tok=22.251B dt=8559 wait=250 +20:00:30 step 169100 loss=3.0134 lr=2.30e-05 tok=22.253B dt=8624 wait=250 +20:01:34 step 169120 loss=3.0145 lr=2.30e-05 tok=22.256B dt=8689 wait=250 +20:02:39 step 169140 loss=2.9574 lr=2.30e-05 tok=22.259B dt=8753 wait=250 +20:03:42 step 169160 loss=3.1261 lr=2.30e-05 tok=22.261B dt=8817 wait=250 +20:04:46 step 169180 loss=2.8594 lr=2.30e-05 tok=22.264B dt=8881 wait=251 +20:05:50 step 169200 loss=3.1221 lr=2.30e-05 tok=22.267B dt=8944 wait=251 +20:06:06 pushed ckpt_w153.pt,log_w153.txt +20:09:08 waited 181s for global r=689 +20:09:16 global r=689 load 4.279 -> 4.280 +20:10:29 step 169220 loss=3.0274 lr=2.30e-05 tok=22.269B dt=9224 wait=251 +20:11:45 step 169240 loss=3.1024 lr=2.30e-05 tok=22.272B dt=9300 wait=251 +20:12:48 step 169260 loss=3.1591 lr=2.30e-05 tok=22.274B dt=9363 wait=251 +20:13:52 step 169280 loss=3.0612 lr=2.30e-05 tok=22.277B dt=9427 wait=251 +20:14:57 step 169300 loss=2.9855 lr=2.29e-05 tok=22.280B dt=9491 wait=252 +20:16:01 step 169320 loss=3.1349 lr=2.29e-05 tok=22.282B dt=9555 wait=252 +20:17:05 step 169340 loss=2.9969 lr=2.29e-05 tok=22.285B dt=9620 wait=252 +20:17:55 pushed ckpt_w153.pt,log_w153.txt +20:18:55 waited 60s for global r=690 +20:19:04 global r=690 load 4.280 -> 4.284 +20:19:43 step 169360 loss=3.1485 lr=2.29e-05 tok=22.287B dt=9777 wait=252 +20:21:01 step 169380 loss=2.9217 lr=2.29e-05 tok=22.290B dt=9855 wait=252 +20:22:04 step 169400 loss=2.9202 lr=2.29e-05 tok=22.293B dt=9919 wait=252 +20:23:07 step 169420 loss=2.9745 lr=2.29e-05 tok=22.295B dt=9981 wait=252 +20:24:11 step 169440 loss=2.8407 lr=2.29e-05 tok=22.298B dt=10045 wait=253 +20:25:15 step 169460 loss=3.0080 lr=2.29e-05 tok=22.301B dt=10109 wait=253 +20:26:19 step 169480 loss=2.9489 lr=2.29e-05 tok=22.303B dt=10173 wait=253 +20:27:23 step 169500 loss=2.9789 lr=2.28e-05 tok=22.306B dt=10237 wait=253 +20:27:38 pushed ckpt_w153.pt,log_w153.txt +20:28:23 waited 45s for global r=691 +20:28:32 global r=691 load 4.287 -> 4.287 +20:29:45 step 169520 loss=2.9284 lr=2.28e-05 tok=22.308B dt=10380 wait=253 +20:31:01 step 169540 loss=3.0736 lr=2.28e-05 tok=22.311B dt=10455 wait=253 +20:32:04 step 169560 loss=3.0002 lr=2.28e-05 tok=22.314B dt=10518 wait=253 +20:33:08 step 169580 loss=3.0116 lr=2.28e-05 tok=22.316B dt=10582 wait=254 +20:34:11 step 169600 loss=3.0559 lr=2.28e-05 tok=22.319B dt=10646 wait=254 +20:35:15 step 169620 loss=2.9927 lr=2.28e-05 tok=22.322B dt=10710 wait=254 +20:36:18 step 169640 loss=2.9355 lr=2.28e-05 tok=22.324B dt=10772 wait=254 +20:37:03 pushed ckpt_w153.pt,log_w153.txt +20:38:03 waited 60s for global r=692 +20:38:09 global r=692 load 4.281 -> 4.284 +20:38:42 step 169660 loss=3.0709 lr=2.28e-05 tok=22.327B dt=10917 wait=254 +20:40:02 step 169680 loss=2.9816 lr=2.28e-05 tok=22.329B dt=10997 wait=254 +20:41:10 step 169700 loss=2.9334 lr=2.27e-05 tok=22.332B dt=11065 wait=255 +20:42:14 step 169720 loss=2.9775 lr=2.27e-05 tok=22.335B dt=11128 wait=255 +20:43:18 step 169740 loss=2.8960 lr=2.27e-05 tok=22.337B dt=11192 wait=255 +20:44:22 step 169760 loss=2.9370 lr=2.27e-05 tok=22.340B dt=11257 wait=255 +20:45:26 step 169780 loss=2.9243 lr=2.27e-05 tok=22.343B dt=11321 wait=255 +20:46:31 step 169800 loss=2.8316 lr=2.27e-05 tok=22.345B dt=11385 wait=255 +20:46:44 pushed ckpt_w153.pt,log_w153.txt +20:47:44 waited 60s for global r=693 +20:47:50 global r=693 load 4.283 -> 4.282 +20:49:04 step 169820 loss=2.7768 lr=2.27e-05 tok=22.348B dt=11539 wait=256 +20:50:22 step 169840 loss=2.6418 lr=2.27e-05 tok=22.350B dt=11617 wait=256 +20:51:27 step 169860 loss=2.8337 lr=2.27e-05 tok=22.353B dt=11681 wait=256 +20:52:31 step 169880 loss=2.9406 lr=2.27e-05 tok=22.356B dt=11745 wait=256 +20:53:36 step 169900 loss=2.5509 lr=2.26e-05 tok=22.358B dt=11810 wait=256 +20:54:42 step 169920 loss=2.4155 lr=2.26e-05 tok=22.361B dt=11876 wait=256 +20:55:47 step 169940 loss=2.7931 lr=2.26e-05 tok=22.364B dt=11941 wait=256 +20:56:31 pushed ckpt_w153.pt,log_w153.txt +20:57:31 waited 60s for global r=694 +20:57:38 global r=694 load 4.287 -> 4.282 +20:58:11 step 169960 loss=2.8556 lr=2.26e-05 tok=22.366B dt=12085 wait=257 +20:59:36 step 169980 loss=2.9000 lr=2.26e-05 tok=22.369B dt=12171 wait=257 +21:00:41 step 170000 loss=2.9781 lr=2.26e-05 tok=22.371B dt=12235 wait=257 +21:01:44 step 170020 loss=3.1605 lr=2.26e-05 tok=22.374B dt=12299 wait=257 +21:02:48 step 170040 loss=2.9811 lr=2.26e-05 tok=22.377B dt=12363 wait=257 +21:03:53 step 170060 loss=2.8819 lr=2.26e-05 tok=22.379B dt=12427 wait=258 +21:04:56 step 170080 loss=2.9844 lr=2.26e-05 tok=22.382B dt=12490 wait=258 +21:06:00 step 170100 loss=2.9735 lr=2.26e-05 tok=22.384B dt=12554 wait=258 diff --git a/log_w154.txt b/log_w154.txt new file mode 100644 index 0000000000000000000000000000000000000000..a110f597aff004600f9994883201e7bd4205b1d8 --- /dev/null +++ b/log_w154.txt @@ -0,0 +1,157 @@ +launch +13:40:41 name=w154 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 101/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:40:48 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:40:48 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:40:57 params 1.08B +13:40:59 resumed step=163650 seen=21.59B round=652 +13:40:59 stream salt=163650 +13:42:52 blocks compiled +13:42:52 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:47:19 step 163651 loss=3.0189 lr=2.60e-05 tok=21.586B dt=267 wait=231 +13:47:35 step 163652 loss=3.0864 lr=2.60e-05 tok=21.587B dt=283 wait=231 +13:48:00 step 163660 loss=3.5047 lr=2.60e-05 tok=21.588B dt=308 wait=231 +13:48:18 global r=653 load 4.278 -> 4.276 +13:49:22 step 163820 loss=3.2925 lr=2.59e-05 tok=21.590B dt=391 wait=232 +13:50:26 step 163840 loss=3.3989 lr=2.59e-05 tok=21.593B dt=454 wait=232 +13:51:29 step 163860 loss=3.3969 lr=2.59e-05 tok=21.595B dt=517 wait=232 +13:52:32 step 163880 loss=3.3354 lr=2.59e-05 tok=21.598B dt=580 wait=232 +13:53:35 step 163900 loss=3.3068 lr=2.59e-05 tok=21.601B dt=643 wait=232 +13:54:38 step 163920 loss=3.4182 lr=2.58e-05 tok=21.603B dt=707 wait=232 +13:55:41 step 163940 loss=3.2178 lr=2.58e-05 tok=21.606B dt=770 wait=233 +13:56:30 pushed ckpt_w154.pt,log_w154.txt +13:56:30 waited 0s for global r=654 +13:56:39 global r=654 load 4.266 -> 4.277 +13:57:13 step 163960 loss=3.4013 lr=2.58e-05 tok=21.609B dt=861 wait=233 +13:58:16 step 163980 loss=3.2260 lr=2.58e-05 tok=21.611B dt=924 wait=233 +13:59:18 step 164000 loss=3.2807 lr=2.58e-05 tok=21.614B dt=987 wait=233 +14:00:21 step 164020 loss=3.3163 lr=2.58e-05 tok=21.616B dt=1050 wait=233 +14:01:24 step 164040 loss=3.2429 lr=2.58e-05 tok=21.619B dt=1113 wait=234 +14:02:28 step 164060 loss=3.3452 lr=2.58e-05 tok=21.622B dt=1176 wait=234 +14:03:31 step 164080 loss=3.3135 lr=2.57e-05 tok=21.624B dt=1239 wait=234 +14:04:33 step 164100 loss=3.3111 lr=2.57e-05 tok=21.627B dt=1302 wait=234 +14:04:47 pushed ckpt_w154.pt,log_w154.txt +14:06:02 waited 76s for global r=655 +14:06:09 global r=655 load 4.277 -> 4.278 +14:07:14 step 164120 loss=3.3021 lr=2.57e-05 tok=21.630B dt=1462 wait=234 +14:08:17 step 164140 loss=3.2671 lr=2.57e-05 tok=21.632B dt=1525 wait=235 +14:09:20 step 164160 loss=3.1637 lr=2.57e-05 tok=21.635B dt=1588 wait=235 +14:10:22 step 164180 loss=3.2525 lr=2.57e-05 tok=21.637B dt=1651 wait=235 +14:11:24 step 164200 loss=3.2345 lr=2.57e-05 tok=21.640B dt=1713 wait=235 +14:12:27 step 164220 loss=3.3416 lr=2.57e-05 tok=21.643B dt=1775 wait=235 +14:13:29 step 164240 loss=3.2164 lr=2.57e-05 tok=21.645B dt=1838 wait=236 +14:14:11 pushed ckpt_w154.pt,log_w154.txt +14:15:57 waited 106s for global r=656 +14:16:04 global r=656 load 4.280 -> 4.279 +14:16:37 step 164260 loss=3.3199 lr=2.56e-05 tok=21.648B dt=2025 wait=236 +14:17:40 step 164280 loss=3.1930 lr=2.56e-05 tok=21.650B dt=2088 wait=236 +14:18:43 step 164300 loss=3.1590 lr=2.56e-05 tok=21.653B dt=2151 wait=236 +14:19:46 step 164320 loss=3.3299 lr=2.56e-05 tok=21.656B dt=2214 wait=236 +14:20:49 step 164340 loss=3.2111 lr=2.56e-05 tok=21.658B dt=2277 wait=237 +14:21:52 step 164360 loss=3.2482 lr=2.56e-05 tok=21.661B dt=2341 wait=237 +14:22:55 step 164380 loss=3.2328 lr=2.56e-05 tok=21.664B dt=2404 wait=237 +14:23:58 step 164400 loss=3.3427 lr=2.56e-05 tok=21.666B dt=2467 wait=237 +14:24:09 pushed ckpt_w154.pt,log_w154.txt +14:25:40 waited 91s for global r=657 +14:25:49 global r=657 load 4.276 -> 4.276 +14:26:55 step 164420 loss=3.2093 lr=2.56e-05 tok=21.669B dt=2643 wait=237 +14:27:58 step 164440 loss=3.2307 lr=2.55e-05 tok=21.671B dt=2707 wait=238 +14:29:01 step 164460 loss=3.2879 lr=2.55e-05 tok=21.674B dt=2770 wait=238 +14:30:05 step 164480 loss=3.2305 lr=2.55e-05 tok=21.677B dt=2833 wait=238 +14:31:08 step 164500 loss=3.2863 lr=2.55e-05 tok=21.679B dt=2896 wait=238 +14:32:11 step 164520 loss=3.3872 lr=2.55e-05 tok=21.682B dt=2960 wait=238 +14:33:15 step 164540 loss=3.2720 lr=2.55e-05 tok=21.685B dt=3024 wait=239 +14:33:58 pushed ckpt_w154.pt,log_w154.txt +14:35:13 waited 76s for global r=658 +14:35:19 global r=658 load 4.276 -> 4.278 +14:35:52 step 164560 loss=3.3627 lr=2.55e-05 tok=21.687B dt=3181 wait=239 +14:36:56 step 164580 loss=3.2297 lr=2.55e-05 tok=21.690B dt=3245 wait=239 +14:38:00 step 164600 loss=3.2053 lr=2.55e-05 tok=21.692B dt=3309 wait=239 +14:39:05 step 164620 loss=3.1861 lr=2.54e-05 tok=21.695B dt=3374 wait=239 +14:40:09 step 164640 loss=3.1879 lr=2.54e-05 tok=21.698B dt=3438 wait=240 +14:41:13 step 164660 loss=3.2569 lr=2.54e-05 tok=21.700B dt=3501 wait=240 +14:42:17 step 164680 loss=3.1547 lr=2.54e-05 tok=21.703B dt=3565 wait=240 +14:43:21 step 164700 loss=3.2075 lr=2.54e-05 tok=21.706B dt=3629 wait=240 +14:43:31 pushed ckpt_w154.pt,log_w154.txt +14:44:47 waited 76s for global r=659 +14:44:56 global r=659 load 4.277 -> 4.277 +14:46:00 step 164720 loss=3.1378 lr=2.54e-05 tok=21.708B dt=3789 wait=240 +14:47:05 step 164740 loss=3.1439 lr=2.54e-05 tok=21.711B dt=3853 wait=241 +14:48:08 step 164760 loss=3.2155 lr=2.54e-05 tok=21.713B dt=3917 wait=241 +14:49:12 step 164780 loss=3.0485 lr=2.53e-05 tok=21.716B dt=3981 wait=241 +14:50:16 step 164800 loss=3.2464 lr=2.53e-05 tok=21.719B dt=4044 wait=241 +14:51:20 step 164820 loss=3.2481 lr=2.53e-05 tok=21.721B dt=4108 wait=241 +14:52:23 step 164840 loss=3.1649 lr=2.53e-05 tok=21.724B dt=4171 wait=242 +14:53:08 pushed ckpt_w154.pt,log_w154.txt +14:54:38 waited 91s for global r=660 +14:54:46 global r=660 load 4.278 -> 4.275 +14:55:19 step 164860 loss=3.3102 lr=2.53e-05 tok=21.726B dt=4347 wait=242 +14:56:22 step 164880 loss=3.2458 lr=2.53e-05 tok=21.729B dt=4410 wait=242 +14:57:24 step 164900 loss=3.1974 lr=2.53e-05 tok=21.732B dt=4473 wait=242 +14:58:28 step 164920 loss=3.3308 lr=2.53e-05 tok=21.734B dt=4536 wait=242 +14:59:31 step 164940 loss=3.1163 lr=2.53e-05 tok=21.737B dt=4600 wait=243 +15:00:34 step 164960 loss=3.2534 lr=2.52e-05 tok=21.740B dt=4662 wait=243 +15:01:37 step 164980 loss=3.2057 lr=2.52e-05 tok=21.742B dt=4725 wait=243 +15:02:39 step 165000 loss=3.1761 lr=2.52e-05 tok=21.745B dt=4788 wait=243 +15:02:51 pushed ckpt_w154.pt,log_w154.txt +15:04:36 waited 106s for global r=661 +15:04:45 global r=661 load 4.271 -> 4.275 +15:05:48 step 165020 loss=3.2224 lr=2.52e-05 tok=21.747B dt=4977 wait=243 +15:06:51 step 165040 loss=3.2073 lr=2.52e-05 tok=21.750B dt=5039 wait=244 +15:07:54 step 165060 loss=3.1479 lr=2.52e-05 tok=21.753B dt=5102 wait=244 +15:08:58 step 165080 loss=3.1360 lr=2.52e-05 tok=21.755B dt=5166 wait=244 +15:10:00 step 165100 loss=3.2933 lr=2.52e-05 tok=21.758B dt=5229 wait=244 +15:11:04 step 165120 loss=3.2721 lr=2.52e-05 tok=21.761B dt=5292 wait=244 +15:12:06 step 165140 loss=3.1869 lr=2.51e-05 tok=21.763B dt=5355 wait=245 +15:12:49 pushed ckpt_w154.pt,log_w154.txt +15:16:24 step 165160 loss=3.3256 lr=2.51e-05 tok=21.766B dt=5613 wait=245 +15:16:31 global r=662 load 4.279 -> 4.277 +15:17:03 step 165160 loss=3.3795 lr=2.51e-05 tok=21.767B dt=5651 wait=245 +15:18:06 step 165180 loss=3.2790 lr=2.51e-05 tok=21.770B dt=5715 wait=245 +15:19:10 step 165200 loss=3.2294 lr=2.51e-05 tok=21.772B dt=5778 wait=245 +15:20:14 step 165220 loss=3.3138 lr=2.51e-05 tok=21.775B dt=5842 wait=245 +15:21:17 step 165240 loss=3.0986 lr=2.51e-05 tok=21.778B dt=5906 wait=246 +15:22:21 step 165260 loss=3.2723 lr=2.51e-05 tok=21.780B dt=5969 wait=246 +15:23:24 step 165280 loss=3.1846 lr=2.51e-05 tok=21.783B dt=6032 wait=246 +15:24:27 step 165300 loss=3.1807 lr=2.51e-05 tok=21.785B dt=6095 wait=246 +15:24:38 pushed ckpt_w154.pt,log_w154.txt +15:25:39 waited 61s for global r=663 +15:25:46 global r=663 load 4.279 -> 4.276 +15:26:50 step 165320 loss=3.2086 lr=2.50e-05 tok=21.788B dt=6238 wait=246 +15:27:52 step 165340 loss=3.2733 lr=2.50e-05 tok=21.791B dt=6301 wait=247 +15:28:55 step 165360 loss=3.1761 lr=2.50e-05 tok=21.793B dt=6364 wait=247 +15:29:59 step 165380 loss=3.1026 lr=2.50e-05 tok=21.796B dt=6428 wait=247 +15:31:02 step 165400 loss=3.1405 lr=2.50e-05 tok=21.799B dt=6490 wait=247 +15:32:05 step 165420 loss=3.2265 lr=2.50e-05 tok=21.801B dt=6554 wait=247 +15:33:09 step 165440 loss=3.0721 lr=2.50e-05 tok=21.804B dt=6617 wait=247 +15:33:51 pushed ckpt_w154.pt,log_w154.txt +15:35:06 waited 76s for global r=664 +15:35:14 global r=664 load 4.273 -> 4.273 +15:35:47 step 165460 loss=3.2421 lr=2.50e-05 tok=21.806B dt=6775 wait=248 +15:36:50 step 165480 loss=3.1240 lr=2.50e-05 tok=21.809B dt=6838 wait=248 +15:37:53 step 165500 loss=3.1759 lr=2.49e-05 tok=21.812B dt=6901 wait=248 +15:38:57 step 165520 loss=3.1271 lr=2.49e-05 tok=21.814B dt=6965 wait=248 +15:40:00 step 165540 loss=3.1377 lr=2.49e-05 tok=21.817B dt=7028 wait=248 +15:41:03 step 165560 loss=3.1898 lr=2.49e-05 tok=21.820B dt=7091 wait=249 +15:42:06 step 165580 loss=3.0716 lr=2.49e-05 tok=21.822B dt=7154 wait=249 +15:43:09 step 165600 loss=3.1848 lr=2.49e-05 tok=21.825B dt=7217 wait=249 +15:43:20 pushed ckpt_w154.pt,log_w154.txt +15:44:35 waited 76s for global r=665 +15:44:42 global r=665 load 4.272 -> 4.275 +15:45:46 step 165620 loss=3.2320 lr=2.49e-05 tok=21.827B dt=7374 wait=249 +15:46:49 step 165640 loss=3.2625 lr=2.49e-05 tok=21.830B dt=7437 wait=249 +15:47:52 step 165660 loss=3.1485 lr=2.49e-05 tok=21.833B dt=7501 wait=250 +15:48:56 step 165680 loss=3.1415 lr=2.48e-05 tok=21.835B dt=7564 wait=250 +15:49:59 step 165700 loss=3.2497 lr=2.48e-05 tok=21.838B dt=7628 wait=250 +15:51:03 step 165720 loss=3.2937 lr=2.48e-05 tok=21.841B dt=7691 wait=250 +15:52:06 step 165740 loss=3.1956 lr=2.48e-05 tok=21.843B dt=7755 wait=250 +15:52:52 pushed ckpt_w154.pt,log_w154.txt +15:54:08 waited 76s for global r=666 +15:54:14 global r=666 load 4.280 -> 4.276 +15:54:47 step 165760 loss=3.1169 lr=2.48e-05 tok=21.846B dt=7916 wait=251 +15:55:50 step 165780 loss=3.1846 lr=2.48e-05 tok=21.848B dt=7979 wait=251 +15:56:54 step 165800 loss=3.1890 lr=2.48e-05 tok=21.851B dt=8042 wait=251 +15:57:58 step 165820 loss=3.2131 lr=2.48e-05 tok=21.854B dt=8106 wait=251 +15:59:01 step 165840 loss=3.1610 lr=2.48e-05 tok=21.856B dt=8169 wait=251 +16:00:09 step 165860 loss=3.3157 lr=2.47e-05 tok=21.859B dt=8237 wait=252 +16:01:12 step 165880 loss=3.2176 lr=2.47e-05 tok=21.861B dt=8301 wait=252 +16:02:16 step 165900 loss=3.3031 lr=2.47e-05 tok=21.864B dt=8365 wait=252 diff --git a/log_w155.txt b/log_w155.txt new file mode 100644 index 0000000000000000000000000000000000000000..b412cbe3e518e53d4a2109139f1c00b92d81c0a8 --- /dev/null +++ b/log_w155.txt @@ -0,0 +1,695 @@ +launch +13:35:08 name=w155 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 102/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:35:14 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:35:15 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:35:24 params 1.08B +13:35:26 resumed step=163500 seen=21.57B round=651 +13:35:26 stream salt=163500 +13:37:18 blocks compiled +13:37:18 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:41:51 step 163501 loss=2.8958 lr=2.61e-05 tok=21.567B dt=272 wait=246 +13:42:06 step 163502 loss=2.7221 lr=2.61e-05 tok=21.567B dt=288 wait=246 +13:42:47 global r=652 load 4.283 -> 4.278 +13:43:19 step 163660 loss=3.1914 lr=2.60e-05 tok=21.569B dt=360 wait=247 +13:44:23 step 163680 loss=3.0013 lr=2.60e-05 tok=21.572B dt=425 wait=247 +13:45:03 global r=653 load 4.277 -> 4.276 +13:46:06 step 163820 loss=3.1660 lr=2.59e-05 tok=21.576B dt=528 wait=247 +13:47:10 step 163840 loss=3.1778 lr=2.59e-05 tok=21.578B dt=592 wait=247 +13:48:15 step 163860 loss=3.1864 lr=2.59e-05 tok=21.581B dt=656 wait=248 +13:49:19 step 163880 loss=3.0230 lr=2.59e-05 tok=21.584B dt=721 wait=248 +13:50:24 step 163900 loss=3.0889 lr=2.59e-05 tok=21.586B dt=785 wait=248 +13:51:28 step 163920 loss=3.1969 lr=2.58e-05 tok=21.589B dt=850 wait=248 +13:52:32 step 163940 loss=3.0273 lr=2.58e-05 tok=21.591B dt=914 wait=248 +13:53:20 pushed ckpt_w155.pt,log_w155.txt +13:56:06 waited 166s for global r=654 +13:56:17 global r=654 load 4.277 -> 4.277 +13:56:53 step 163960 loss=3.1782 lr=2.58e-05 tok=21.594B dt=1174 wait=248 +13:57:58 step 163980 loss=3.1027 lr=2.58e-05 tok=21.597B dt=1239 wait=249 +13:59:03 step 164000 loss=3.0685 lr=2.58e-05 tok=21.599B dt=1304 wait=249 +14:00:08 step 164020 loss=3.1916 lr=2.58e-05 tok=21.602B dt=1369 wait=249 +14:01:12 step 164040 loss=3.1543 lr=2.58e-05 tok=21.605B dt=1434 wait=249 +14:02:17 step 164060 loss=3.2562 lr=2.58e-05 tok=21.607B dt=1498 wait=249 +14:03:21 step 164080 loss=3.2041 lr=2.57e-05 tok=21.610B dt=1563 wait=250 +14:04:26 step 164100 loss=3.2292 lr=2.57e-05 tok=21.612B dt=1627 wait=250 +14:04:39 pushed ckpt_w155.pt,log_w155.txt +14:06:10 waited 91s for global r=655 +14:06:20 global r=655 load 4.284 -> 4.278 +14:07:26 step 164120 loss=3.2868 lr=2.57e-05 tok=21.615B dt=1807 wait=250 +14:08:30 step 164140 loss=3.2354 lr=2.57e-05 tok=21.618B dt=1872 wait=250 +14:09:33 step 164160 loss=3.2846 lr=2.57e-05 tok=21.620B dt=1935 wait=250 +14:10:37 step 164180 loss=3.1107 lr=2.57e-05 tok=21.623B dt=1999 wait=251 +14:11:41 step 164200 loss=3.2510 lr=2.57e-05 tok=21.626B dt=2063 wait=251 +14:12:45 step 164220 loss=3.3148 lr=2.57e-05 tok=21.628B dt=2127 wait=251 +14:13:49 step 164240 loss=3.1557 lr=2.57e-05 tok=21.631B dt=2191 wait=251 +14:14:36 pushed ckpt_w155.pt,log_w155.txt +14:15:51 waited 76s for global r=656 +14:16:06 global r=656 load 4.286 -> 4.279 +14:16:39 step 164260 loss=3.3063 lr=2.56e-05 tok=21.633B dt=2361 wait=251 +14:17:43 step 164280 loss=3.2332 lr=2.56e-05 tok=21.636B dt=2424 wait=252 +14:18:46 step 164300 loss=3.2349 lr=2.56e-05 tok=21.639B dt=2488 wait=252 +14:19:50 step 164320 loss=3.3118 lr=2.56e-05 tok=21.641B dt=2552 wait=252 +14:20:53 step 164340 loss=3.2280 lr=2.56e-05 tok=21.644B dt=2615 wait=252 +14:21:56 step 164360 loss=3.2733 lr=2.56e-05 tok=21.647B dt=2678 wait=252 +14:22:59 step 164380 loss=3.2527 lr=2.56e-05 tok=21.649B dt=2741 wait=252 +14:24:02 step 164400 loss=3.3533 lr=2.56e-05 tok=21.652B dt=2804 wait=253 +14:24:14 pushed ckpt_w155.pt,log_w155.txt +14:25:30 waited 76s for global r=657 +14:25:42 global r=657 load 4.282 -> 4.276 +14:26:46 step 164420 loss=3.2945 lr=2.56e-05 tok=21.654B dt=2967 wait=253 +14:27:48 step 164440 loss=3.2246 lr=2.55e-05 tok=21.657B dt=3030 wait=253 +14:28:51 step 164460 loss=3.2792 lr=2.55e-05 tok=21.660B dt=3093 wait=253 +14:29:54 step 164480 loss=3.1540 lr=2.55e-05 tok=21.662B dt=3156 wait=253 +14:30:58 step 164500 loss=3.1681 lr=2.55e-05 tok=21.665B dt=3220 wait=254 +14:32:01 step 164520 loss=3.4124 lr=2.55e-05 tok=21.668B dt=3283 wait=254 +14:33:03 step 164540 loss=3.1482 lr=2.55e-05 tok=21.670B dt=3345 wait=254 +14:33:51 pushed ckpt_w155.pt,log_w155.txt +14:35:07 waited 76s for global r=658 +14:35:15 global r=658 load 4.285 -> 4.278 +14:35:49 step 164560 loss=3.2431 lr=2.55e-05 tok=21.673B dt=3510 wait=254 +14:36:52 step 164580 loss=3.2352 lr=2.55e-05 tok=21.675B dt=3573 wait=254 +14:37:55 step 164600 loss=3.1504 lr=2.55e-05 tok=21.678B dt=3637 wait=254 +14:38:59 step 164620 loss=3.3323 lr=2.54e-05 tok=21.681B dt=3701 wait=255 +14:40:02 step 164640 loss=3.2325 lr=2.54e-05 tok=21.683B dt=3764 wait=255 +14:41:05 step 164660 loss=3.3728 lr=2.54e-05 tok=21.686B dt=3827 wait=255 +14:42:09 step 164680 loss=3.3044 lr=2.54e-05 tok=21.688B dt=3891 wait=255 +14:43:13 step 164700 loss=3.3645 lr=2.54e-05 tok=21.691B dt=3954 wait=255 +14:43:26 pushed ckpt_w155.pt,log_w155.txt +14:44:41 waited 76s for global r=659 +14:44:48 global r=659 load 4.276 -> 4.277 +14:45:54 step 164720 loss=3.3572 lr=2.54e-05 tok=21.694B dt=4116 wait=256 +14:46:57 step 164740 loss=3.4390 lr=2.54e-05 tok=21.696B dt=4179 wait=256 +14:48:01 step 164760 loss=3.4494 lr=2.54e-05 tok=21.699B dt=4243 wait=256 +14:49:05 step 164780 loss=3.4265 lr=2.53e-05 tok=21.702B dt=4307 wait=256 +14:50:09 step 164800 loss=3.3369 lr=2.53e-05 tok=21.704B dt=4371 wait=256 +14:51:13 step 164820 loss=3.5428 lr=2.53e-05 tok=21.707B dt=4435 wait=257 +14:52:16 step 164840 loss=3.3584 lr=2.53e-05 tok=21.709B dt=4498 wait=257 +14:53:00 pushed ckpt_w155.pt,log_w155.txt +14:54:45 waited 106s for global r=660 +14:54:53 global r=660 load 4.278 -> 4.275 +14:55:26 step 164860 loss=3.4089 lr=2.53e-05 tok=21.712B dt=4688 wait=257 +14:56:28 step 164880 loss=3.2455 lr=2.53e-05 tok=21.715B dt=4750 wait=257 +14:57:32 step 164900 loss=3.2698 lr=2.53e-05 tok=21.717B dt=4813 wait=258 +14:58:35 step 164920 loss=3.3101 lr=2.53e-05 tok=21.720B dt=4877 wait=258 +14:59:38 step 164940 loss=3.1941 lr=2.53e-05 tok=21.723B dt=4939 wait=258 +15:00:40 step 164960 loss=3.2774 lr=2.52e-05 tok=21.725B dt=5002 wait=258 +15:01:42 step 164980 loss=3.2799 lr=2.52e-05 tok=21.728B dt=5064 wait=258 +15:02:44 step 165000 loss=3.3688 lr=2.52e-05 tok=21.730B dt=5126 wait=259 +15:03:01 pushed ckpt_w155.pt,log_w155.txt +15:04:32 waited 91s for global r=661 +15:04:39 global r=661 load 4.281 -> 4.275 +15:05:44 step 165020 loss=3.2629 lr=2.52e-05 tok=21.733B dt=5305 wait=259 +15:06:46 step 165040 loss=3.3119 lr=2.52e-05 tok=21.736B dt=5368 wait=259 +15:07:49 step 165060 loss=3.3458 lr=2.52e-05 tok=21.738B dt=5430 wait=259 +15:08:51 step 165080 loss=3.2938 lr=2.52e-05 tok=21.741B dt=5493 wait=259 +15:09:54 step 165100 loss=3.2294 lr=2.52e-05 tok=21.744B dt=5555 wait=260 +15:10:58 step 165120 loss=3.2710 lr=2.52e-05 tok=21.746B dt=5619 wait=260 +15:12:01 step 165140 loss=3.1852 lr=2.51e-05 tok=21.749B dt=5682 wait=260 +15:12:45 pushed ckpt_w155.pt,log_w155.txt +15:16:21 step 165160 loss=3.1400 lr=2.51e-05 tok=21.751B dt=5942 wait=260 +15:16:30 global r=662 load 4.275 -> 4.277 +15:17:01 step 165160 loss=3.0906 lr=2.51e-05 tok=21.753B dt=5983 wait=260 +15:18:04 step 165180 loss=3.1561 lr=2.51e-05 tok=21.755B dt=6046 wait=260 +15:19:07 step 165200 loss=3.1466 lr=2.51e-05 tok=21.758B dt=6109 wait=261 +15:20:11 step 165220 loss=3.2652 lr=2.51e-05 tok=21.761B dt=6173 wait=261 +15:21:15 step 165240 loss=3.1340 lr=2.51e-05 tok=21.763B dt=6237 wait=261 +15:22:20 step 165260 loss=3.3064 lr=2.51e-05 tok=21.766B dt=6301 wait=261 +15:23:23 step 165280 loss=3.1577 lr=2.51e-05 tok=21.768B dt=6365 wait=261 +15:24:26 step 165300 loss=3.1361 lr=2.51e-05 tok=21.771B dt=6428 wait=262 +15:24:38 pushed ckpt_w155.pt,log_w155.txt +15:25:44 waited 66s for global r=663 +15:25:53 global r=663 load 4.270 -> 4.276 +15:26:57 step 165320 loss=3.1899 lr=2.50e-05 tok=21.774B dt=6579 wait=262 +15:28:01 step 165340 loss=3.1164 lr=2.50e-05 tok=21.776B dt=6643 wait=262 +15:29:04 step 165360 loss=3.1557 lr=2.50e-05 tok=21.779B dt=6705 wait=262 +15:30:07 step 165380 loss=3.1502 lr=2.50e-05 tok=21.782B dt=6769 wait=262 +15:31:09 step 165400 loss=3.1912 lr=2.50e-05 tok=21.784B dt=6831 wait=263 +15:32:12 step 165420 loss=3.2333 lr=2.50e-05 tok=21.787B dt=6894 wait=263 +15:33:15 step 165440 loss=3.2075 lr=2.50e-05 tok=21.789B dt=6957 wait=263 +15:34:00 pushed ckpt_w155.pt,log_w155.txt +15:35:00 waited 61s for global r=664 +15:35:07 global r=664 load 4.272 -> 4.273 +15:35:41 step 165460 loss=3.2145 lr=2.50e-05 tok=21.792B dt=7102 wait=263 +15:36:44 step 165480 loss=3.0881 lr=2.50e-05 tok=21.795B dt=7166 wait=263 +15:37:47 step 165500 loss=3.1431 lr=2.49e-05 tok=21.797B dt=7229 wait=264 +15:38:50 step 165520 loss=3.4217 lr=2.49e-05 tok=21.800B dt=7292 wait=264 +15:39:54 step 165540 loss=3.1536 lr=2.49e-05 tok=21.803B dt=7355 wait=264 +15:40:56 step 165560 loss=3.2277 lr=2.49e-05 tok=21.805B dt=7418 wait=264 +15:41:59 step 165580 loss=3.1341 lr=2.49e-05 tok=21.808B dt=7481 wait=264 +15:43:01 step 165600 loss=3.1631 lr=2.49e-05 tok=21.810B dt=7543 wait=265 +15:43:15 pushed ckpt_w155.pt,log_w155.txt +15:44:31 waited 76s for global r=665 +15:44:37 global r=665 load 4.271 -> 4.275 +15:45:41 step 165620 loss=3.1965 lr=2.49e-05 tok=21.813B dt=7703 wait=265 +15:46:44 step 165640 loss=3.1664 lr=2.49e-05 tok=21.816B dt=7766 wait=265 +15:47:47 step 165660 loss=3.1228 lr=2.49e-05 tok=21.818B dt=7829 wait=266 +15:48:49 step 165680 loss=3.2022 lr=2.48e-05 tok=21.821B dt=7891 wait=266 +15:49:52 step 165700 loss=3.1381 lr=2.48e-05 tok=21.823B dt=7953 wait=266 +15:50:54 step 165720 loss=3.3019 lr=2.48e-05 tok=21.826B dt=8015 wait=266 +15:51:56 step 165740 loss=3.0978 lr=2.48e-05 tok=21.829B dt=8078 wait=266 +15:52:40 pushed ckpt_w155.pt,log_w155.txt +15:54:11 waited 91s for global r=666 +15:54:17 global r=666 load 4.277 -> 4.276 +15:54:50 step 165760 loss=3.3067 lr=2.48e-05 tok=21.831B dt=8251 wait=267 +15:55:53 step 165780 loss=3.1567 lr=2.48e-05 tok=21.834B dt=8315 wait=267 +15:56:56 step 165800 loss=3.1859 lr=2.48e-05 tok=21.837B dt=8378 wait=267 +15:57:59 step 165820 loss=3.2750 lr=2.48e-05 tok=21.839B dt=8441 wait=267 +15:59:02 step 165840 loss=3.1768 lr=2.48e-05 tok=21.842B dt=8504 wait=267 +16:00:05 step 165860 loss=3.2719 lr=2.47e-05 tok=21.844B dt=8567 wait=268 +16:01:08 step 165880 loss=3.1832 lr=2.47e-05 tok=21.847B dt=8630 wait=268 +16:02:11 step 165900 loss=3.1876 lr=2.47e-05 tok=21.850B dt=8693 wait=268 +16:02:26 pushed ckpt_w155.pt,log_w155.txt +16:03:42 waited 76s for global r=667 +16:03:47 global r=667 load 4.268 -> 4.273 +16:04:51 step 165920 loss=3.1373 lr=2.47e-05 tok=21.852B dt=8853 wait=268 +16:05:54 step 165940 loss=3.1845 lr=2.47e-05 tok=21.855B dt=8916 wait=269 +16:06:56 step 165960 loss=3.2090 lr=2.47e-05 tok=21.858B dt=8978 wait=269 +16:07:58 step 165980 loss=3.1277 lr=2.47e-05 tok=21.860B dt=9040 wait=269 +16:09:01 step 166000 loss=3.1573 lr=2.47e-05 tok=21.863B dt=9103 wait=269 +16:10:04 step 166020 loss=3.2634 lr=2.47e-05 tok=21.865B dt=9166 wait=269 +16:11:07 step 166040 loss=3.1569 lr=2.47e-05 tok=21.868B dt=9228 wait=269 +16:11:50 pushed ckpt_w155.pt,log_w155.txt +16:15:26 step 166060 loss=3.1745 lr=2.46e-05 tok=21.871B dt=9488 wait=270 +16:15:32 global r=668 load 4.276 -> 4.276 +16:16:03 step 166060 loss=3.2296 lr=2.46e-05 tok=21.872B dt=9525 wait=270 +16:17:06 step 166080 loss=3.1568 lr=2.46e-05 tok=21.875B dt=9588 wait=270 +16:18:08 step 166100 loss=3.0601 lr=2.46e-05 tok=21.877B dt=9650 wait=270 +16:19:11 step 166120 loss=3.1427 lr=2.46e-05 tok=21.880B dt=9713 wait=270 +16:20:15 step 166140 loss=3.1080 lr=2.46e-05 tok=21.882B dt=9776 wait=270 +16:21:17 step 166160 loss=3.2544 lr=2.46e-05 tok=21.885B dt=9839 wait=271 +16:22:21 step 166180 loss=3.1272 lr=2.46e-05 tok=21.888B dt=9902 wait=271 +16:23:25 step 166200 loss=3.0576 lr=2.46e-05 tok=21.890B dt=9967 wait=271 +16:23:37 pushed ckpt_w155.pt,log_w155.txt +16:27:19 global r=669 load 4.266 -> 4.276 +16:28:22 step 166220 loss=3.0903 lr=2.46e-05 tok=21.894B dt=10263 wait=271 +16:29:25 step 166240 loss=3.2026 lr=2.45e-05 tok=21.897B dt=10327 wait=272 +16:30:28 step 166260 loss=3.1707 lr=2.45e-05 tok=21.900B dt=10390 wait=272 +16:31:32 step 166280 loss=3.1138 lr=2.45e-05 tok=21.902B dt=10454 wait=272 +16:32:35 step 166300 loss=3.1581 lr=2.45e-05 tok=21.905B dt=10517 wait=272 +16:33:39 step 166320 loss=3.2854 lr=2.45e-05 tok=21.907B dt=10580 wait=272 +16:34:43 step 166340 loss=3.1704 lr=2.45e-05 tok=21.910B dt=10644 wait=272 +16:35:29 pushed ckpt_w155.pt,log_w155.txt +16:39:05 step 166360 loss=3.2558 lr=2.45e-05 tok=21.913B dt=10906 wait=273 +16:39:11 global r=670 load 4.278 -> 4.282 +16:39:43 step 166360 loss=3.2563 lr=2.45e-05 tok=21.914B dt=10945 wait=273 +16:40:47 step 166380 loss=3.0904 lr=2.45e-05 tok=21.917B dt=11009 wait=273 +16:41:50 step 166400 loss=3.1401 lr=2.45e-05 tok=21.919B dt=11072 wait=273 +16:42:54 step 166420 loss=3.1139 lr=2.44e-05 tok=21.922B dt=11136 wait=273 +16:43:58 step 166440 loss=3.0880 lr=2.44e-05 tok=21.924B dt=11199 wait=274 +16:45:01 step 166460 loss=3.1362 lr=2.44e-05 tok=21.927B dt=11263 wait=274 +16:46:04 step 166480 loss=3.1101 lr=2.44e-05 tok=21.930B dt=11326 wait=274 +16:47:08 step 166500 loss=3.1388 lr=2.44e-05 tok=21.932B dt=11389 wait=274 +16:47:20 pushed ckpt_w155.pt,log_w155.txt +launch +16:48:19 name=w155 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 102/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:48:24 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:48:24 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:48:34 params 1.08B +16:48:35 resumed step=166350 seen=21.93B round=670 +16:48:35 stream salt=166350 +16:50:23 blocks compiled +16:50:23 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:54:14 step 166351 loss=2.8834 lr=2.45e-05 tok=21.935B dt=231 wait=222 +16:54:21 step 166352 loss=2.9610 lr=2.45e-05 tok=21.935B dt=239 wait=222 +16:54:47 step 166360 loss=3.3121 lr=2.45e-05 tok=21.936B dt=264 wait=222 +16:54:59 global r=671 load 4.283 -> 4.284 +16:56:04 step 166520 loss=3.1487 lr=2.44e-05 tok=21.939B dt=341 wait=222 +16:57:08 step 166540 loss=3.0184 lr=2.44e-05 tok=21.941B dt=406 wait=222 +16:58:12 step 166560 loss=3.2992 lr=2.44e-05 tok=21.944B dt=469 wait=222 +16:59:15 step 166580 loss=3.0976 lr=2.44e-05 tok=21.947B dt=533 wait=222 +17:00:19 step 166600 loss=3.1874 lr=2.43e-05 tok=21.949B dt=597 wait=222 +17:01:24 step 166620 loss=3.3593 lr=2.43e-05 tok=21.952B dt=661 wait=223 +17:02:28 step 166640 loss=3.1917 lr=2.43e-05 tok=21.955B dt=726 wait=223 +17:03:16 pushed ckpt_w155.pt,log_w155.txt +launch +17:04:41 name=w155 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 102/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:04:45 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:04:45 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:04:55 params 1.08B +17:04:56 resumed step=166500 seen=21.95B round=671 +17:04:56 stream salt=166500 +17:06:43 blocks compiled +17:06:43 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:13:22 mix source 8 paused 30 min (no parquet files for openbmb/UltraData-Math UltraData-Math-L3-Textbook-Exercise-) +17:13:34 step 166501 loss=2.8538 lr=2.44e-05 tok=21.955B dt=411 wait=400 +17:13:41 step 166502 loss=2.9967 lr=2.44e-05 tok=21.955B dt=418 wait=400 +17:14:20 global r=672 load 4.289 -> 4.280 +17:14:52 step 166660 loss=3.3340 lr=2.43e-05 tok=21.957B dt=489 wait=400 +17:15:56 step 166680 loss=3.1631 lr=2.43e-05 tok=21.960B dt=554 wait=400 +17:16:38 global r=673 load 4.281 -> 4.268 +17:17:42 step 166820 loss=3.2501 lr=2.42e-05 tok=21.964B dt=659 wait=400 +17:18:46 step 166840 loss=3.3884 lr=2.42e-05 tok=21.966B dt=723 wait=400 +launch +17:19:45 name=w155 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 102/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:19:50 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:19:50 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:20:00 params 1.08B +17:20:01 resumed step=166800 seen=21.98B round=673 +17:20:01 stream salt=166800 +17:21:50 blocks compiled +17:21:50 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:23:07 mix source 0 paused 30 min ((Request ID: Root=1-6abbf3fb-5a134b0749f49daa4ecb8698;fed82737-8ff9-444e-b45e-f5) +17:23:48 mix source 5 paused 30 min ((Request ID: Root=1-6abbf424-608ea3dc3eaf96741d3bcbba;d4875fb7-6e6d-4775-a8ea-22) +17:23:48 mix source 6 paused 30 min ((Request ID: Root=1-6abbf424-43a0443258624bc639a27ecb;dd4e905a-19f2-469f-9ae7-b6) +17:27:40 step 166801 loss=2.8555 lr=2.42e-05 tok=21.976B dt=350 wait=341 +17:27:48 step 166802 loss=2.8703 lr=2.42e-05 tok=21.976B dt=358 wait=341 +17:28:45 step 166820 loss=3.0802 lr=2.42e-05 tok=21.978B dt=415 wait=341 +17:29:50 step 166840 loss=3.0365 lr=2.42e-05 tok=21.981B dt=479 wait=341 +17:30:54 step 166860 loss=3.0527 lr=2.42e-05 tok=21.983B dt=543 wait=341 +17:31:57 step 166880 loss=3.1242 lr=2.42e-05 tok=21.986B dt=607 wait=342 +17:33:01 step 166900 loss=3.0990 lr=2.42e-05 tok=21.989B dt=671 wait=342 +17:34:06 step 166920 loss=3.1834 lr=2.42e-05 tok=21.991B dt=736 wait=342 +17:35:10 step 166940 loss=3.0459 lr=2.42e-05 tok=21.994B dt=800 wait=342 +17:35:57 pushed ckpt_w155.pt,log_w155.txt +17:35:57 waited 0s for global r=674 +17:36:09 global r=674 load 4.282 -> 4.258 +17:36:44 step 166960 loss=3.2581 lr=2.42e-05 tok=21.997B dt=893 wait=343 +launch +17:37:25 name=w155 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 102/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:37:31 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:37:31 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:37:40 params 1.08B +17:37:42 resumed step=166950 seen=22.00B round=674 +17:37:42 stream salt=166950 +17:39:37 blocks compiled +17:39:37 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:44:09 step 166951 loss=2.8834 lr=2.42e-05 tok=21.995B dt=272 wait=261 +17:44:17 step 166952 loss=3.1532 lr=2.42e-05 tok=21.995B dt=280 wait=261 +17:44:42 step 166960 loss=3.3037 lr=2.42e-05 tok=21.997B dt=306 wait=261 +17:45:48 step 166980 loss=3.0506 lr=2.41e-05 tok=21.999B dt=371 wait=261 +17:46:53 step 167000 loss=3.1435 lr=2.41e-05 tok=22.002B dt=437 wait=261 +17:47:37 global r=675 load 4.286 -> 4.260 +17:48:42 step 167120 loss=3.0892 lr=2.41e-05 tok=22.006B dt=545 wait=261 +17:49:46 step 167140 loss=3.2181 lr=2.41e-05 tok=22.008B dt=610 wait=262 +17:50:52 step 167160 loss=3.1367 lr=2.40e-05 tok=22.011B dt=675 wait=262 +17:51:57 step 167180 loss=3.1050 lr=2.40e-05 tok=22.014B dt=740 wait=262 +17:53:02 step 167200 loss=3.0804 lr=2.40e-05 tok=22.016B dt=805 wait=262 +17:54:08 step 167220 loss=2.9871 lr=2.40e-05 tok=22.019B dt=871 wait=262 +17:55:12 step 167240 loss=2.9656 lr=2.40e-05 tok=22.021B dt=936 wait=262 +17:55:57 pushed ckpt_w155.pt,log_w155.txt +17:59:34 step 167260 loss=3.0835 lr=2.40e-05 tok=22.024B dt=1197 wait=262 +17:59:43 global r=676 load 4.289 -> 4.285 +18:00:15 step 167260 loss=3.0966 lr=2.40e-05 tok=22.025B dt=1239 wait=263 +18:01:20 step 167280 loss=2.9611 lr=2.40e-05 tok=22.028B dt=1303 wait=263 +18:02:24 step 167300 loss=2.9234 lr=2.40e-05 tok=22.031B dt=1368 wait=263 +18:03:29 step 167320 loss=3.0411 lr=2.40e-05 tok=22.033B dt=1432 wait=263 +18:04:32 step 167340 loss=2.9832 lr=2.40e-05 tok=22.036B dt=1496 wait=263 +18:05:37 step 167360 loss=2.9777 lr=2.39e-05 tok=22.038B dt=1560 wait=263 +18:06:41 step 167380 loss=2.9818 lr=2.39e-05 tok=22.041B dt=1624 wait=263 +18:07:45 step 167400 loss=3.0460 lr=2.39e-05 tok=22.044B dt=1688 wait=264 +18:07:59 pushed ckpt_w155.pt,log_w155.txt +18:09:15 waited 76s for global r=677 +18:09:24 global r=677 load 4.274 -> 4.280 +18:10:29 step 167420 loss=2.8805 lr=2.39e-05 tok=22.046B dt=1853 wait=264 +18:11:32 step 167440 loss=3.1044 lr=2.39e-05 tok=22.049B dt=1916 wait=264 +18:12:37 step 167460 loss=2.9466 lr=2.39e-05 tok=22.052B dt=1980 wait=264 +18:13:41 step 167480 loss=2.9942 lr=2.39e-05 tok=22.054B dt=2045 wait=264 +18:14:45 step 167500 loss=3.0176 lr=2.39e-05 tok=22.057B dt=2109 wait=264 +18:15:50 step 167520 loss=3.0268 lr=2.39e-05 tok=22.059B dt=2174 wait=264 +18:16:55 step 167540 loss=2.9013 lr=2.38e-05 tok=22.062B dt=2238 wait=264 +18:17:39 pushed ckpt_w155.pt,log_w155.txt +18:19:10 waited 91s for global r=678 +18:19:16 global r=678 load 4.280 -> 4.281 +18:19:50 step 167560 loss=2.9412 lr=2.38e-05 tok=22.065B dt=2413 wait=265 +18:20:54 step 167580 loss=2.8892 lr=2.38e-05 tok=22.067B dt=2478 wait=265 +18:21:58 step 167600 loss=3.0055 lr=2.38e-05 tok=22.070B dt=2542 wait=265 +18:23:03 step 167620 loss=3.0295 lr=2.38e-05 tok=22.073B dt=2606 wait=265 +18:24:07 step 167640 loss=2.9761 lr=2.38e-05 tok=22.075B dt=2670 wait=265 +18:25:11 step 167660 loss=2.9602 lr=2.38e-05 tok=22.078B dt=2734 wait=265 +18:26:16 step 167680 loss=2.9240 lr=2.38e-05 tok=22.080B dt=2799 wait=266 +18:27:20 step 167700 loss=2.9372 lr=2.38e-05 tok=22.083B dt=2863 wait=266 +18:27:36 pushed ckpt_w155.pt,log_w155.txt +18:28:51 waited 76s for global r=679 +18:29:01 global r=679 load 4.288 -> 4.283 +18:30:07 step 167720 loss=2.9372 lr=2.38e-05 tok=22.086B dt=3030 wait=266 +18:31:11 step 167740 loss=3.0281 lr=2.37e-05 tok=22.088B dt=3094 wait=266 +18:32:16 step 167760 loss=2.9892 lr=2.37e-05 tok=22.091B dt=3159 wait=266 +18:33:20 step 167780 loss=3.0246 lr=2.37e-05 tok=22.093B dt=3223 wait=266 +18:34:24 step 167800 loss=2.8855 lr=2.37e-05 tok=22.096B dt=3288 wait=266 +18:35:29 step 167820 loss=3.0636 lr=2.37e-05 tok=22.099B dt=3352 wait=267 +18:36:34 step 167840 loss=2.8272 lr=2.37e-05 tok=22.101B dt=3418 wait=267 +18:37:20 pushed ckpt_w155.pt,log_w155.txt +18:38:35 waited 76s for global r=680 +18:38:46 global r=680 load 4.287 -> 4.282 +18:39:20 step 167860 loss=3.0338 lr=2.37e-05 tok=22.104B dt=3583 wait=267 +18:40:24 step 167880 loss=2.8216 lr=2.37e-05 tok=22.107B dt=3647 wait=267 +18:41:27 step 167900 loss=2.8877 lr=2.37e-05 tok=22.109B dt=3711 wait=267 +18:42:32 step 167920 loss=2.9737 lr=2.36e-05 tok=22.112B dt=3775 wait=267 +18:43:36 step 167940 loss=2.9998 lr=2.36e-05 tok=22.114B dt=3839 wait=267 +18:44:39 step 167960 loss=2.9161 lr=2.36e-05 tok=22.117B dt=3903 wait=268 +18:45:44 step 167980 loss=2.9036 lr=2.36e-05 tok=22.120B dt=3967 wait=268 +18:46:48 step 168000 loss=2.8881 lr=2.36e-05 tok=22.122B dt=4032 wait=268 +18:47:01 pushed ckpt_w155.pt,log_w155.txt +18:48:32 waited 91s for global r=681 +18:48:38 global r=681 load 4.288 -> 4.285 +18:49:44 step 168020 loss=2.8905 lr=2.36e-05 tok=22.125B dt=4207 wait=268 +18:50:48 step 168040 loss=2.9116 lr=2.36e-05 tok=22.128B dt=4271 wait=268 +18:51:52 step 168060 loss=2.9033 lr=2.36e-05 tok=22.130B dt=4335 wait=268 +18:52:56 step 168080 loss=2.9390 lr=2.36e-05 tok=22.133B dt=4400 wait=269 +18:54:01 step 168100 loss=2.9215 lr=2.36e-05 tok=22.135B dt=4464 wait=269 +18:55:05 step 168120 loss=2.9917 lr=2.35e-05 tok=22.138B dt=4529 wait=269 +18:56:09 step 168140 loss=3.0011 lr=2.35e-05 tok=22.141B dt=4593 wait=269 +18:56:54 pushed ckpt_w155.pt,log_w155.txt +18:58:40 waited 106s for global r=682 +18:58:50 global r=682 load 4.283 -> 4.283 +18:59:24 step 168160 loss=2.9143 lr=2.35e-05 tok=22.143B dt=4788 wait=269 +19:00:30 step 168180 loss=2.8232 lr=2.35e-05 tok=22.146B dt=4854 wait=269 +19:01:35 step 168200 loss=2.8474 lr=2.35e-05 tok=22.149B dt=4919 wait=269 +19:02:40 step 168220 loss=2.9306 lr=2.35e-05 tok=22.151B dt=4983 wait=270 +19:03:44 step 168240 loss=2.8911 lr=2.35e-05 tok=22.154B dt=5048 wait=270 +19:04:48 step 168260 loss=3.0123 lr=2.35e-05 tok=22.156B dt=5112 wait=270 +19:05:53 step 168280 loss=2.9655 lr=2.35e-05 tok=22.159B dt=5176 wait=270 +19:06:58 step 168300 loss=2.7096 lr=2.35e-05 tok=22.162B dt=5242 wait=270 +19:07:13 pushed ckpt_w155.pt,log_w155.txt +19:08:28 waited 76s for global r=683 +19:08:35 global r=683 load 4.287 -> 4.281 +19:09:41 step 168320 loss=2.8543 lr=2.34e-05 tok=22.164B dt=5404 wait=270 +19:10:45 step 168340 loss=3.0219 lr=2.34e-05 tok=22.167B dt=5468 wait=270 +19:11:49 step 168360 loss=2.8498 lr=2.34e-05 tok=22.170B dt=5533 wait=271 +19:12:53 step 168380 loss=2.8796 lr=2.34e-05 tok=22.172B dt=5597 wait=271 +19:13:58 step 168400 loss=2.8983 lr=2.34e-05 tok=22.175B dt=5661 wait=271 +19:15:03 step 168420 loss=2.9485 lr=2.34e-05 tok=22.177B dt=5726 wait=271 +19:16:07 step 168440 loss=2.8459 lr=2.34e-05 tok=22.180B dt=5791 wait=271 +19:16:56 pushed ckpt_w155.pt,log_w155.txt +19:18:12 waited 76s for global r=684 +19:18:21 global r=684 load 4.285 -> 4.282 +19:18:55 step 168460 loss=2.9622 lr=2.34e-05 tok=22.183B dt=5958 wait=271 +19:19:59 step 168480 loss=2.8992 lr=2.34e-05 tok=22.185B dt=6023 wait=271 +19:21:04 step 168500 loss=2.8656 lr=2.33e-05 tok=22.188B dt=6088 wait=272 +19:22:08 step 168520 loss=2.9938 lr=2.33e-05 tok=22.190B dt=6152 wait=272 +19:23:13 step 168540 loss=2.8882 lr=2.33e-05 tok=22.193B dt=6216 wait=272 +19:24:17 step 168560 loss=3.0163 lr=2.33e-05 tok=22.196B dt=6280 wait=272 +19:25:21 step 168580 loss=2.9415 lr=2.33e-05 tok=22.198B dt=6344 wait=272 +19:26:25 step 168600 loss=2.8548 lr=2.33e-05 tok=22.201B dt=6409 wait=272 +19:26:38 pushed ckpt_w155.pt,log_w155.txt +19:27:54 waited 76s for global r=685 +19:28:01 global r=685 load 4.291 -> 4.283 +19:29:06 step 168620 loss=2.8151 lr=2.33e-05 tok=22.204B dt=6570 wait=273 +19:30:11 step 168640 loss=2.8329 lr=2.33e-05 tok=22.206B dt=6634 wait=273 +19:31:15 step 168660 loss=2.8922 lr=2.33e-05 tok=22.209B dt=6699 wait=273 +19:32:19 step 168680 loss=2.9086 lr=2.33e-05 tok=22.211B dt=6763 wait=273 +19:33:23 step 168700 loss=2.9414 lr=2.32e-05 tok=22.214B dt=6826 wait=273 +19:34:27 step 168720 loss=3.0066 lr=2.32e-05 tok=22.217B dt=6890 wait=273 +19:35:32 step 168740 loss=2.8268 lr=2.32e-05 tok=22.219B dt=6955 wait=273 +19:36:17 pushed ckpt_w155.pt,log_w155.txt +19:37:47 waited 91s for global r=686 +19:37:52 global r=686 load 4.290 -> 4.286 +19:38:26 step 168760 loss=2.9075 lr=2.32e-05 tok=22.222B dt=7130 wait=274 +19:39:31 step 168780 loss=2.8566 lr=2.32e-05 tok=22.225B dt=7194 wait=274 +19:40:36 step 168800 loss=2.9057 lr=2.32e-05 tok=22.227B dt=7259 wait=274 +19:41:40 step 168820 loss=2.9791 lr=2.32e-05 tok=22.230B dt=7323 wait=274 +19:42:44 step 168840 loss=2.9126 lr=2.32e-05 tok=22.232B dt=7388 wait=274 +19:43:49 step 168860 loss=3.0665 lr=2.32e-05 tok=22.235B dt=7453 wait=274 +19:44:54 step 168880 loss=2.9786 lr=2.32e-05 tok=22.238B dt=7517 wait=274 +19:45:59 step 168900 loss=3.0008 lr=2.31e-05 tok=22.240B dt=7582 wait=275 +19:46:12 pushed ckpt_w155.pt,log_w155.txt +19:47:28 waited 76s for global r=687 +19:47:33 global r=687 load 4.292 -> 4.284 +19:48:40 step 168920 loss=3.0698 lr=2.31e-05 tok=22.243B dt=7743 wait=275 +19:49:45 step 168940 loss=3.0437 lr=2.31e-05 tok=22.246B dt=7808 wait=275 +19:50:49 step 168960 loss=3.0881 lr=2.31e-05 tok=22.248B dt=7873 wait=275 +19:51:54 step 168980 loss=3.0031 lr=2.31e-05 tok=22.251B dt=7938 wait=275 +19:52:59 step 169000 loss=2.9864 lr=2.31e-05 tok=22.253B dt=8002 wait=275 +19:54:03 step 169020 loss=3.0533 lr=2.31e-05 tok=22.256B dt=8066 wait=275 +19:55:07 step 169040 loss=2.9578 lr=2.31e-05 tok=22.259B dt=8130 wait=276 +19:55:53 pushed ckpt_w155.pt,log_w155.txt +19:57:24 waited 91s for global r=688 +19:57:31 global r=688 load 4.281 -> 4.284 +19:58:05 step 169060 loss=3.0231 lr=2.31e-05 tok=22.261B dt=8309 wait=276 +19:59:10 step 169080 loss=2.8003 lr=2.31e-05 tok=22.264B dt=8373 wait=276 +20:00:13 step 169100 loss=2.9502 lr=2.30e-05 tok=22.267B dt=8437 wait=276 +20:01:17 step 169120 loss=3.0057 lr=2.30e-05 tok=22.269B dt=8500 wait=276 +20:02:21 step 169140 loss=2.5886 lr=2.30e-05 tok=22.272B dt=8565 wait=276 +20:03:24 step 169160 loss=2.9211 lr=2.30e-05 tok=22.274B dt=8628 wait=276 +20:04:28 step 169180 loss=2.8939 lr=2.30e-05 tok=22.277B dt=8692 wait=277 +20:05:32 step 169200 loss=2.9627 lr=2.30e-05 tok=22.280B dt=8756 wait=277 +20:05:45 pushed ckpt_w155.pt,log_w155.txt +20:09:28 global r=689 load 4.280 -> 4.280 +20:10:32 step 169220 loss=2.9456 lr=2.30e-05 tok=22.284B dt=9055 wait=277 +20:11:35 step 169240 loss=2.9715 lr=2.30e-05 tok=22.286B dt=9119 wait=277 +20:12:39 step 169260 loss=2.9676 lr=2.30e-05 tok=22.289B dt=9183 wait=277 +20:13:44 step 169280 loss=2.9771 lr=2.30e-05 tok=22.291B dt=9247 wait=277 +20:14:47 step 169300 loss=2.8788 lr=2.29e-05 tok=22.294B dt=9311 wait=278 +20:15:52 step 169320 loss=2.9885 lr=2.29e-05 tok=22.297B dt=9375 wait=278 +20:16:56 step 169340 loss=2.8370 lr=2.29e-05 tok=22.299B dt=9440 wait=278 +20:17:41 pushed ckpt_w155.pt,log_w155.txt +20:18:57 waited 76s for global r=690 +20:19:02 global r=690 load 4.286 -> 4.284 +20:19:35 step 169360 loss=2.9746 lr=2.29e-05 tok=22.302B dt=9598 wait=278 +20:20:39 step 169380 loss=2.9379 lr=2.29e-05 tok=22.305B dt=9662 wait=278 +20:21:43 step 169400 loss=2.9021 lr=2.29e-05 tok=22.307B dt=9727 wait=278 +20:22:48 step 169420 loss=3.0603 lr=2.29e-05 tok=22.310B dt=9791 wait=279 +20:23:52 step 169440 loss=2.7908 lr=2.29e-05 tok=22.312B dt=9855 wait=279 +20:24:55 step 169460 loss=3.0398 lr=2.29e-05 tok=22.315B dt=9919 wait=279 +20:26:00 step 169480 loss=2.9264 lr=2.29e-05 tok=22.318B dt=9984 wait=279 +20:27:04 step 169500 loss=2.9907 lr=2.28e-05 tok=22.320B dt=10048 wait=279 +20:27:19 pushed ckpt_w155.pt,log_w155.txt +20:28:35 waited 76s for global r=691 +20:28:41 global r=691 load 4.289 -> 4.287 +20:29:46 step 169520 loss=2.9185 lr=2.28e-05 tok=22.323B dt=10209 wait=279 +20:30:50 step 169540 loss=2.9260 lr=2.28e-05 tok=22.325B dt=10273 wait=279 +20:31:54 step 169560 loss=2.9135 lr=2.28e-05 tok=22.328B dt=10338 wait=280 +20:32:58 step 169580 loss=2.9105 lr=2.28e-05 tok=22.331B dt=10402 wait=280 +20:34:02 step 169600 loss=2.8810 lr=2.28e-05 tok=22.333B dt=10466 wait=280 +20:35:07 step 169620 loss=2.9947 lr=2.28e-05 tok=22.336B dt=10530 wait=280 +20:36:11 step 169640 loss=2.9162 lr=2.28e-05 tok=22.339B dt=10594 wait=280 +20:36:56 pushed ckpt_w155.pt,log_w155.txt +20:37:57 waited 61s for global r=692 +20:38:04 global r=692 load 4.288 -> 4.284 +20:38:38 step 169660 loss=3.1140 lr=2.28e-05 tok=22.341B dt=10741 wait=280 +20:39:42 step 169680 loss=3.0303 lr=2.28e-05 tok=22.344B dt=10805 wait=281 +20:40:46 step 169700 loss=3.0045 lr=2.27e-05 tok=22.346B dt=10869 wait=281 +20:41:50 step 169720 loss=3.0222 lr=2.27e-05 tok=22.349B dt=10934 wait=281 +20:42:55 step 169740 loss=2.8424 lr=2.27e-05 tok=22.352B dt=10998 wait=281 +20:43:59 step 169760 loss=3.0089 lr=2.27e-05 tok=22.354B dt=11062 wait=281 +20:45:03 step 169780 loss=2.8823 lr=2.27e-05 tok=22.357B dt=11127 wait=281 +20:46:08 step 169800 loss=2.9548 lr=2.27e-05 tok=22.360B dt=11191 wait=281 +20:46:20 pushed ckpt_w155.pt,log_w155.txt +20:47:36 waited 76s for global r=693 +20:47:43 global r=693 load 4.290 -> 4.282 +20:48:49 step 169820 loss=2.9947 lr=2.27e-05 tok=22.362B dt=11352 wait=282 +20:49:53 step 169840 loss=2.9135 lr=2.27e-05 tok=22.365B dt=11416 wait=282 +20:50:57 step 169860 loss=2.9779 lr=2.27e-05 tok=22.367B dt=11480 wait=282 +20:52:01 step 169880 loss=2.9038 lr=2.27e-05 tok=22.370B dt=11544 wait=282 +20:53:05 step 169900 loss=2.9984 lr=2.26e-05 tok=22.373B dt=11608 wait=282 +20:54:09 step 169920 loss=2.9515 lr=2.26e-05 tok=22.375B dt=11672 wait=282 +20:55:14 step 169940 loss=2.8212 lr=2.26e-05 tok=22.378B dt=11737 wait=283 +20:56:01 pushed ckpt_w155.pt,log_w155.txt +20:57:32 waited 91s for global r=694 +20:57:40 global r=694 load 4.286 -> 4.282 +20:58:14 step 169960 loss=2.9066 lr=2.26e-05 tok=22.381B dt=11917 wait=283 +20:59:19 step 169980 loss=2.8137 lr=2.26e-05 tok=22.383B dt=11983 wait=283 +21:00:24 step 170000 loss=2.8695 lr=2.26e-05 tok=22.386B dt=12047 wait=283 +21:01:29 step 170020 loss=2.9038 lr=2.26e-05 tok=22.388B dt=12113 wait=283 +21:02:35 step 170040 loss=2.7755 lr=2.26e-05 tok=22.391B dt=12178 wait=283 +21:03:40 step 170060 loss=2.9484 lr=2.26e-05 tok=22.394B dt=12243 wait=283 +21:04:45 step 170080 loss=2.8412 lr=2.26e-05 tok=22.396B dt=12308 wait=284 +21:05:50 step 170100 loss=2.8861 lr=2.26e-05 tok=22.399B dt=12373 wait=284 +21:06:06 pushed ckpt_w155.pt,log_w155.txt +21:07:06 waited 61s for global r=695 +21:07:16 global r=695 load 4.304 -> 4.282 +21:08:22 step 170120 loss=2.9001 lr=2.25e-05 tok=22.402B dt=12526 wait=284 +21:09:28 step 170140 loss=2.8827 lr=2.25e-05 tok=22.404B dt=12591 wait=284 +21:10:33 step 170160 loss=2.9000 lr=2.25e-05 tok=22.407B dt=12656 wait=284 +21:11:38 step 170180 loss=2.8361 lr=2.25e-05 tok=22.409B dt=12722 wait=284 +21:12:44 step 170200 loss=2.7777 lr=2.25e-05 tok=22.412B dt=12787 wait=284 +21:13:49 step 170220 loss=2.8726 lr=2.25e-05 tok=22.415B dt=12853 wait=285 +21:14:54 step 170240 loss=2.7717 lr=2.25e-05 tok=22.417B dt=12917 wait=285 +21:15:39 pushed ckpt_w155.pt,log_w155.txt +21:19:15 step 170260 loss=2.8552 lr=2.25e-05 tok=22.420B dt=13178 wait=285 +21:19:22 global r=696 load 4.291 -> 4.281 +21:19:55 step 170260 loss=2.9381 lr=2.25e-05 tok=22.421B dt=13218 wait=285 +21:21:00 step 170280 loss=2.7902 lr=2.25e-05 tok=22.424B dt=13283 wait=285 +21:22:06 step 170300 loss=2.8977 lr=2.25e-05 tok=22.426B dt=13349 wait=285 +21:23:12 step 170320 loss=2.8704 lr=2.24e-05 tok=22.429B dt=13415 wait=285 +21:24:18 step 170340 loss=2.7293 lr=2.24e-05 tok=22.432B dt=13481 wait=286 +21:25:23 step 170360 loss=2.9438 lr=2.24e-05 tok=22.434B dt=13546 wait=286 +21:26:28 step 170380 loss=2.7444 lr=2.24e-05 tok=22.437B dt=13611 wait=286 +21:27:34 step 170400 loss=2.7630 lr=2.24e-05 tok=22.440B dt=13677 wait=286 +21:27:48 pushed ckpt_w155.pt,log_w155.txt +21:30:49 waited 181s for global r=697 +21:30:55 global r=697 load 4.308 -> 4.282 +21:32:02 step 170420 loss=2.7605 lr=2.24e-05 tok=22.442B dt=13945 wait=286 +21:33:08 step 170440 loss=2.8006 lr=2.24e-05 tok=22.445B dt=14011 wait=286 +21:34:13 step 170460 loss=2.8007 lr=2.24e-05 tok=22.447B dt=14076 wait=287 +21:35:18 step 170480 loss=2.7706 lr=2.24e-05 tok=22.450B dt=14142 wait=287 +21:36:24 step 170500 loss=2.7201 lr=2.24e-05 tok=22.453B dt=14207 wait=287 +21:37:29 step 170520 loss=2.8095 lr=2.23e-05 tok=22.455B dt=14273 wait=287 +21:38:34 step 170540 loss=2.7439 lr=2.23e-05 tok=22.458B dt=14338 wait=287 +21:39:19 pushed ckpt_w155.pt,log_w155.txt +21:42:54 step 170560 loss=2.9125 lr=2.23e-05 tok=22.460B dt=14598 wait=287 +21:43:02 global r=698 load 4.297 -> 4.285 +21:43:34 step 170560 loss=2.9947 lr=2.23e-05 tok=22.462B dt=14638 wait=287 +21:44:39 step 170580 loss=2.8076 lr=2.23e-05 tok=22.464B dt=14702 wait=288 +21:45:42 step 170600 loss=2.8521 lr=2.23e-05 tok=22.467B dt=14766 wait=288 +21:46:47 step 170620 loss=2.8821 lr=2.23e-05 tok=22.470B dt=14830 wait=288 +21:47:50 step 170640 loss=2.8354 lr=2.23e-05 tok=22.472B dt=14894 wait=288 +21:48:55 step 170660 loss=2.9450 lr=2.23e-05 tok=22.475B dt=14958 wait=288 +21:49:59 step 170680 loss=2.9016 lr=2.23e-05 tok=22.478B dt=15022 wait=288 +21:51:03 step 170700 loss=3.0400 lr=2.23e-05 tok=22.480B dt=15086 wait=289 +21:51:21 pushed ckpt_w155.pt,log_w155.txt +21:54:07 waited 166s for global r=699 +21:54:13 global r=699 load 4.288 -> 4.282 +21:55:18 step 170720 loss=2.9619 lr=2.23e-05 tok=22.483B dt=15341 wait=289 +21:56:23 step 170740 loss=3.0069 lr=2.22e-05 tok=22.485B dt=15406 wait=289 +21:57:27 step 170760 loss=3.1083 lr=2.22e-05 tok=22.488B dt=15470 wait=289 +21:58:31 step 170780 loss=3.0300 lr=2.22e-05 tok=22.491B dt=15534 wait=289 +21:59:35 step 170800 loss=2.9637 lr=2.22e-05 tok=22.493B dt=15598 wait=289 +22:00:38 step 170820 loss=3.1673 lr=2.22e-05 tok=22.496B dt=15662 wait=289 +22:01:42 step 170840 loss=2.9288 lr=2.22e-05 tok=22.499B dt=15725 wait=290 +22:02:28 pushed ckpt_w155.pt,log_w155.txt +22:06:03 step 170860 loss=3.1104 lr=2.22e-05 tok=22.501B dt=15986 wait=290 +22:06:10 global r=700 load 4.281 -> 4.283 +22:06:43 step 170860 loss=3.0429 lr=2.22e-05 tok=22.502B dt=16026 wait=290 +22:07:47 step 170880 loss=2.9366 lr=2.22e-05 tok=22.505B dt=16090 wait=290 +22:08:51 step 170900 loss=2.9366 lr=2.22e-05 tok=22.508B dt=16154 wait=291 +22:09:55 step 170920 loss=2.9632 lr=2.22e-05 tok=22.510B dt=16219 wait=291 +22:10:59 step 170940 loss=2.6248 lr=2.21e-05 tok=22.513B dt=16283 wait=291 +22:12:03 step 170960 loss=2.9646 lr=2.21e-05 tok=22.516B dt=16346 wait=291 +22:13:06 step 170980 loss=2.8411 lr=2.21e-05 tok=22.518B dt=16409 wait=291 +22:14:10 step 171000 loss=2.8926 lr=2.21e-05 tok=22.521B dt=16473 wait=291 +22:14:27 pushed ckpt_w155.pt,log_w155.txt +22:17:15 waited 168s for global r=701 +22:17:23 global r=701 load 4.290 -> 4.287 +22:18:29 step 171020 loss=2.8222 lr=2.21e-05 tok=22.523B dt=16733 wait=292 +22:19:34 step 171040 loss=3.0136 lr=2.21e-05 tok=22.526B dt=16797 wait=292 +22:20:38 step 171060 loss=2.9376 lr=2.21e-05 tok=22.529B dt=16861 wait=292 +22:21:42 step 171080 loss=2.9649 lr=2.21e-05 tok=22.531B dt=16925 wait=292 +22:22:46 step 171100 loss=2.8833 lr=2.21e-05 tok=22.534B dt=16989 wait=292 +22:23:50 step 171120 loss=2.9513 lr=2.21e-05 tok=22.537B dt=17053 wait=292 +22:24:54 step 171140 loss=2.9143 lr=2.21e-05 tok=22.539B dt=17117 wait=292 +22:25:42 pushed ckpt_w155.pt,log_w155.txt +22:29:17 step 171160 loss=3.0449 lr=2.20e-05 tok=22.542B dt=17381 wait=293 +22:29:29 global r=702 load 4.290 -> 4.284 +22:30:01 step 171160 loss=3.0403 lr=2.20e-05 tok=22.543B dt=17424 wait=293 +22:31:05 step 171180 loss=2.8643 lr=2.20e-05 tok=22.546B dt=17489 wait=293 +22:32:10 step 171200 loss=2.7075 lr=2.20e-05 tok=22.548B dt=17554 wait=293 +22:33:14 step 171220 loss=3.0439 lr=2.20e-05 tok=22.551B dt=17618 wait=293 +22:34:18 step 171240 loss=2.8991 lr=2.20e-05 tok=22.554B dt=17682 wait=293 +22:35:22 step 171260 loss=3.0290 lr=2.20e-05 tok=22.556B dt=17746 wait=293 +22:36:26 step 171280 loss=2.9014 lr=2.20e-05 tok=22.559B dt=17809 wait=294 +22:37:29 step 171300 loss=3.0579 lr=2.20e-05 tok=22.561B dt=17873 wait=294 +22:37:42 pushed ckpt_w155.pt,log_w155.txt +22:40:43 waited 181s for global r=703 +22:40:49 global r=703 load 4.278 -> 4.279 +22:41:54 step 171320 loss=2.8678 lr=2.20e-05 tok=22.564B dt=18137 wait=294 +22:42:58 step 171340 loss=2.9052 lr=2.20e-05 tok=22.567B dt=18201 wait=294 +22:44:01 step 171360 loss=2.8547 lr=2.19e-05 tok=22.569B dt=18265 wait=294 +22:45:06 step 171380 loss=2.8487 lr=2.19e-05 tok=22.572B dt=18329 wait=294 +22:46:09 step 171400 loss=2.8820 lr=2.19e-05 tok=22.575B dt=18392 wait=294 +22:47:13 step 171420 loss=2.9560 lr=2.19e-05 tok=22.577B dt=18456 wait=295 +22:48:16 step 171440 loss=2.8629 lr=2.19e-05 tok=22.580B dt=18520 wait=295 +22:49:02 pushed ckpt_w155.pt,log_w155.txt +22:52:37 step 171460 loss=3.0195 lr=2.19e-05 tok=22.582B dt=18780 wait=295 +22:52:43 global r=704 load 4.277 -> 4.282 +22:53:15 step 171460 loss=2.9694 lr=2.19e-05 tok=22.584B dt=18818 wait=295 +22:54:18 step 171480 loss=2.9078 lr=2.19e-05 tok=22.586B dt=18882 wait=295 +22:55:22 step 171500 loss=2.7019 lr=2.19e-05 tok=22.589B dt=18946 wait=295 +22:56:27 step 171520 loss=2.7837 lr=2.19e-05 tok=22.592B dt=19010 wait=296 +22:57:30 step 171540 loss=2.8703 lr=2.19e-05 tok=22.594B dt=19074 wait=296 +22:58:34 step 171560 loss=2.8662 lr=2.19e-05 tok=22.597B dt=19137 wait=296 +22:59:37 step 171580 loss=2.9004 lr=2.18e-05 tok=22.599B dt=19201 wait=296 +23:00:41 step 171600 loss=2.9352 lr=2.18e-05 tok=22.602B dt=19265 wait=296 +23:00:53 pushed ckpt_w155.pt,log_w155.txt +23:03:40 waited 167s for global r=705 +23:03:48 global r=705 load 4.282 -> 4.282 +23:04:53 step 171620 loss=2.8422 lr=2.18e-05 tok=22.605B dt=19517 wait=296 +23:05:57 step 171640 loss=2.9402 lr=2.18e-05 tok=22.607B dt=19581 wait=296 +23:07:02 step 171660 loss=2.8667 lr=2.18e-05 tok=22.610B dt=19645 wait=297 +23:08:05 step 171680 loss=2.8969 lr=2.18e-05 tok=22.613B dt=19708 wait=297 +23:09:08 step 171700 loss=2.9302 lr=2.18e-05 tok=22.615B dt=19771 wait=297 +23:10:12 step 171720 loss=2.9724 lr=2.18e-05 tok=22.618B dt=19835 wait=297 +23:11:15 step 171740 loss=2.8115 lr=2.18e-05 tok=22.620B dt=19898 wait=297 +23:12:00 pushed ckpt_w155.pt,log_w155.txt +23:15:34 step 171760 loss=2.9389 lr=2.18e-05 tok=22.623B dt=20158 wait=297 +23:15:42 global r=706 load 4.283 -> 4.279 +23:16:14 step 171760 loss=3.0008 lr=2.18e-05 tok=22.624B dt=20197 wait=298 +23:17:17 step 171780 loss=2.8850 lr=2.18e-05 tok=22.627B dt=20261 wait=298 +23:18:21 step 171800 loss=2.9006 lr=2.17e-05 tok=22.630B dt=20325 wait=298 +23:19:25 step 171820 loss=2.9919 lr=2.17e-05 tok=22.632B dt=20388 wait=298 +23:20:28 step 171840 loss=2.8701 lr=2.17e-05 tok=22.635B dt=20451 wait=298 +23:21:31 step 171860 loss=3.0066 lr=2.17e-05 tok=22.637B dt=20514 wait=298 +23:22:35 step 171880 loss=2.9039 lr=2.17e-05 tok=22.640B dt=20578 wait=298 +23:23:39 step 171900 loss=2.8043 lr=2.17e-05 tok=22.643B dt=20642 wait=299 +23:24:06 pushed ckpt_w155.pt,log_w155.txt +23:27:08 waited 181s for global r=707 +23:27:15 global r=707 load 4.285 -> 4.277 +23:28:20 step 171920 loss=2.9327 lr=2.17e-05 tok=22.645B dt=20923 wait=299 +23:29:23 step 171940 loss=2.9558 lr=2.17e-05 tok=22.648B dt=20987 wait=299 +23:30:26 step 171960 loss=2.9614 lr=2.17e-05 tok=22.651B dt=21050 wait=299 +23:31:30 step 171980 loss=2.8551 lr=2.17e-05 tok=22.653B dt=21113 wait=299 +23:32:34 step 172000 loss=2.9333 lr=2.17e-05 tok=22.656B dt=21177 wait=299 +23:33:38 step 172020 loss=2.9603 lr=2.16e-05 tok=22.658B dt=21242 wait=300 +23:34:41 step 172040 loss=2.9232 lr=2.16e-05 tok=22.661B dt=21305 wait=300 +23:35:27 pushed ckpt_w155.pt,log_w155.txt +23:38:29 waited 181s for global r=708 +23:38:36 global r=708 load 4.280 -> 4.281 +23:39:09 step 172060 loss=2.9461 lr=2.16e-05 tok=22.664B dt=21572 wait=300 +23:40:12 step 172080 loss=2.9636 lr=2.16e-05 tok=22.666B dt=21636 wait=300 +23:41:16 step 172100 loss=2.8199 lr=2.16e-05 tok=22.669B dt=21700 wait=300 +23:42:20 step 172120 loss=2.8170 lr=2.16e-05 tok=22.672B dt=21764 wait=300 +23:43:24 step 172140 loss=2.9280 lr=2.16e-05 tok=22.674B dt=21828 wait=301 +23:44:28 step 172160 loss=2.8854 lr=2.16e-05 tok=22.677B dt=21891 wait=301 +23:45:32 step 172180 loss=2.8630 lr=2.16e-05 tok=22.679B dt=21955 wait=301 +23:46:35 step 172200 loss=2.9118 lr=2.16e-05 tok=22.682B dt=22019 wait=301 +23:46:48 pushed ckpt_w155.pt,log_w155.txt +23:50:29 global r=709 load 4.284 -> 4.280 +23:51:32 step 172220 loss=2.8824 lr=2.16e-05 tok=22.686B dt=22316 wait=301 +23:52:35 step 172240 loss=2.9353 lr=2.15e-05 tok=22.689B dt=22378 wait=301 +23:53:38 step 172260 loss=2.9401 lr=2.15e-05 tok=22.691B dt=22441 wait=302 +23:54:41 step 172280 loss=2.8631 lr=2.15e-05 tok=22.694B dt=22505 wait=302 +23:55:44 step 172300 loss=2.9652 lr=2.15e-05 tok=22.696B dt=22567 wait=302 +23:56:47 step 172320 loss=2.9003 lr=2.15e-05 tok=22.699B dt=22631 wait=302 +23:57:50 step 172340 loss=2.8771 lr=2.15e-05 tok=22.702B dt=22694 wait=302 +23:58:34 pushed ckpt_w155.pt,log_w155.txt +00:01:20 waited 166s for global r=710 +00:01:26 global r=710 load 4.280 -> 4.280 +00:01:59 step 172360 loss=2.9284 lr=2.15e-05 tok=22.704B dt=22942 wait=302 +00:03:02 step 172380 loss=2.7896 lr=2.15e-05 tok=22.707B dt=23005 wait=303 +00:04:05 step 172400 loss=2.9327 lr=2.15e-05 tok=22.710B dt=23068 wait=303 +00:05:08 step 172420 loss=3.0773 lr=2.15e-05 tok=22.712B dt=23131 wait=303 +00:06:11 step 172440 loss=2.9017 lr=2.15e-05 tok=22.715B dt=23195 wait=303 +00:07:14 step 172460 loss=3.0215 lr=2.14e-05 tok=22.717B dt=23258 wait=303 +00:08:17 step 172480 loss=2.9658 lr=2.14e-05 tok=22.720B dt=23320 wait=303 +00:09:20 step 172500 loss=2.9996 lr=2.14e-05 tok=22.723B dt=23384 wait=303 +00:09:33 pushed ckpt_w155.pt,log_w155.txt +00:13:39 step 172520 loss=2.9536 lr=2.14e-05 tok=22.725B dt=23642 wait=304 +00:14:42 step 172540 loss=2.9480 lr=2.14e-05 tok=22.728B dt=23705 wait=304 +00:15:44 step 172560 loss=2.9214 lr=2.14e-05 tok=22.731B dt=23768 wait=304 +00:16:47 step 172580 loss=2.8846 lr=2.14e-05 tok=22.733B dt=23831 wait=304 +00:17:50 step 172600 loss=2.9188 lr=2.14e-05 tok=22.736B dt=23893 wait=304 +00:18:52 step 172620 loss=3.0560 lr=2.14e-05 tok=22.738B dt=23955 wait=304 +00:19:55 step 172640 loss=2.9730 lr=2.14e-05 tok=22.741B dt=24018 wait=305 +00:20:39 pushed ckpt_w155.pt,log_w155.txt +00:24:14 step 172660 loss=2.9738 lr=2.14e-05 tok=22.744B dt=24278 wait=305 +00:25:17 step 172680 loss=3.1113 lr=2.13e-05 tok=22.746B dt=24341 wait=305 +00:26:21 step 172700 loss=2.8973 lr=2.13e-05 tok=22.749B dt=24405 wait=305 +00:27:24 step 172720 loss=2.9246 lr=2.13e-05 tok=22.751B dt=24468 wait=305 +00:28:28 step 172740 loss=3.0415 lr=2.13e-05 tok=22.754B dt=24531 wait=305 +00:29:31 step 172760 loss=2.9690 lr=2.13e-05 tok=22.757B dt=24594 wait=305 +00:30:35 step 172780 loss=2.8725 lr=2.13e-05 tok=22.759B dt=24658 wait=306 +00:31:38 step 172800 loss=3.0269 lr=2.13e-05 tok=22.762B dt=24721 wait=306 +00:31:51 pushed ckpt_w155.pt,log_w155.txt +00:35:59 step 172820 loss=2.7502 lr=2.13e-05 tok=22.765B dt=24982 wait=306 +00:37:01 step 172840 loss=2.9895 lr=2.13e-05 tok=22.767B dt=25045 wait=306 +00:38:04 step 172860 loss=3.0243 lr=2.13e-05 tok=22.770B dt=25107 wait=306 +00:39:07 step 172880 loss=2.9198 lr=2.13e-05 tok=22.772B dt=25171 wait=306 +00:40:11 step 172900 loss=2.9495 lr=2.12e-05 tok=22.775B dt=25234 wait=307 +00:41:15 step 172920 loss=3.0384 lr=2.12e-05 tok=22.778B dt=25298 wait=307 +00:42:18 step 172940 loss=2.9612 lr=2.12e-05 tok=22.780B dt=25361 wait=307 +00:43:04 pushed ckpt_w155.pt,log_w155.txt +00:46:40 step 172960 loss=3.0624 lr=2.12e-05 tok=22.783B dt=25623 wait=307 +00:47:43 step 172980 loss=2.9868 lr=2.12e-05 tok=22.786B dt=25687 wait=307 +00:48:46 step 173000 loss=2.9984 lr=2.12e-05 tok=22.788B dt=25750 wait=307 +00:49:50 step 173020 loss=2.9047 lr=2.12e-05 tok=22.791B dt=25814 wait=308 +00:50:55 step 173040 loss=2.9697 lr=2.12e-05 tok=22.793B dt=25878 wait=308 +00:51:59 step 173060 loss=3.1457 lr=2.12e-05 tok=22.796B dt=25942 wait=308 +00:53:02 step 173080 loss=2.9560 lr=2.12e-05 tok=22.799B dt=26006 wait=308 +00:54:06 step 173100 loss=3.0280 lr=2.12e-05 tok=22.801B dt=26070 wait=308 +00:54:20 pushed ckpt_w155.pt,log_w155.txt +00:58:28 step 173120 loss=2.9421 lr=2.11e-05 tok=22.804B dt=26331 wait=308 +00:59:32 step 173140 loss=3.0952 lr=2.11e-05 tok=22.807B dt=26395 wait=309 +01:00:35 step 173160 loss=3.0205 lr=2.11e-05 tok=22.809B dt=26459 wait=309 +01:01:40 step 173180 loss=3.0320 lr=2.11e-05 tok=22.812B dt=26523 wait=309 +01:02:43 step 173200 loss=3.0302 lr=2.11e-05 tok=22.814B dt=26586 wait=309 +01:03:46 step 173220 loss=3.0090 lr=2.11e-05 tok=22.817B dt=26650 wait=309 +01:04:50 step 173240 loss=2.9531 lr=2.11e-05 tok=22.820B dt=26714 wait=309 diff --git a/log_w156.txt b/log_w156.txt new file mode 100644 index 0000000000000000000000000000000000000000..bb64c57d8bb811b927cdf3a55d43adbb4441e342 --- /dev/null +++ b/log_w156.txt @@ -0,0 +1,147 @@ +launch +13:48:05 name=w156 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 103/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:48:12 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:48:12 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:48:21 params 1.08B +13:48:23 resumed step=163800 seen=21.61B round=653 +13:48:23 stream salt=163800 +13:50:53 blocks compiled +13:50:53 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:55:17 step 163801 loss=2.8979 lr=2.59e-05 tok=21.606B dt=264 wait=233 +13:55:33 step 163802 loss=2.9427 lr=2.59e-05 tok=21.606B dt=280 wait=233 +13:56:32 step 163820 loss=3.2704 lr=2.59e-05 tok=21.609B dt=338 wait=234 +13:56:47 global r=654 load 4.269 -> 4.277 +13:57:19 step 163960 loss=3.1663 lr=2.58e-05 tok=21.610B dt=386 wait=234 +13:58:25 step 163980 loss=3.1781 lr=2.58e-05 tok=21.612B dt=451 wait=234 +13:59:29 step 164000 loss=3.2890 lr=2.58e-05 tok=21.615B dt=516 wait=234 +14:00:34 step 164020 loss=3.3053 lr=2.58e-05 tok=21.618B dt=581 wait=234 +14:01:39 step 164040 loss=3.2145 lr=2.58e-05 tok=21.620B dt=645 wait=234 +14:02:44 step 164060 loss=3.1876 lr=2.58e-05 tok=21.623B dt=711 wait=235 +14:03:49 step 164080 loss=3.1435 lr=2.57e-05 tok=21.626B dt=775 wait=235 +14:04:54 step 164100 loss=3.2087 lr=2.57e-05 tok=21.628B dt=840 wait=235 +14:05:07 pushed ckpt_w156.pt,log_w156.txt +14:06:07 waited 61s for global r=655 +14:06:15 global r=655 load 4.280 -> 4.278 +14:07:21 step 164120 loss=3.1879 lr=2.57e-05 tok=21.631B dt=987 wait=235 +14:08:25 step 164140 loss=3.1958 lr=2.57e-05 tok=21.633B dt=1052 wait=235 +14:09:31 step 164160 loss=3.1643 lr=2.57e-05 tok=21.636B dt=1118 wait=236 +14:10:36 step 164180 loss=3.1462 lr=2.57e-05 tok=21.639B dt=1183 wait=236 +14:11:42 step 164200 loss=3.3052 lr=2.57e-05 tok=21.641B dt=1248 wait=236 +14:12:46 step 164220 loss=3.2311 lr=2.57e-05 tok=21.644B dt=1313 wait=236 +14:13:51 step 164240 loss=3.1873 lr=2.57e-05 tok=21.647B dt=1378 wait=236 +14:14:35 pushed ckpt_w156.pt,log_w156.txt +14:15:51 waited 76s for global r=656 +14:16:04 global r=656 load 4.285 -> 4.279 +14:16:38 step 164260 loss=3.2923 lr=2.56e-05 tok=21.649B dt=1545 wait=236 +14:17:43 step 164280 loss=3.2055 lr=2.56e-05 tok=21.652B dt=1610 wait=237 +14:18:48 step 164300 loss=3.1752 lr=2.56e-05 tok=21.654B dt=1674 wait=237 +14:19:52 step 164320 loss=3.3534 lr=2.56e-05 tok=21.657B dt=1739 wait=237 +14:20:57 step 164340 loss=3.1774 lr=2.56e-05 tok=21.660B dt=1803 wait=237 +14:22:02 step 164360 loss=3.2296 lr=2.56e-05 tok=21.662B dt=1868 wait=237 +14:23:06 step 164380 loss=3.1495 lr=2.56e-05 tok=21.665B dt=1933 wait=238 +14:24:11 step 164400 loss=3.2426 lr=2.56e-05 tok=21.668B dt=1997 wait=238 +14:24:25 pushed ckpt_w156.pt,log_w156.txt +14:25:26 waited 61s for global r=657 +14:25:33 global r=657 load 4.283 -> 4.276 +14:26:39 step 164420 loss=3.1110 lr=2.56e-05 tok=21.670B dt=2145 wait=238 +14:27:44 step 164440 loss=3.1680 lr=2.55e-05 tok=21.673B dt=2210 wait=238 +14:28:48 step 164460 loss=3.1367 lr=2.55e-05 tok=21.675B dt=2275 wait=238 +14:29:53 step 164480 loss=3.1096 lr=2.55e-05 tok=21.678B dt=2340 wait=238 +14:30:57 step 164500 loss=3.1674 lr=2.55e-05 tok=21.681B dt=2404 wait=239 +14:32:02 step 164520 loss=3.2330 lr=2.55e-05 tok=21.683B dt=2468 wait=239 +14:33:06 step 164540 loss=3.2096 lr=2.55e-05 tok=21.686B dt=2532 wait=239 +14:33:52 pushed ckpt_w156.pt,log_w156.txt +14:35:08 waited 76s for global r=658 +14:35:13 global r=658 load 4.285 -> 4.278 +14:35:47 step 164560 loss=3.2328 lr=2.55e-05 tok=21.688B dt=2693 wait=239 +14:36:52 step 164580 loss=3.2729 lr=2.55e-05 tok=21.691B dt=2758 wait=239 +14:37:56 step 164600 loss=3.2646 lr=2.55e-05 tok=21.694B dt=2823 wait=240 +14:39:01 step 164620 loss=3.3363 lr=2.54e-05 tok=21.696B dt=2888 wait=240 +14:40:06 step 164640 loss=3.1492 lr=2.54e-05 tok=21.699B dt=2953 wait=240 +14:41:11 step 164660 loss=3.2361 lr=2.54e-05 tok=21.702B dt=3018 wait=240 +14:42:15 step 164680 loss=3.1762 lr=2.54e-05 tok=21.704B dt=3082 wait=240 +14:43:19 step 164700 loss=3.2294 lr=2.54e-05 tok=21.707B dt=3146 wait=241 +14:43:31 pushed ckpt_w156.pt,log_w156.txt +14:44:47 waited 76s for global r=659 +14:44:53 global r=659 load 4.280 -> 4.277 +14:45:58 step 164720 loss=3.2006 lr=2.54e-05 tok=21.709B dt=3304 wait=241 +14:47:02 step 164740 loss=3.1063 lr=2.54e-05 tok=21.712B dt=3368 wait=241 +14:48:06 step 164760 loss=3.2061 lr=2.54e-05 tok=21.715B dt=3432 wait=241 +14:49:10 step 164780 loss=3.2073 lr=2.53e-05 tok=21.717B dt=3496 wait=241 +14:50:14 step 164800 loss=3.2261 lr=2.53e-05 tok=21.720B dt=3560 wait=242 +14:51:18 step 164820 loss=3.2833 lr=2.53e-05 tok=21.723B dt=3625 wait=242 +14:52:22 step 164840 loss=3.1528 lr=2.53e-05 tok=21.725B dt=3689 wait=242 +14:53:09 pushed ckpt_w156.pt,log_w156.txt +14:54:45 waited 96s for global r=660 +14:54:53 global r=660 load 4.283 -> 4.275 +14:55:27 step 164860 loss=3.3470 lr=2.53e-05 tok=21.728B dt=3873 wait=242 +14:56:32 step 164880 loss=3.2297 lr=2.53e-05 tok=21.730B dt=3938 wait=242 +14:57:37 step 164900 loss=3.2962 lr=2.53e-05 tok=21.733B dt=4004 wait=242 +14:58:42 step 164920 loss=3.4012 lr=2.53e-05 tok=21.736B dt=4069 wait=243 +14:59:47 step 164940 loss=3.2610 lr=2.53e-05 tok=21.738B dt=4133 wait=243 +15:00:51 step 164960 loss=3.3059 lr=2.52e-05 tok=21.741B dt=4198 wait=243 +15:01:56 step 164980 loss=3.1769 lr=2.52e-05 tok=21.744B dt=4262 wait=243 +15:03:01 step 165000 loss=3.1359 lr=2.52e-05 tok=21.746B dt=4327 wait=243 +15:03:13 pushed ckpt_w156.pt,log_w156.txt +15:04:44 waited 91s for global r=661 +15:04:52 global r=661 load 4.271 -> 4.275 +15:05:57 step 165020 loss=3.3610 lr=2.52e-05 tok=21.749B dt=4504 wait=243 +15:07:02 step 165040 loss=3.3171 lr=2.52e-05 tok=21.751B dt=4568 wait=244 +15:08:07 step 165060 loss=3.2600 lr=2.52e-05 tok=21.754B dt=4633 wait=244 +15:09:12 step 165080 loss=3.2997 lr=2.52e-05 tok=21.757B dt=4698 wait=244 +15:10:17 step 165100 loss=3.3441 lr=2.52e-05 tok=21.759B dt=4763 wait=244 +15:11:21 step 165120 loss=3.4575 lr=2.52e-05 tok=21.762B dt=4828 wait=244 +15:12:26 step 165140 loss=3.3092 lr=2.51e-05 tok=21.765B dt=4892 wait=245 +15:13:13 pushed ckpt_w156.pt,log_w156.txt +15:15:59 waited 166s for global r=662 +15:16:06 global r=662 load 4.281 -> 4.277 +15:16:39 step 165160 loss=3.4551 lr=2.51e-05 tok=21.767B dt=5145 wait=245 +15:17:43 step 165180 loss=3.3310 lr=2.51e-05 tok=21.770B dt=5210 wait=245 +15:18:48 step 165200 loss=3.2654 lr=2.51e-05 tok=21.772B dt=5274 wait=245 +15:19:52 step 165220 loss=3.4382 lr=2.51e-05 tok=21.775B dt=5339 wait=245 +15:20:56 step 165240 loss=3.3539 lr=2.51e-05 tok=21.778B dt=5403 wait=246 +15:22:01 step 165260 loss=3.3445 lr=2.51e-05 tok=21.780B dt=5467 wait=246 +15:23:05 step 165280 loss=3.3347 lr=2.51e-05 tok=21.783B dt=5531 wait=246 +15:24:09 step 165300 loss=3.3694 lr=2.51e-05 tok=21.785B dt=5596 wait=246 +15:24:25 pushed ckpt_w156.pt,log_w156.txt +15:25:41 waited 76s for global r=663 +15:25:50 global r=663 load 4.277 -> 4.276 +15:26:55 step 165320 loss=3.2189 lr=2.50e-05 tok=21.788B dt=5761 wait=246 +15:27:59 step 165340 loss=3.3249 lr=2.50e-05 tok=21.791B dt=5825 wait=246 +15:29:03 step 165360 loss=3.2721 lr=2.50e-05 tok=21.793B dt=5890 wait=247 +15:30:08 step 165380 loss=3.3916 lr=2.50e-05 tok=21.796B dt=5954 wait=247 +15:31:12 step 165400 loss=3.3221 lr=2.50e-05 tok=21.799B dt=6018 wait=247 +15:32:16 step 165420 loss=3.4195 lr=2.50e-05 tok=21.801B dt=6082 wait=247 +15:33:19 step 165440 loss=3.2885 lr=2.50e-05 tok=21.804B dt=6146 wait=247 +15:34:08 pushed ckpt_w156.pt,log_w156.txt +15:35:08 waited 60s for global r=664 +15:35:16 global r=664 load 4.265 -> 4.273 +15:35:49 step 165460 loss=3.3774 lr=2.50e-05 tok=21.806B dt=6295 wait=248 +15:36:52 step 165480 loss=3.2042 lr=2.50e-05 tok=21.809B dt=6359 wait=248 +15:37:56 step 165500 loss=3.2544 lr=2.49e-05 tok=21.812B dt=6423 wait=248 +15:39:00 step 165520 loss=3.2949 lr=2.49e-05 tok=21.814B dt=6486 wait=248 +15:40:04 step 165540 loss=3.3057 lr=2.49e-05 tok=21.817B dt=6550 wait=248 +15:41:08 step 165560 loss=3.2819 lr=2.49e-05 tok=21.820B dt=6615 wait=249 +15:42:12 step 165580 loss=3.2474 lr=2.49e-05 tok=21.822B dt=6678 wait=249 +15:43:15 step 165600 loss=3.1683 lr=2.49e-05 tok=21.825B dt=6742 wait=249 +15:43:28 pushed ckpt_w156.pt,log_w156.txt +15:44:43 waited 76s for global r=665 +15:44:50 global r=665 load 4.277 -> 4.275 +15:45:54 step 165620 loss=3.1133 lr=2.49e-05 tok=21.827B dt=6901 wait=249 +15:46:58 step 165640 loss=3.1774 lr=2.49e-05 tok=21.830B dt=6965 wait=249 +15:48:03 step 165660 loss=3.1972 lr=2.49e-05 tok=21.833B dt=7030 wait=250 +15:49:08 step 165680 loss=3.2900 lr=2.48e-05 tok=21.835B dt=7094 wait=250 +15:50:13 step 165700 loss=3.2631 lr=2.48e-05 tok=21.838B dt=7159 wait=250 +15:51:17 step 165720 loss=3.2695 lr=2.48e-05 tok=21.841B dt=7224 wait=250 +15:52:22 step 165740 loss=3.0958 lr=2.48e-05 tok=21.843B dt=7288 wait=250 +15:53:07 pushed ckpt_w156.pt,log_w156.txt +15:54:07 waited 61s for global r=666 +15:54:14 global r=666 load 4.275 -> 4.276 +15:54:47 step 165760 loss=3.3462 lr=2.48e-05 tok=21.846B dt=7433 wait=250 +15:55:51 step 165780 loss=3.2951 lr=2.48e-05 tok=21.848B dt=7497 wait=251 +15:56:55 step 165800 loss=3.1680 lr=2.48e-05 tok=21.851B dt=7562 wait=251 +15:57:59 step 165820 loss=3.3282 lr=2.48e-05 tok=21.854B dt=7626 wait=251 +15:59:04 step 165840 loss=3.2610 lr=2.48e-05 tok=21.856B dt=7690 wait=251 +16:00:08 step 165860 loss=3.3421 lr=2.47e-05 tok=21.859B dt=7755 wait=251 +16:01:12 step 165880 loss=3.2270 lr=2.47e-05 tok=21.861B dt=7819 wait=252 +16:02:16 step 165900 loss=3.1902 lr=2.47e-05 tok=21.864B dt=7883 wait=252 diff --git a/log_w157.txt b/log_w157.txt new file mode 100644 index 0000000000000000000000000000000000000000..f68c55d7d535a353654c054d0e9ecd8814fe6ec1 --- /dev/null +++ b/log_w157.txt @@ -0,0 +1,173 @@ +launch +13:23:16 name=w157 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 104/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:23:22 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:23:22 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:23:32 params 1.08B +13:23:34 resumed step=163350 seen=21.55B round=650 +13:23:34 stream salt=163350 +13:25:41 blocks compiled +13:25:41 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:30:01 step 163351 loss=3.1748 lr=2.62e-05 tok=21.547B dt=260 wait=234 +13:30:17 step 163352 loss=3.2561 lr=2.62e-05 tok=21.547B dt=276 wait=234 +13:30:43 step 163360 loss=3.5427 lr=2.62e-05 tok=21.548B dt=302 wait=234 +13:30:59 global r=651 load 4.277 -> 4.282 +13:32:03 step 163520 loss=3.3577 lr=2.61e-05 tok=21.551B dt=382 wait=234 +13:33:07 step 163540 loss=3.3037 lr=2.61e-05 tok=21.553B dt=446 wait=234 +13:34:10 step 163560 loss=3.4532 lr=2.60e-05 tok=21.556B dt=510 wait=235 +13:35:14 step 163580 loss=3.3988 lr=2.60e-05 tok=21.559B dt=573 wait=235 +13:35:55 global r=652 load 4.279 -> 4.278 +13:36:27 step 163660 loss=3.3714 lr=2.60e-05 tok=21.561B dt=647 wait=235 +13:37:31 step 163680 loss=3.3671 lr=2.60e-05 tok=21.564B dt=711 wait=235 +13:38:35 step 163700 loss=3.3177 lr=2.60e-05 tok=21.567B dt=774 wait=235 +13:39:39 step 163720 loss=3.4299 lr=2.60e-05 tok=21.569B dt=838 wait=236 +13:40:43 step 163740 loss=3.3189 lr=2.59e-05 tok=21.572B dt=903 wait=236 +13:41:47 step 163760 loss=3.4873 lr=2.59e-05 tok=21.574B dt=966 wait=236 +13:42:50 step 163780 loss=3.4125 lr=2.59e-05 tok=21.577B dt=1029 wait=236 +13:43:54 step 163800 loss=3.3850 lr=2.59e-05 tok=21.580B dt=1093 wait=237 +13:44:06 pushed ckpt_w157.pt,log_w157.txt +13:44:52 waited 45s for global r=653 +13:44:58 global r=653 load 4.276 -> 4.276 +13:46:03 step 163820 loss=3.3783 lr=2.59e-05 tok=21.582B dt=1222 wait=237 +13:47:06 step 163840 loss=3.4585 lr=2.59e-05 tok=21.585B dt=1285 wait=237 +13:48:09 step 163860 loss=3.5142 lr=2.59e-05 tok=21.588B dt=1348 wait=237 +13:49:12 step 163880 loss=3.3051 lr=2.59e-05 tok=21.590B dt=1411 wait=237 +13:50:14 step 163900 loss=3.3166 lr=2.59e-05 tok=21.593B dt=1473 wait=238 +13:51:17 step 163920 loss=3.4202 lr=2.58e-05 tok=21.595B dt=1536 wait=238 +13:52:20 step 163940 loss=3.2711 lr=2.58e-05 tok=21.598B dt=1599 wait=238 +13:53:04 pushed ckpt_w157.pt,log_w157.txt +13:56:38 step 163960 loss=3.4350 lr=2.58e-05 tok=21.601B dt=1857 wait=238 +13:56:44 global r=654 load 4.278 -> 4.277 +13:57:17 step 163960 loss=3.2199 lr=2.58e-05 tok=21.602B dt=1896 wait=238 +13:58:20 step 163980 loss=3.3488 lr=2.58e-05 tok=21.605B dt=1959 wait=238 +13:59:23 step 164000 loss=3.3140 lr=2.58e-05 tok=21.607B dt=2023 wait=239 +14:00:27 step 164020 loss=3.2082 lr=2.58e-05 tok=21.610B dt=2086 wait=239 +14:01:31 step 164040 loss=3.1756 lr=2.58e-05 tok=21.612B dt=2150 wait=239 +14:02:34 step 164060 loss=3.3194 lr=2.58e-05 tok=21.615B dt=2214 wait=239 +14:03:38 step 164080 loss=3.2890 lr=2.57e-05 tok=21.618B dt=2277 wait=239 +14:04:42 step 164100 loss=3.2829 lr=2.57e-05 tok=21.620B dt=2342 wait=240 +14:04:57 pushed ckpt_w157.pt,log_w157.txt +14:06:12 waited 76s for global r=655 +14:06:22 global r=655 load 4.273 -> 4.278 +14:07:27 step 164120 loss=3.3092 lr=2.57e-05 tok=21.623B dt=2506 wait=240 +14:08:30 step 164140 loss=3.3319 lr=2.57e-05 tok=21.626B dt=2570 wait=240 +14:09:34 step 164160 loss=3.3921 lr=2.57e-05 tok=21.628B dt=2633 wait=240 +14:10:38 step 164180 loss=3.2740 lr=2.57e-05 tok=21.631B dt=2697 wait=240 +14:11:41 step 164200 loss=3.2705 lr=2.57e-05 tok=21.633B dt=2761 wait=241 +14:12:45 step 164220 loss=3.4026 lr=2.57e-05 tok=21.636B dt=2825 wait=241 +14:13:49 step 164240 loss=3.2718 lr=2.57e-05 tok=21.639B dt=2889 wait=241 +14:14:34 pushed ckpt_w157.pt,log_w157.txt +14:15:49 waited 76s for global r=656 +14:16:02 global r=656 load 4.277 -> 4.279 +14:16:39 step 164260 loss=3.2905 lr=2.56e-05 tok=21.641B dt=3059 wait=241 +14:17:43 step 164280 loss=3.2188 lr=2.56e-05 tok=21.644B dt=3122 wait=242 +14:18:46 step 164300 loss=3.2512 lr=2.56e-05 tok=21.647B dt=3186 wait=242 +14:19:50 step 164320 loss=3.3480 lr=2.56e-05 tok=21.649B dt=3250 wait=242 +14:20:55 step 164340 loss=3.1303 lr=2.56e-05 tok=21.652B dt=3314 wait=242 +14:22:00 step 164360 loss=3.2486 lr=2.56e-05 tok=21.654B dt=3379 wait=242 +14:23:04 step 164380 loss=3.1212 lr=2.56e-05 tok=21.657B dt=3444 wait=243 +14:24:09 step 164400 loss=3.2137 lr=2.56e-05 tok=21.660B dt=3508 wait=243 +14:24:21 pushed ckpt_w157.pt,log_w157.txt +14:25:37 waited 76s for global r=657 +14:25:45 global r=657 load 4.277 -> 4.276 +14:26:51 step 164420 loss=3.1132 lr=2.56e-05 tok=21.662B dt=3670 wait=243 +14:27:56 step 164440 loss=3.2245 lr=2.55e-05 tok=21.665B dt=3735 wait=243 +14:29:00 step 164460 loss=3.1989 lr=2.55e-05 tok=21.668B dt=3800 wait=243 +14:30:05 step 164480 loss=3.1412 lr=2.55e-05 tok=21.670B dt=3864 wait=244 +14:31:10 step 164500 loss=3.0934 lr=2.55e-05 tok=21.673B dt=3929 wait=244 +14:32:14 step 164520 loss=3.2325 lr=2.55e-05 tok=21.675B dt=3994 wait=244 +14:33:19 step 164540 loss=3.0905 lr=2.55e-05 tok=21.678B dt=4059 wait=244 +14:34:06 pushed ckpt_w157.pt,log_w157.txt +14:35:06 waited 61s for global r=658 +14:35:12 global r=658 load 4.273 -> 4.278 +14:35:46 step 164560 loss=3.1541 lr=2.55e-05 tok=21.681B dt=4205 wait=244 +14:36:50 step 164580 loss=2.9424 lr=2.55e-05 tok=21.683B dt=4270 wait=245 +14:37:55 step 164600 loss=3.1049 lr=2.55e-05 tok=21.686B dt=4334 wait=245 +14:38:58 step 164620 loss=3.1577 lr=2.54e-05 tok=21.688B dt=4398 wait=245 +14:40:02 step 164640 loss=3.0194 lr=2.54e-05 tok=21.691B dt=4461 wait=245 +14:41:06 step 164660 loss=3.0524 lr=2.54e-05 tok=21.694B dt=4526 wait=245 +14:42:11 step 164680 loss=3.0852 lr=2.54e-05 tok=21.696B dt=4590 wait=246 +14:43:15 step 164700 loss=3.1543 lr=2.54e-05 tok=21.699B dt=4655 wait=246 +14:43:28 pushed ckpt_w157.pt,log_w157.txt +14:44:43 waited 76s for global r=659 +14:44:52 global r=659 load 4.290 -> 4.277 +14:45:57 step 164720 loss=3.1297 lr=2.54e-05 tok=21.702B dt=4816 wait=246 +14:47:01 step 164740 loss=3.2948 lr=2.54e-05 tok=21.704B dt=4880 wait=246 +14:48:06 step 164760 loss=3.1306 lr=2.54e-05 tok=21.707B dt=4945 wait=247 +14:49:09 step 164780 loss=3.1620 lr=2.53e-05 tok=21.709B dt=5008 wait=247 +14:50:14 step 164800 loss=3.1516 lr=2.53e-05 tok=21.712B dt=5073 wait=247 +14:51:18 step 164820 loss=3.1771 lr=2.53e-05 tok=21.715B dt=5137 wait=247 +14:52:21 step 164840 loss=3.0938 lr=2.53e-05 tok=21.717B dt=5201 wait=247 +14:53:07 pushed ckpt_w157.pt,log_w157.txt +14:54:38 waited 91s for global r=660 +14:54:47 global r=660 load 4.285 -> 4.275 +14:55:21 step 164860 loss=3.1578 lr=2.53e-05 tok=21.720B dt=5380 wait=248 +14:56:24 step 164880 loss=3.1226 lr=2.53e-05 tok=21.723B dt=5443 wait=248 +14:57:28 step 164900 loss=3.0825 lr=2.53e-05 tok=21.725B dt=5507 wait=248 +14:58:31 step 164920 loss=3.0750 lr=2.53e-05 tok=21.728B dt=5570 wait=248 +14:59:36 step 164940 loss=3.0187 lr=2.53e-05 tok=21.730B dt=5635 wait=248 +15:00:40 step 164960 loss=3.1520 lr=2.52e-05 tok=21.733B dt=5699 wait=249 +15:01:44 step 164980 loss=3.1150 lr=2.52e-05 tok=21.736B dt=5763 wait=249 +15:02:48 step 165000 loss=3.0154 lr=2.52e-05 tok=21.738B dt=5827 wait=249 +15:03:06 pushed ckpt_w157.pt,log_w157.txt +15:04:37 waited 91s for global r=661 +15:04:46 global r=661 load 4.277 -> 4.275 +15:05:51 step 165020 loss=3.1360 lr=2.52e-05 tok=21.741B dt=6011 wait=249 +15:06:56 step 165040 loss=3.1524 lr=2.52e-05 tok=21.744B dt=6075 wait=249 +15:08:00 step 165060 loss=2.9895 lr=2.52e-05 tok=21.746B dt=6139 wait=250 +15:09:04 step 165080 loss=3.0902 lr=2.52e-05 tok=21.749B dt=6203 wait=250 +15:10:07 step 165100 loss=3.0313 lr=2.52e-05 tok=21.751B dt=6267 wait=250 +15:11:11 step 165120 loss=3.1784 lr=2.52e-05 tok=21.754B dt=6330 wait=250 +15:12:15 step 165140 loss=3.0409 lr=2.51e-05 tok=21.757B dt=6394 wait=250 +15:13:06 pushed ckpt_w157.pt,log_w157.txt +15:15:52 waited 166s for global r=662 +15:16:00 global r=662 load 4.289 -> 4.277 +15:16:33 step 165160 loss=3.1347 lr=2.51e-05 tok=21.759B dt=6653 wait=251 +15:17:38 step 165180 loss=3.0179 lr=2.51e-05 tok=21.762B dt=6717 wait=251 +15:18:42 step 165200 loss=2.9441 lr=2.51e-05 tok=21.765B dt=6781 wait=251 +15:19:46 step 165220 loss=3.0948 lr=2.51e-05 tok=21.767B dt=6845 wait=251 +15:20:50 step 165240 loss=3.0656 lr=2.51e-05 tok=21.770B dt=6909 wait=251 +15:21:54 step 165260 loss=2.9954 lr=2.51e-05 tok=21.772B dt=6973 wait=251 +15:22:58 step 165280 loss=3.0262 lr=2.51e-05 tok=21.775B dt=7038 wait=252 +15:24:02 step 165300 loss=3.0569 lr=2.51e-05 tok=21.778B dt=7101 wait=252 +15:24:16 pushed ckpt_w157.pt,log_w157.txt +15:25:31 waited 76s for global r=663 +15:25:41 global r=663 load 4.280 -> 4.276 +15:26:46 step 165320 loss=3.0643 lr=2.50e-05 tok=21.780B dt=7265 wait=252 +15:27:51 step 165340 loss=2.9454 lr=2.50e-05 tok=21.783B dt=7330 wait=252 +15:28:55 step 165360 loss=3.0204 lr=2.50e-05 tok=21.785B dt=7395 wait=252 +15:29:59 step 165380 loss=2.9196 lr=2.50e-05 tok=21.788B dt=7458 wait=253 +15:31:03 step 165400 loss=2.9446 lr=2.50e-05 tok=21.791B dt=7522 wait=253 +15:32:07 step 165420 loss=3.0395 lr=2.50e-05 tok=21.793B dt=7586 wait=253 +15:33:10 step 165440 loss=2.9229 lr=2.50e-05 tok=21.796B dt=7650 wait=253 +15:33:58 pushed ckpt_w157.pt,log_w157.txt +15:35:14 waited 76s for global r=664 +15:35:26 global r=664 load 4.282 -> 4.273 +15:35:59 step 165460 loss=2.9747 lr=2.50e-05 tok=21.799B dt=7818 wait=253 +15:37:02 step 165480 loss=3.0243 lr=2.50e-05 tok=21.801B dt=7881 wait=254 +15:38:06 step 165500 loss=2.9413 lr=2.49e-05 tok=21.804B dt=7945 wait=254 +15:39:10 step 165520 loss=2.9297 lr=2.49e-05 tok=21.806B dt=8009 wait=254 +15:40:14 step 165540 loss=2.9152 lr=2.49e-05 tok=21.809B dt=8073 wait=255 +15:41:17 step 165560 loss=3.0034 lr=2.49e-05 tok=21.812B dt=8136 wait=255 +15:42:21 step 165580 loss=2.8562 lr=2.49e-05 tok=21.814B dt=8200 wait=255 +15:43:24 step 165600 loss=2.9844 lr=2.49e-05 tok=21.817B dt=8263 wait=255 +15:43:39 pushed ckpt_w157.pt,log_w157.txt +15:44:39 waited 61s for global r=665 +15:44:48 global r=665 load 4.279 -> 4.275 +15:45:52 step 165620 loss=2.9906 lr=2.49e-05 tok=21.820B dt=8411 wait=255 +15:46:55 step 165640 loss=3.0095 lr=2.49e-05 tok=21.822B dt=8475 wait=255 +15:47:59 step 165660 loss=2.9619 lr=2.49e-05 tok=21.825B dt=8538 wait=256 +15:49:03 step 165680 loss=2.9311 lr=2.48e-05 tok=21.827B dt=8602 wait=256 +15:50:06 step 165700 loss=3.0046 lr=2.48e-05 tok=21.830B dt=8666 wait=256 +15:51:10 step 165720 loss=3.1293 lr=2.48e-05 tok=21.833B dt=8729 wait=256 +15:52:14 step 165740 loss=2.9573 lr=2.48e-05 tok=21.835B dt=8793 wait=256 +15:52:58 pushed ckpt_w157.pt,log_w157.txt +15:54:14 waited 76s for global r=666 +15:54:21 global r=666 load 4.281 -> 4.276 +15:54:54 step 165760 loss=3.0949 lr=2.48e-05 tok=21.838B dt=8953 wait=257 +15:55:58 step 165780 loss=2.9546 lr=2.48e-05 tok=21.841B dt=9017 wait=257 +15:57:01 step 165800 loss=2.9822 lr=2.48e-05 tok=21.843B dt=9081 wait=257 +15:58:05 step 165820 loss=3.0527 lr=2.48e-05 tok=21.846B dt=9145 wait=257 +15:59:09 step 165840 loss=2.9831 lr=2.48e-05 tok=21.848B dt=9209 wait=257 +16:00:14 step 165860 loss=3.0411 lr=2.47e-05 tok=21.851B dt=9273 wait=258 +16:01:18 step 165880 loss=3.0241 lr=2.47e-05 tok=21.854B dt=9337 wait=258 +16:02:21 step 165900 loss=3.0561 lr=2.47e-05 tok=21.856B dt=9401 wait=258 diff --git a/log_w158.txt b/log_w158.txt new file mode 100644 index 0000000000000000000000000000000000000000..3b8f1f05938e2a94f44afa78d37da3ee56d60670 --- /dev/null +++ b/log_w158.txt @@ -0,0 +1,227 @@ +launch +17:04:29 name=w158 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 105/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:04:35 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:04:35 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:04:45 params 1.08B +17:04:47 resumed step=166500 seen=21.95B round=671 +17:04:47 stream salt=166500 +17:06:34 blocks compiled +17:06:34 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:09:09 mix source 4 paused 30 min ((Request ID: Root=1-6abbf0b5-45eae3c51dcf8224251fc2ad;43e53546-b38c-4ec4-9576-5c) +17:09:46 step 166501 loss=3.1438 lr=2.44e-05 tok=21.955B dt=192 wait=155 +17:10:09 step 166502 loss=3.3013 lr=2.44e-05 tok=21.955B dt=214 wait=155 +17:12:18 global r=672 load 4.291 -> 4.280 +17:12:50 step 166660 loss=3.5468 lr=2.43e-05 tok=21.957B dt=376 wait=155 +17:13:54 step 166680 loss=3.4402 lr=2.43e-05 tok=21.960B dt=440 wait=155 +17:14:58 step 166700 loss=3.4966 lr=2.43e-05 tok=21.962B dt=504 wait=156 +17:16:01 step 166720 loss=3.4101 lr=2.43e-05 tok=21.965B dt=567 wait=156 +17:16:42 global r=673 load 4.279 -> 4.268 +17:17:45 step 166820 loss=3.3683 lr=2.42e-05 tok=21.969B dt=671 wait=156 +17:18:49 step 166840 loss=3.4838 lr=2.42e-05 tok=21.972B dt=735 wait=156 +launch +17:19:49 name=w158 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 105/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:19:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:19:54 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:20:03 params 1.08B +17:20:05 resumed step=166800 seen=21.98B round=673 +17:20:05 stream salt=166800 +17:21:52 blocks compiled +17:21:52 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:22:55 parquet file skipped data/UltraData-Code-L3/go/UltraData-Code-L3-go-part-00010-of-00012.parquet: (Request ID: Root=1-6abbf3ef-6c7df5384acca2ee1a626e2f;fd1eb966-5d47-488c-b75c-2f537d3adc71) + +429 Too Many Requests: you +17:22:55 parquet file skipped data/UltraData-Code-L3/cs/UltraData-Code-L3-cs-part-00002-of-00028.parquet: (Request ID: Root=1-6abbf3ef-162b9abb1b9961cd23049b8a;49d3d99b-c4e1-4b95-9623-3ba636e5788a) + +429 Too Many Requests: you +17:22:59 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00129-of-00167.parquet: (Request ID: Root=1-6abbf3f3-2a376d1847b218cb2ea51e63;10949f6c-b696-40fa-9ab3-00d075f5bba4) + +429 Too Many Requests: you +17:23:01 mix source 6 paused 30 min ((Request ID: Root=1-6abbf3f5-3bb271f417d2b3a273e354f9;5fce20fa-f2f2-4e04-ae18-07) +17:23:44 mix source 1 paused 30 min ((Request ID: Root=1-6abbf420-0eca42585409cf214dd6e07b;dd6f3c0e-051d-42da-9a2b-12) +17:23:44 mix source 5 paused 30 min ((Request ID: Root=1-6abbf420-381e62965ba8517001d4960d;5483cf31-4862-4dc4-8f29-63) +17:23:44 mix source 8 paused 30 min ((Request ID: Root=1-6abbf420-68bffecc084e5132577679fd;54b06926-c935-4073-9905-96) +17:24:04 step 166801 loss=2.9955 lr=2.42e-05 tok=21.976B dt=131 wait=112 +17:24:15 step 166802 loss=3.8055 lr=2.42e-05 tok=21.976B dt=142 wait=112 +17:24:30 dolma file skipped data/ingredient1-common_crawl-high-quality_20_entertainment/shard_00000688.jsonl.zst: (Request ID: Root=1-6abbf44e-68e8810c627438151deb57d6;4922584f-cdc3-4590-a91e-616693c22de3) + +429 Too Many Requests: you +17:24:30 dolma file skipped data/ingredient1-common_crawl-high-quality_19_education_and_jobs/shard_00000716.jsonl.zst: (Request ID: Root=1-6abbf44e-2d1549ba6fe2cfe048b7bdef;86c345b9-265a-4fea-8a77-0a59840efbd8) + +429 Too Many Requests: you +17:24:32 dolma file skipped data/ingredient1-common_crawl-high-quality_20_politics/shard_00000914.jsonl.zst: (Request ID: Root=1-6abbf450-1cafc632543df4d02a0683f2;f857748f-e514-494f-9745-7175b978364d) + +429 Too Many Requests: you +17:24:32 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00057_f317.jsonl.zst: (Request ID: Root=1-6abbf450-15c4c47133a191636bfc82fd;25801364-3f2c-48d6-8042-5659d5ed69fd) + +429 Too Many Requests: you +17:25:17 step 166820 loss=3.6638 lr=2.42e-05 tok=21.978B dt=205 wait=113 +17:25:44 dolma file skipped data/ingredient1-common_crawl-high-quality_19_travel_and_tourism/shard_00000235.jsonl.zst: (Request ID: Root=1-6abbf498-0298b3ed1a89da7628c299be;4eb035a5-48e2-45b8-bcad-74b5da23fd0c) + +429 Too Many Requests: you +17:25:44 dolma file skipped data/ingredient1-stem-heavy-crawl/home_and_hobbies.00000097.jsonl.zst: (Request ID: Root=1-6abbf498-1b7ab2b448db746c7d587206;4f302f6a-a6f2-4653-a2cf-ac210987e785) + +429 Too Many Requests: you +17:25:44 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00036_f122.jsonl.zst: (Request ID: Root=1-6abbf498-0848527c7c8dca4673309339;e09cde01-02f6-4ede-983e-b9b8f48814bb) + +429 Too Many Requests: you +17:25:44 dolma file skipped data/ingredient1-stem-heavy-crawl/home_and_hobbies.00000016.jsonl.zst: (Request ID: Root=1-6abbf498-6494285c63f247d3470edeec;17aec9ef-6a5f-4f66-bc28-d447a34e0603) + +429 Too Many Requests: you +17:25:47 dolma file skipped data/ingredient1-nemotron-synth-qa/CC-MAIN-2021-31-part-00025.jsonl.zst: (Request ID: Root=1-6abbf49b-202134d80aecfc513904690f;bafebbc6-ae57-4f2e-836d-23f9e214cc26) + +429 Too Many Requests: you +17:25:48 dolma file skipped data/ingredient1-common_crawl-high-quality_19_entertainment/shard_00000871.jsonl.zst: (Request ID: Root=1-6abbf49c-1085e8ab77b0757b03b1f4d5;11356c17-f77d-4c58-a192-939dec947b64) + +429 Too Many Requests: you +17:25:48 dolma file skipped data/ingredient1-stem-heavy-crawl/finance_and_business.00000037.jsonl.zst: (Request ID: Root=1-6abbf49c-7766e8ef2af3c26824c85147;7adbef4b-6202-4010-b313-26e2dc93f7b2) + +429 Too Many Requests: you +17:26:23 step 166840 loss=3.4042 lr=2.42e-05 tok=21.981B dt=271 wait=113 +17:27:27 step 166860 loss=3.3407 lr=2.42e-05 tok=21.983B dt=335 wait=114 +17:28:31 step 166880 loss=3.4271 lr=2.42e-05 tok=21.986B dt=399 wait=114 +17:29:35 step 166900 loss=3.4893 lr=2.42e-05 tok=21.989B dt=463 wait=114 +17:30:40 step 166920 loss=3.2938 lr=2.42e-05 tok=21.991B dt=527 wait=114 +17:31:43 step 166940 loss=3.2586 lr=2.42e-05 tok=21.994B dt=591 wait=115 +17:32:30 pushed ckpt_w158.pt,log_w158.txt +17:35:32 waited 181s for global r=674 +17:35:45 global r=674 load 4.272 -> 4.258 +17:36:20 step 166960 loss=3.3390 lr=2.42e-05 tok=21.997B dt=868 wait=115 +17:37:24 step 166980 loss=3.2451 lr=2.41e-05 tok=21.999B dt=932 wait=115 +17:38:27 step 167000 loss=3.2579 lr=2.41e-05 tok=22.002B dt=994 wait=115 +17:39:30 step 167020 loss=3.4104 lr=2.41e-05 tok=22.004B dt=1058 wait=116 +launch +17:40:10 name=w158 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 105/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:40:15 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:40:15 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:40:25 params 1.08B +17:40:26 resumed step=166950 seen=22.00B round=674 +17:40:26 stream salt=166950 +17:42:22 blocks compiled +17:42:22 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:46:18 step 166951 loss=2.6207 lr=2.42e-05 tok=21.995B dt=235 wait=224 +17:46:25 step 166952 loss=2.7498 lr=2.42e-05 tok=21.995B dt=243 wait=224 +17:46:51 step 166960 loss=3.2895 lr=2.42e-05 tok=21.997B dt=269 wait=224 +17:47:37 global r=675 load 4.279 -> 4.260 +17:48:41 step 167120 loss=3.0271 lr=2.41e-05 tok=22.000B dt=379 wait=224 +17:49:46 step 167140 loss=3.0245 lr=2.41e-05 tok=22.003B dt=444 wait=224 +17:50:51 step 167160 loss=3.1273 lr=2.40e-05 tok=22.006B dt=508 wait=224 +17:51:55 step 167180 loss=3.0422 lr=2.40e-05 tok=22.008B dt=573 wait=224 +17:53:00 step 167200 loss=3.0333 lr=2.40e-05 tok=22.011B dt=637 wait=225 +17:54:05 step 167220 loss=3.0965 lr=2.40e-05 tok=22.014B dt=702 wait=225 +17:55:09 step 167240 loss=2.9850 lr=2.40e-05 tok=22.016B dt=767 wait=225 +17:55:55 pushed ckpt_w158.pt,log_w158.txt +17:59:32 step 167260 loss=3.0773 lr=2.40e-05 tok=22.019B dt=1030 wait=225 +17:59:38 global r=676 load 4.296 -> 4.285 +18:00:10 step 167260 loss=3.0899 lr=2.40e-05 tok=22.020B dt=1068 wait=225 +18:01:15 step 167280 loss=3.0543 lr=2.40e-05 tok=22.023B dt=1132 wait=225 +18:02:20 step 167300 loss=2.9985 lr=2.40e-05 tok=22.025B dt=1197 wait=225 +18:03:25 step 167320 loss=3.0269 lr=2.40e-05 tok=22.028B dt=1262 wait=226 +18:04:30 step 167340 loss=2.9988 lr=2.40e-05 tok=22.031B dt=1327 wait=226 +18:05:34 step 167360 loss=3.0631 lr=2.39e-05 tok=22.033B dt=1392 wait=226 +18:06:38 step 167380 loss=3.1521 lr=2.39e-05 tok=22.036B dt=1456 wait=226 +18:07:43 step 167400 loss=3.0846 lr=2.39e-05 tok=22.038B dt=1520 wait=226 +18:07:57 pushed ckpt_w158.pt,log_w158.txt +18:09:12 waited 76s for global r=677 +18:09:22 global r=677 load 4.297 -> 4.280 +18:10:28 step 167420 loss=3.0393 lr=2.39e-05 tok=22.041B dt=1685 wait=226 +18:11:32 step 167440 loss=3.0305 lr=2.39e-05 tok=22.044B dt=1750 wait=226 +18:12:38 step 167460 loss=3.1222 lr=2.39e-05 tok=22.046B dt=1815 wait=227 +18:13:42 step 167480 loss=2.9594 lr=2.39e-05 tok=22.049B dt=1880 wait=227 +18:14:49 step 167500 loss=2.6534 lr=2.39e-05 tok=22.052B dt=1946 wait=227 +18:15:54 step 167520 loss=3.0433 lr=2.39e-05 tok=22.054B dt=2011 wait=227 +18:16:58 step 167540 loss=2.9242 lr=2.38e-05 tok=22.057B dt=2076 wait=227 +18:17:44 pushed ckpt_w158.pt,log_w158.txt +18:18:59 waited 76s for global r=678 +18:19:05 global r=678 load 4.284 -> 4.281 +18:19:39 step 167560 loss=3.0418 lr=2.38e-05 tok=22.059B dt=2236 wait=227 +18:20:45 step 167580 loss=2.9281 lr=2.38e-05 tok=22.062B dt=2302 wait=228 +18:21:50 step 167600 loss=2.9316 lr=2.38e-05 tok=22.065B dt=2368 wait=228 +18:22:55 step 167620 loss=3.0155 lr=2.38e-05 tok=22.067B dt=2433 wait=228 +18:24:01 step 167640 loss=2.8585 lr=2.38e-05 tok=22.070B dt=2499 wait=228 +18:25:06 step 167660 loss=2.9236 lr=2.38e-05 tok=22.073B dt=2563 wait=228 +18:26:10 step 167680 loss=2.9236 lr=2.38e-05 tok=22.075B dt=2628 wait=228 +18:27:15 step 167700 loss=2.9400 lr=2.38e-05 tok=22.078B dt=2693 wait=228 +18:27:27 pushed ckpt_w158.pt,log_w158.txt +18:28:43 waited 76s for global r=679 +18:28:52 global r=679 load 4.297 -> 4.283 +18:29:58 step 167720 loss=2.8469 lr=2.38e-05 tok=22.080B dt=2856 wait=229 +18:31:03 step 167740 loss=2.8952 lr=2.37e-05 tok=22.083B dt=2921 wait=229 +18:32:08 step 167760 loss=2.8970 lr=2.37e-05 tok=22.086B dt=2986 wait=229 +18:33:14 step 167780 loss=2.8552 lr=2.37e-05 tok=22.088B dt=3051 wait=229 +18:34:19 step 167800 loss=2.7595 lr=2.37e-05 tok=22.091B dt=3117 wait=229 +18:35:24 step 167820 loss=2.9298 lr=2.37e-05 tok=22.093B dt=3182 wait=229 +18:36:29 step 167840 loss=2.8044 lr=2.37e-05 tok=22.096B dt=3247 wait=229 +18:37:20 pushed ckpt_w158.pt,log_w158.txt +18:38:35 waited 76s for global r=680 +18:38:45 global r=680 load 4.299 -> 4.282 +18:39:18 step 167860 loss=2.9440 lr=2.37e-05 tok=22.099B dt=3416 wait=230 +18:40:25 step 167880 loss=2.3696 lr=2.37e-05 tok=22.101B dt=3482 wait=230 +18:41:30 step 167900 loss=2.8242 lr=2.37e-05 tok=22.104B dt=3547 wait=230 +18:42:34 step 167920 loss=2.9149 lr=2.36e-05 tok=22.107B dt=3612 wait=230 +18:43:39 step 167940 loss=2.8424 lr=2.36e-05 tok=22.109B dt=3677 wait=230 +18:44:44 step 167960 loss=2.9880 lr=2.36e-05 tok=22.112B dt=3741 wait=230 +18:45:49 step 167980 loss=2.9039 lr=2.36e-05 tok=22.114B dt=3806 wait=230 +18:46:53 step 168000 loss=2.8958 lr=2.36e-05 tok=22.117B dt=3871 wait=231 +18:47:05 pushed ckpt_w158.pt,log_w158.txt +18:48:36 waited 91s for global r=681 +18:48:46 global r=681 load 4.295 -> 4.285 +18:49:53 step 168020 loss=2.8228 lr=2.36e-05 tok=22.120B dt=4050 wait=231 +18:50:58 step 168040 loss=2.8943 lr=2.36e-05 tok=22.122B dt=4115 wait=231 +18:52:02 step 168060 loss=2.8840 lr=2.36e-05 tok=22.125B dt=4180 wait=231 +18:53:06 step 168080 loss=2.9074 lr=2.36e-05 tok=22.128B dt=4244 wait=231 +18:54:11 step 168100 loss=2.8980 lr=2.36e-05 tok=22.130B dt=4308 wait=231 +18:55:14 step 168120 loss=3.0058 lr=2.35e-05 tok=22.133B dt=4372 wait=231 +18:56:18 step 168140 loss=2.7802 lr=2.35e-05 tok=22.135B dt=4436 wait=232 +18:57:03 pushed ckpt_w158.pt,log_w158.txt +18:58:34 waited 91s for global r=682 +18:58:45 global r=682 load 4.285 -> 4.283 +18:59:18 step 168160 loss=3.0898 lr=2.35e-05 tok=22.138B dt=4616 wait=232 +19:00:21 step 168180 loss=2.9833 lr=2.35e-05 tok=22.141B dt=4679 wait=232 +19:01:25 step 168200 loss=2.8902 lr=2.35e-05 tok=22.143B dt=4742 wait=232 +19:02:29 step 168220 loss=3.0387 lr=2.35e-05 tok=22.146B dt=4806 wait=232 +19:03:32 step 168240 loss=2.9040 lr=2.35e-05 tok=22.149B dt=4870 wait=232 +19:04:35 step 168260 loss=3.0348 lr=2.35e-05 tok=22.151B dt=4933 wait=232 +19:05:38 step 168280 loss=2.9794 lr=2.35e-05 tok=22.154B dt=4996 wait=233 +19:06:41 step 168300 loss=3.0047 lr=2.35e-05 tok=22.156B dt=5059 wait=233 +19:06:54 pushed ckpt_w158.pt,log_w158.txt +19:08:25 waited 91s for global r=683 +19:08:33 global r=683 load 4.287 -> 4.281 +19:09:37 step 168320 loss=2.9471 lr=2.34e-05 tok=22.159B dt=5235 wait=233 +19:10:42 step 168340 loss=2.9462 lr=2.34e-05 tok=22.162B dt=5299 wait=233 +19:11:45 step 168360 loss=2.9752 lr=2.34e-05 tok=22.164B dt=5363 wait=233 +19:12:48 step 168380 loss=2.9776 lr=2.34e-05 tok=22.167B dt=5426 wait=233 +19:13:52 step 168400 loss=2.9794 lr=2.34e-05 tok=22.170B dt=5489 wait=233 +19:14:55 step 168420 loss=3.1384 lr=2.34e-05 tok=22.172B dt=5553 wait=233 +19:15:59 step 168440 loss=3.0771 lr=2.34e-05 tok=22.175B dt=5617 wait=234 +19:16:44 pushed ckpt_w158.pt,log_w158.txt +19:18:15 waited 91s for global r=684 +19:18:22 global r=684 load 4.280 -> 4.282 +19:18:55 step 168460 loss=3.1539 lr=2.34e-05 tok=22.177B dt=5793 wait=234 +19:20:00 step 168480 loss=3.0050 lr=2.34e-05 tok=22.180B dt=5857 wait=234 +19:21:04 step 168500 loss=3.0747 lr=2.33e-05 tok=22.183B dt=5922 wait=234 +19:22:08 step 168520 loss=3.1598 lr=2.33e-05 tok=22.185B dt=5986 wait=234 +19:23:13 step 168540 loss=3.0624 lr=2.33e-05 tok=22.188B dt=6050 wait=234 +19:24:17 step 168560 loss=3.1168 lr=2.33e-05 tok=22.190B dt=6114 wait=234 +19:25:21 step 168580 loss=3.0611 lr=2.33e-05 tok=22.193B dt=6178 wait=235 +19:26:24 step 168600 loss=3.1617 lr=2.33e-05 tok=22.196B dt=6242 wait=235 +19:26:36 pushed ckpt_w158.pt,log_w158.txt +19:27:51 waited 76s for global r=685 +19:27:58 global r=685 load 4.279 -> 4.283 +19:29:03 step 168620 loss=3.1724 lr=2.33e-05 tok=22.198B dt=6401 wait=235 +19:30:07 step 168640 loss=3.2134 lr=2.33e-05 tok=22.201B dt=6464 wait=235 +19:31:11 step 168660 loss=3.1666 lr=2.33e-05 tok=22.204B dt=6528 wait=235 +19:32:14 step 168680 loss=3.0103 lr=2.33e-05 tok=22.206B dt=6592 wait=235 +19:33:18 step 168700 loss=3.0063 lr=2.32e-05 tok=22.209B dt=6656 wait=235 +19:34:22 step 168720 loss=3.1015 lr=2.32e-05 tok=22.211B dt=6719 wait=236 +19:35:24 step 168740 loss=3.0189 lr=2.32e-05 tok=22.214B dt=6782 wait=236 +19:36:08 pushed ckpt_w158.pt,log_w158.txt +19:37:39 waited 91s for global r=686 +19:37:50 global r=686 load 4.288 -> 4.286 +19:38:23 step 168760 loss=3.0897 lr=2.32e-05 tok=22.217B dt=6961 wait=236 +19:39:27 step 168780 loss=2.9328 lr=2.32e-05 tok=22.219B dt=7025 wait=236 +19:40:31 step 168800 loss=2.8976 lr=2.32e-05 tok=22.222B dt=7089 wait=236 +19:41:35 step 168820 loss=3.0203 lr=2.32e-05 tok=22.225B dt=7152 wait=236 +19:42:38 step 168840 loss=2.9380 lr=2.32e-05 tok=22.227B dt=7216 wait=236 +19:43:42 step 168860 loss=2.9999 lr=2.32e-05 tok=22.230B dt=7280 wait=237 +19:44:45 step 168880 loss=2.9644 lr=2.32e-05 tok=22.232B dt=7343 wait=237 +19:45:49 step 168900 loss=2.9869 lr=2.31e-05 tok=22.235B dt=7406 wait=237 diff --git a/log_w159.txt b/log_w159.txt new file mode 100644 index 0000000000000000000000000000000000000000..85c3fa990e82eb3fb161ca6ab7776bc388036900 --- /dev/null +++ b/log_w159.txt @@ -0,0 +1,278 @@ +launch +17:04:29 name=w159 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 106/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:04:35 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:04:35 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:04:46 params 1.08B +17:04:48 resumed step=166500 seen=21.95B round=671 +17:04:48 stream salt=166500 +17:06:35 blocks compiled +17:06:35 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:09:32 mix source 3 paused 30 min ((Request ID: Root=1-6abbf0cc-2d4ee01916ec6c46351cb5a8;7b2983db-392c-4498-a6eb-5a) +17:09:32 mix source 6 paused 30 min ((Request ID: Root=1-6abbf0cc-1338901c5aa2e70b2ee04391;655cb7eb-486e-4c56-b03c-0e) +17:09:32 mix source 4 paused 30 min ((Request ID: Root=1-6abbf0cc-43c0c8df2aefa6346f02d74f;8c67b219-1d0d-411e-953c-0a) +17:10:03 step 166501 loss=3.1360 lr=2.44e-05 tok=21.955B dt=208 wait=177 +17:10:19 step 166502 loss=3.3573 lr=2.44e-05 tok=21.955B dt=224 wait=177 +17:12:01 global r=672 load 4.278 -> 4.280 +17:12:33 step 166660 loss=3.6499 lr=2.43e-05 tok=21.957B dt=358 wait=177 +17:13:37 step 166680 loss=3.5028 lr=2.43e-05 tok=21.960B dt=422 wait=178 +17:14:42 step 166700 loss=3.4584 lr=2.43e-05 tok=21.962B dt=487 wait=178 +17:15:46 step 166720 loss=3.6378 lr=2.43e-05 tok=21.965B dt=551 wait=178 +17:16:50 step 166740 loss=3.5884 lr=2.43e-05 tok=21.968B dt=615 wait=178 +17:16:58 global r=673 load 4.272 -> 4.268 +17:18:02 step 166820 loss=3.4886 lr=2.42e-05 tok=21.970B dt=687 wait=179 +17:19:05 step 166840 loss=3.6009 lr=2.42e-05 tok=21.973B dt=750 wait=179 +launch +17:19:50 name=w159 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 106/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:19:55 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:19:55 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:20:04 params 1.08B +17:20:06 resumed step=166800 seen=21.98B round=673 +17:20:06 stream salt=166800 +17:21:58 blocks compiled +17:21:58 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:23:02 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00060-of-00147.parquet: (Request ID: Root=1-6abbf3f6-51ba687d12bc27922f6189aa;20c0ea8b-f1f7-47c5-a582-3a3762fed6dd) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00083-of-00167.parquet: (Request ID: Root=1-6abbf3f6-2de0fbbd0ef545d86f743054;9e916b19-3832-4720-b827-758dd38ecb52) + +429 Too Many Requests: you +17:23:02 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00040-of-00137.parquet: (Request ID: Root=1-6abbf3f6-337fb11f070e1f876f47e1b8;6bbd2d2d-6635-4bea-82cc-c088c75be6fa) + +429 Too Many Requests: you +17:23:21 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00078-of-00137.parquet: (Request ID: Root=1-6abbf409-3245548611817a2e0ed181e5;171d4bd0-4e2b-4560-afb6-ca4fa77f6188) + +429 Too Many Requests: you +17:23:21 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00050-of-00167.parquet: (Request ID: Root=1-6abbf409-3cab9bea246b99b5578bd863;70d4690f-47c6-4595-9898-b922ab46b1e9) + +429 Too Many Requests: you +17:23:21 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00090-of-00137.parquet: (Request ID: Root=1-6abbf409-2cdb185b0fd06e78058a42d6;6cd92fa4-d160-4bf3-a83a-3b83777961a6) + +429 Too Many Requests: you +17:23:21 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00089-of-00167.parquet: (Request ID: Root=1-6abbf409-2526112c13f5a2235ce51644;000304ac-cb91-4da9-a1e4-309be77c5bbf) + +429 Too Many Requests: you +17:23:21 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00049-of-00147.parquet: (Request ID: Root=1-6abbf409-3b1bcc4f3c6cf2f50f88cd39;576c651c-5c33-4fbc-836c-1d430bb897b0) + +429 Too Many Requests: you +17:23:21 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00144-of-00147.parquet: (Request ID: Root=1-6abbf409-00dc9dc9192c45823ff48db0;6df4faa1-3c03-47b5-a4d1-fc082e88db27) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00030-of-00147.parquet: (Request ID: Root=1-6abbf418-6bc3f13c751ff07b79193aad;a8c6a509-f01e-472b-b3f0-9ae348228258) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00025-of-00147.parquet: (Request ID: Root=1-6abbf418-670eab477b4b43f245dfa790;7559e39f-42cc-4380-baf8-798f6b4b32c2) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00074-of-00137.parquet: (Request ID: Root=1-6abbf418-457922b57627c3ef47d8d30d;80ab1da0-231d-41df-85c4-58ae21baff52) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00077-of-00167.parquet: (Request ID: Root=1-6abbf418-7128d7307d7273c307205283;a442f59e-146a-4309-8091-cd375c4ea462) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00080-of-00137.parquet: (Request ID: Root=1-6abbf418-5017c01b59a6174e433e0701;456b6de9-c487-4074-875c-e277a3af0005) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00145-of-00167.parquet: (Request ID: Root=1-6abbf418-6a67c54416fb1dfa680faf99;9b366a52-f983-4c09-9bab-d496f6386965) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00068-of-00137.parquet: (Request ID: Root=1-6abbf418-366693ae0d7f81ac7947b32d;fcdc5e37-2709-43e9-b63d-eef79f8fda67) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00059-of-00147.parquet: (Request ID: Root=1-6abbf418-1c567cdf3c4e125116c79b70;033a3d8f-6bfe-4b0b-991f-31ffdb931957) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00065-of-00147.parquet: (Request ID: Root=1-6abbf418-609410f74caae19b3fb744e2;9e9e2f14-69e2-4341-81f1-77b503a0fa7f) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00086-of-00137.parquet: (Request ID: Root=1-6abbf418-47670c824e9530653e009d15;14943452-d9a8-4959-a5ff-3915aa358721) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00079-of-00137.parquet: (Request ID: Root=1-6abbf418-18015baa31901d4654fdbcf9;63dafde4-2ff7-4e17-9510-603dbca7c768) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00128-of-00167.parquet: (Request ID: Root=1-6abbf418-33ade8041a45e3f67cfa2cb7;5fd6decb-ac3b-4f88-b427-f2f9047c80f5) + +429 Too Many Requests: you +17:23:36 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00143-of-00147.parquet: (Request ID: Root=1-6abbf418-0a98cf3d283fbae00a186b33;d654ad0d-71e1-47b1-997e-5c8cebdf0069) + +429 Too Many Requests: you +17:23:37 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00003-of-00167.parquet: (Request ID: Root=1-6abbf418-189405f33c65c86e188c5e9b;0c179ac4-4b83-4b6c-b0c4-9e0ddf4f0793) + +429 Too Many Requests: you +17:23:37 parquet file skipped data/UltraData-Code-L3/py/UltraData-Code-L3-py-part-00097-of-00147.parquet: (Request ID: Root=1-6abbf419-06edf06f638084e27fb7e4b2;e4e14ca8-6145-4622-87f6-c314229888ba) + +429 Too Many Requests: you +17:23:37 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00011-of-00137.parquet: (Request ID: Root=1-6abbf419-6bb591bd76db0ce43e40686d;581932d7-7f7f-4a73-8d8f-ad54d5014688) + +429 Too Many Requests: you +17:23:37 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00024-of-00167.parquet: (Request ID: Root=1-6abbf419-6c36c96f1425340470c38303;062bfb3d-2a25-4e28-ba78-585b7a981d71) + +429 Too Many Requests: you +17:23:37 parquet file skipped data/UltraData-Code-L3/cpp/UltraData-Code-L3-cpp-part-00036-of-00167.parquet: (Request ID: Root=1-6abbf419-7bc153a73112cb79791d1fcf;fbfa4151-512f-4f8e-9335-d46175426ed2) + +429 Too Many Requests: you +17:23:37 parquet file skipped data/UltraData-Code-L3/java/UltraData-Code-L3-java-part-00127-of-00137.parquet: (Request ID: Root=1-6abbf419-520cb2784acaf407373ed85c;05156320-5b65-44b9-8e4b-81842735d89b) + +429 Too Many Requests: you +17:23:52 mix source 2 paused 30 min ((Request ID: Root=1-6abbf428-0c345ca32cd948e0314d4971;4d18bc4b-f1e6-4924-9ab0-52) +17:23:52 mix source 0 paused 30 min ((Request ID: Root=1-6abbf428-14566173293332f024b17212;24245eb6-1585-48bd-8bcd-a9) +17:23:52 mix source 5 paused 30 min ((Request ID: Root=1-6abbf428-316cd20315c6c1122d8119f6;48802faa-92a2-445c-b706-ae) +17:23:53 mix source 8 paused 30 min ((Request ID: Root=1-6abbf429-66fb436873e1d0ed7b2ead98;5134e956-d11a-4bf0-905a-27) +17:23:53 mix source 1 paused 30 min ((Request ID: Root=1-6abbf429-4434ebe363ed4e621b4ac24d;516b1119-c709-4d02-b743-a4) +17:23:53 mix source 3 paused 30 min ((Request ID: Root=1-6abbf429-47eaf800082f96eb2d09fc83;4e14de86-5ca9-4ef2-8298-90) +17:23:53 mix source 7 paused 30 min ((Request ID: Root=1-6abbf429-590c9d49441692b531c2ab24;045221e9-53f9-4e50-be01-34) +17:23:53 mix source 6 paused 30 min ((Request ID: Root=1-6abbf429-079fa5ca329ccfe0181c2c99;ecc0a31e-f752-439b-9154-ea) +17:24:11 step 166801 loss=3.0486 lr=2.42e-05 tok=21.976B dt=134 wait=115 +17:24:23 step 166802 loss=3.1661 lr=2.42e-05 tok=21.976B dt=146 wait=115 +17:25:20 step 166820 loss=3.3283 lr=2.42e-05 tok=21.978B dt=203 wait=116 +17:26:23 step 166840 loss=3.3050 lr=2.42e-05 tok=21.981B dt=266 wait=116 +17:27:26 step 166860 loss=3.3409 lr=2.42e-05 tok=21.983B dt=328 wait=117 +17:28:28 step 166880 loss=3.3659 lr=2.42e-05 tok=21.986B dt=390 wait=117 +17:29:30 step 166900 loss=3.3992 lr=2.42e-05 tok=21.989B dt=453 wait=118 +17:30:33 step 166920 loss=3.2842 lr=2.42e-05 tok=21.991B dt=515 wait=118 +17:31:35 step 166940 loss=3.2441 lr=2.42e-05 tok=21.994B dt=578 wait=119 +17:32:19 pushed ckpt_w159.pt,log_w159.txt +17:35:55 step 166960 loss=3.3731 lr=2.42e-05 tok=21.997B dt=838 wait=119 +17:36:08 global r=674 load 4.245 -> 4.258 +17:36:39 step 166960 loss=3.3359 lr=2.42e-05 tok=21.998B dt=881 wait=119 +17:37:41 step 166980 loss=3.2341 lr=2.41e-05 tok=22.000B dt=944 wait=120 +17:38:43 step 167000 loss=3.2362 lr=2.41e-05 tok=22.003B dt=1006 wait=120 +17:39:46 step 167020 loss=3.2964 lr=2.41e-05 tok=22.006B dt=1068 wait=121 +17:40:49 step 167040 loss=3.2281 lr=2.41e-05 tok=22.008B dt=1131 wait=121 +17:41:51 step 167060 loss=3.4011 lr=2.41e-05 tok=22.011B dt=1194 wait=122 +launch +17:42:58 name=w159 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 106/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:43:03 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:43:03 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:43:13 params 1.08B +17:43:15 resumed step=166950 seen=22.00B round=674 +17:43:15 stream salt=166950 +17:45:03 blocks compiled +17:45:03 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:48:43 step 166951 loss=2.8946 lr=2.42e-05 tok=21.995B dt=220 wait=209 +17:48:51 step 166952 loss=3.0315 lr=2.42e-05 tok=21.995B dt=227 wait=209 +17:49:16 step 166960 loss=3.2452 lr=2.42e-05 tok=21.997B dt=253 wait=209 +17:49:27 global r=675 load 4.263 -> 4.260 +17:50:32 step 167120 loss=3.1531 lr=2.41e-05 tok=21.999B dt=329 wait=209 +17:51:37 step 167140 loss=3.2545 lr=2.41e-05 tok=22.002B dt=393 wait=209 +17:52:40 step 167160 loss=3.2584 lr=2.40e-05 tok=22.004B dt=457 wait=209 +17:53:44 step 167180 loss=3.2114 lr=2.40e-05 tok=22.007B dt=521 wait=209 +17:54:48 step 167200 loss=3.2262 lr=2.40e-05 tok=22.010B dt=585 wait=210 +17:55:53 step 167220 loss=3.2025 lr=2.40e-05 tok=22.012B dt=650 wait=210 +17:56:57 step 167240 loss=3.1381 lr=2.40e-05 tok=22.015B dt=714 wait=210 +17:57:41 pushed ckpt_w159.pt,log_w159.txt +17:59:27 waited 106s for global r=676 +17:59:35 global r=676 load 4.277 -> 4.285 +18:00:09 step 167260 loss=3.1717 lr=2.40e-05 tok=22.017B dt=906 wait=210 +18:01:14 step 167280 loss=2.6506 lr=2.40e-05 tok=22.020B dt=971 wait=210 +18:02:18 step 167300 loss=3.0885 lr=2.40e-05 tok=22.023B dt=1035 wait=210 +18:03:23 step 167320 loss=3.1804 lr=2.40e-05 tok=22.025B dt=1099 wait=211 +18:04:27 step 167340 loss=3.1541 lr=2.40e-05 tok=22.028B dt=1163 wait=211 +18:05:31 step 167360 loss=3.1827 lr=2.39e-05 tok=22.031B dt=1228 wait=211 +18:06:35 step 167380 loss=3.1691 lr=2.39e-05 tok=22.033B dt=1292 wait=211 +18:07:39 step 167400 loss=3.2865 lr=2.39e-05 tok=22.036B dt=1356 wait=211 +18:07:50 pushed ckpt_w159.pt,log_w159.txt +18:09:21 waited 91s for global r=677 +18:09:28 global r=677 load 4.279 -> 4.280 +18:10:33 step 167420 loss=3.1205 lr=2.39e-05 tok=22.038B dt=1530 wait=211 +18:11:37 step 167440 loss=3.0988 lr=2.39e-05 tok=22.041B dt=1594 wait=212 +18:12:42 step 167460 loss=3.2889 lr=2.39e-05 tok=22.044B dt=1659 wait=212 +18:13:46 step 167480 loss=3.1519 lr=2.39e-05 tok=22.046B dt=1722 wait=212 +18:14:49 step 167500 loss=3.0811 lr=2.39e-05 tok=22.049B dt=1786 wait=212 +18:15:54 step 167520 loss=3.1782 lr=2.39e-05 tok=22.052B dt=1851 wait=212 +18:16:58 step 167540 loss=3.0196 lr=2.38e-05 tok=22.054B dt=1915 wait=212 +18:17:42 pushed ckpt_w159.pt,log_w159.txt +18:18:58 waited 76s for global r=678 +18:19:04 global r=678 load 4.280 -> 4.281 +18:19:37 step 167560 loss=3.2002 lr=2.38e-05 tok=22.057B dt=2074 wait=213 +18:20:41 step 167580 loss=3.0313 lr=2.38e-05 tok=22.059B dt=2138 wait=213 +18:21:45 step 167600 loss=3.0653 lr=2.38e-05 tok=22.062B dt=2202 wait=213 +18:22:50 step 167620 loss=3.1307 lr=2.38e-05 tok=22.065B dt=2267 wait=213 +18:23:54 step 167640 loss=2.8978 lr=2.38e-05 tok=22.067B dt=2331 wait=213 +18:24:59 step 167660 loss=3.1642 lr=2.38e-05 tok=22.070B dt=2395 wait=213 +18:26:03 step 167680 loss=3.0860 lr=2.38e-05 tok=22.073B dt=2460 wait=214 +18:27:07 step 167700 loss=3.1396 lr=2.38e-05 tok=22.075B dt=2523 wait=214 +18:27:19 pushed ckpt_w159.pt,log_w159.txt +18:28:49 waited 91s for global r=679 +18:28:56 global r=679 load 4.285 -> 4.283 +18:30:02 step 167720 loss=2.9968 lr=2.38e-05 tok=22.078B dt=2699 wait=214 +18:31:06 step 167740 loss=3.0132 lr=2.37e-05 tok=22.080B dt=2763 wait=214 +18:32:10 step 167760 loss=2.9882 lr=2.37e-05 tok=22.083B dt=2827 wait=214 +18:33:14 step 167780 loss=3.0841 lr=2.37e-05 tok=22.086B dt=2891 wait=214 +18:34:18 step 167800 loss=3.0182 lr=2.37e-05 tok=22.088B dt=2955 wait=214 +18:35:23 step 167820 loss=3.0143 lr=2.37e-05 tok=22.091B dt=3020 wait=215 +18:36:27 step 167840 loss=2.9884 lr=2.37e-05 tok=22.093B dt=3083 wait=215 +18:37:12 pushed ckpt_w159.pt,log_w159.txt +18:38:43 waited 91s for global r=680 +18:38:48 global r=680 load 4.283 -> 4.282 +18:39:21 step 167860 loss=3.1006 lr=2.37e-05 tok=22.096B dt=3258 wait=215 +18:40:25 step 167880 loss=2.9186 lr=2.37e-05 tok=22.099B dt=3322 wait=215 +18:41:30 step 167900 loss=2.9643 lr=2.37e-05 tok=22.101B dt=3387 wait=215 +18:42:35 step 167920 loss=3.0649 lr=2.36e-05 tok=22.104B dt=3452 wait=215 +18:43:39 step 167940 loss=2.9613 lr=2.36e-05 tok=22.107B dt=3516 wait=216 +18:44:43 step 167960 loss=3.1239 lr=2.36e-05 tok=22.109B dt=3580 wait=216 +18:45:47 step 167980 loss=2.9483 lr=2.36e-05 tok=22.112B dt=3644 wait=216 +18:46:52 step 168000 loss=2.9063 lr=2.36e-05 tok=22.114B dt=3708 wait=216 +18:47:08 pushed ckpt_w159.pt,log_w159.txt +18:48:38 waited 91s for global r=681 +18:48:43 global r=681 load 4.285 -> 4.285 +18:49:48 step 168020 loss=2.8917 lr=2.36e-05 tok=22.117B dt=3885 wait=216 +18:50:53 step 168040 loss=2.8146 lr=2.36e-05 tok=22.120B dt=3949 wait=216 +18:51:57 step 168060 loss=2.8737 lr=2.36e-05 tok=22.122B dt=4014 wait=216 +18:53:01 step 168080 loss=2.8917 lr=2.36e-05 tok=22.125B dt=4078 wait=217 +18:54:05 step 168100 loss=2.9201 lr=2.36e-05 tok=22.128B dt=4141 wait=217 +18:55:09 step 168120 loss=2.9175 lr=2.35e-05 tok=22.130B dt=4206 wait=217 +18:56:14 step 168140 loss=2.9253 lr=2.35e-05 tok=22.133B dt=4270 wait=217 +18:57:00 pushed ckpt_w159.pt,log_w159.txt +18:58:30 waited 91s for global r=682 +18:58:40 global r=682 load 4.288 -> 4.283 +18:59:14 step 168160 loss=2.7191 lr=2.35e-05 tok=22.135B dt=4451 wait=217 +19:00:18 step 168180 loss=2.8907 lr=2.35e-05 tok=22.138B dt=4515 wait=217 +19:01:22 step 168200 loss=2.8554 lr=2.35e-05 tok=22.141B dt=4578 wait=218 +19:02:26 step 168220 loss=2.9439 lr=2.35e-05 tok=22.143B dt=4643 wait=218 +19:03:31 step 168240 loss=2.8496 lr=2.35e-05 tok=22.146B dt=4708 wait=218 +19:04:35 step 168260 loss=3.0171 lr=2.35e-05 tok=22.149B dt=4772 wait=218 +19:05:39 step 168280 loss=2.9033 lr=2.35e-05 tok=22.151B dt=4836 wait=218 +19:06:44 step 168300 loss=2.8889 lr=2.35e-05 tok=22.154B dt=4901 wait=218 +19:06:55 pushed ckpt_w159.pt,log_w159.txt +19:08:26 waited 91s for global r=683 +19:08:32 global r=683 load 4.283 -> 4.281 +19:09:37 step 168320 loss=2.9212 lr=2.34e-05 tok=22.156B dt=5074 wait=219 +19:10:41 step 168340 loss=2.9607 lr=2.34e-05 tok=22.159B dt=5138 wait=219 +19:11:46 step 168360 loss=2.8520 lr=2.34e-05 tok=22.162B dt=5203 wait=219 +19:12:51 step 168380 loss=2.7796 lr=2.34e-05 tok=22.164B dt=5267 wait=219 +19:13:55 step 168400 loss=2.5447 lr=2.34e-05 tok=22.167B dt=5332 wait=219 +19:15:00 step 168420 loss=2.8735 lr=2.34e-05 tok=22.170B dt=5396 wait=219 +19:16:03 step 168440 loss=2.8093 lr=2.34e-05 tok=22.172B dt=5460 wait=220 +19:16:49 pushed ckpt_w159.pt,log_w159.txt +19:18:19 waited 91s for global r=684 +19:18:25 global r=684 load 4.289 -> 4.282 +19:18:59 step 168460 loss=2.9051 lr=2.34e-05 tok=22.175B dt=5635 wait=220 +19:20:03 step 168480 loss=2.8167 lr=2.34e-05 tok=22.177B dt=5700 wait=220 +19:21:07 step 168500 loss=2.8819 lr=2.33e-05 tok=22.180B dt=5764 wait=220 +19:22:12 step 168520 loss=2.9061 lr=2.33e-05 tok=22.183B dt=5829 wait=220 +19:23:16 step 168540 loss=2.7440 lr=2.33e-05 tok=22.185B dt=5893 wait=220 +19:24:20 step 168560 loss=2.9491 lr=2.33e-05 tok=22.188B dt=5957 wait=221 +19:25:24 step 168580 loss=2.8867 lr=2.33e-05 tok=22.190B dt=6021 wait=221 +19:26:28 step 168600 loss=2.8432 lr=2.33e-05 tok=22.193B dt=6085 wait=221 +19:26:40 pushed ckpt_w159.pt,log_w159.txt +19:27:55 waited 76s for global r=685 +19:28:01 global r=685 load 4.287 -> 4.283 +19:29:06 step 168620 loss=3.0027 lr=2.33e-05 tok=22.196B dt=6243 wait=221 +19:30:10 step 168640 loss=2.8814 lr=2.33e-05 tok=22.198B dt=6307 wait=221 +19:31:14 step 168660 loss=3.0184 lr=2.33e-05 tok=22.201B dt=6371 wait=221 +19:32:19 step 168680 loss=2.8795 lr=2.33e-05 tok=22.204B dt=6435 wait=221 +19:33:23 step 168700 loss=2.8547 lr=2.32e-05 tok=22.206B dt=6500 wait=222 +19:34:28 step 168720 loss=2.9308 lr=2.32e-05 tok=22.209B dt=6564 wait=222 +19:35:32 step 168740 loss=2.8528 lr=2.32e-05 tok=22.211B dt=6629 wait=222 +19:36:16 pushed ckpt_w159.pt,log_w159.txt +19:37:47 waited 91s for global r=686 +19:37:52 global r=686 load 4.291 -> 4.286 +19:38:25 step 168760 loss=2.9785 lr=2.32e-05 tok=22.214B dt=6802 wait=222 +19:39:29 step 168780 loss=2.9519 lr=2.32e-05 tok=22.217B dt=6866 wait=222 +19:40:34 step 168800 loss=2.8308 lr=2.32e-05 tok=22.219B dt=6931 wait=222 +19:41:39 step 168820 loss=2.9214 lr=2.32e-05 tok=22.222B dt=6996 wait=223 +19:42:44 step 168840 loss=2.8855 lr=2.32e-05 tok=22.225B dt=7061 wait=223 +19:43:49 step 168860 loss=2.8692 lr=2.32e-05 tok=22.227B dt=7125 wait=223 +19:44:54 step 168880 loss=2.8757 lr=2.32e-05 tok=22.230B dt=7191 wait=223 +19:46:00 step 168900 loss=2.4673 lr=2.31e-05 tok=22.232B dt=7257 wait=223 diff --git a/log_w160.txt b/log_w160.txt new file mode 100644 index 0000000000000000000000000000000000000000..2eaf86a9f86e116f3cba421e6634685259c96321 --- /dev/null +++ b/log_w160.txt @@ -0,0 +1,185 @@ +launch +17:04:34 name=w160 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 107/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:04:40 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:04:40 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:04:50 params 1.08B +17:04:51 resumed step=166500 seen=21.95B round=671 +17:04:51 stream salt=166500 +17:06:36 blocks compiled +17:06:36 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:14:03 step 166501 loss=2.9966 lr=2.44e-05 tok=21.955B dt=446 wait=417 +17:14:19 step 166502 loss=3.1042 lr=2.44e-05 tok=21.955B dt=462 wait=417 +17:15:01 global r=672 load 4.290 -> 4.280 +17:15:33 step 166660 loss=3.4761 lr=2.43e-05 tok=21.957B dt=537 wait=417 +17:16:37 step 166680 loss=3.2732 lr=2.43e-05 tok=21.960B dt=600 wait=418 +17:16:45 global r=673 load 4.278 -> 4.268 +17:17:49 step 166820 loss=3.2063 lr=2.42e-05 tok=21.962B dt=673 wait=418 +17:18:52 step 166840 loss=3.2895 lr=2.42e-05 tok=21.965B dt=736 wait=418 +launch +17:19:51 name=w160 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 107/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:19:56 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:19:56 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:20:06 params 1.08B +17:20:07 resumed step=166800 seen=21.98B round=673 +17:20:07 stream salt=166800 +17:21:54 blocks compiled +17:21:54 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:24:27 mix source 4 paused 30 min ((Request ID: Root=1-6abbf44b-581300462fa54df84aeaec33;32d69fa5-bb4c-4f44-9459-12) +17:24:27 mix source 0 paused 30 min ((Request ID: Root=1-6abbf44b-39e8532f3e3bc1ff28d5763f;1dc5980a-2da1-4f97-833c-e9) +17:24:27 mix source 5 paused 30 min ((Request ID: Root=1-6abbf44b-323b2fb278f9d9a313f0955b;d07551df-37ad-44cd-a52f-57) +17:24:28 mix source 7 paused 30 min (hf://datasets/OpenCoder-LLM/opc-annealing-corpus@078e4bc1d39725cccc18514d2e3278a) +17:24:29 mix source 8 paused 30 min ((Request ID: Root=1-6abbf44d-61d6a5651e63c6653266c6eb;e314cd3c-b318-451f-83a3-92) +17:24:43 step 166801 loss=2.9681 lr=2.42e-05 tok=21.976B dt=169 wait=154 +17:24:54 step 166802 loss=3.0922 lr=2.42e-05 tok=21.976B dt=180 wait=154 +17:25:55 step 166820 loss=3.3475 lr=2.42e-05 tok=21.978B dt=241 wait=155 +17:27:01 step 166840 loss=3.3014 lr=2.42e-05 tok=21.981B dt=307 wait=155 +17:28:05 step 166860 loss=3.3277 lr=2.42e-05 tok=21.983B dt=371 wait=155 +17:29:08 step 166880 loss=3.3646 lr=2.42e-05 tok=21.986B dt=434 wait=156 +17:30:12 step 166900 loss=3.4564 lr=2.42e-05 tok=21.989B dt=498 wait=156 +17:31:16 step 166920 loss=3.2738 lr=2.42e-05 tok=21.991B dt=562 wait=156 +17:32:19 step 166940 loss=3.3335 lr=2.42e-05 tok=21.994B dt=625 wait=156 +17:33:05 pushed ckpt_w160.pt,log_w160.txt +17:35:36 waited 151s for global r=674 +17:35:48 global r=674 load 4.275 -> 4.258 +17:36:23 step 166960 loss=3.4780 lr=2.42e-05 tok=21.997B dt=869 wait=157 +17:37:27 step 166980 loss=3.4787 lr=2.41e-05 tok=21.999B dt=933 wait=157 +17:38:30 step 167000 loss=3.3321 lr=2.41e-05 tok=22.002B dt=996 wait=157 +17:39:34 step 167020 loss=3.3596 lr=2.41e-05 tok=22.004B dt=1060 wait=158 +17:40:38 step 167040 loss=3.3137 lr=2.41e-05 tok=22.007B dt=1124 wait=158 +17:41:41 step 167060 loss=3.3883 lr=2.41e-05 tok=22.010B dt=1187 wait=158 +17:42:45 step 167080 loss=3.2530 lr=2.41e-05 tok=22.012B dt=1251 wait=158 +17:43:48 step 167100 loss=3.3302 lr=2.41e-05 tok=22.015B dt=1314 wait=159 +17:44:00 pushed ckpt_w160.pt,log_w160.txt +launch +17:46:02 name=w160 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 107/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:46:07 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:46:07 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:46:16 params 1.08B +17:46:18 resumed step=166950 seen=22.00B round=674 +17:46:18 stream salt=166950 +17:48:04 blocks compiled +17:48:04 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:51:50 step 166951 loss=3.0572 lr=2.42e-05 tok=21.995B dt=226 wait=215 +17:51:58 step 166952 loss=3.0185 lr=2.42e-05 tok=21.995B dt=234 wait=215 +17:52:23 step 166960 loss=3.2836 lr=2.42e-05 tok=21.997B dt=259 wait=215 +17:52:35 global r=675 load 4.279 -> 4.260 +17:53:41 step 167120 loss=3.1454 lr=2.41e-05 tok=21.999B dt=337 wait=216 +17:54:47 step 167140 loss=3.2125 lr=2.41e-05 tok=22.002B dt=403 wait=216 +17:55:52 step 167160 loss=3.1576 lr=2.40e-05 tok=22.004B dt=468 wait=216 +17:56:57 step 167180 loss=3.2129 lr=2.40e-05 tok=22.007B dt=533 wait=216 +17:58:02 step 167200 loss=3.2104 lr=2.40e-05 tok=22.010B dt=598 wait=216 +17:59:07 step 167220 loss=3.2871 lr=2.40e-05 tok=22.012B dt=663 wait=216 +17:59:46 global r=676 load 4.286 -> 4.285 +18:00:18 step 167260 loss=3.1488 lr=2.40e-05 tok=22.015B dt=734 wait=217 +18:01:23 step 167280 loss=3.1121 lr=2.40e-05 tok=22.017B dt=799 wait=217 +18:02:28 step 167300 loss=3.1300 lr=2.40e-05 tok=22.020B dt=864 wait=217 +18:03:33 step 167320 loss=3.0307 lr=2.40e-05 tok=22.023B dt=929 wait=217 +18:04:38 step 167340 loss=3.2292 lr=2.40e-05 tok=22.025B dt=994 wait=217 +18:05:44 step 167360 loss=3.2851 lr=2.39e-05 tok=22.028B dt=1060 wait=217 +18:06:49 step 167380 loss=3.1455 lr=2.39e-05 tok=22.031B dt=1125 wait=218 +18:07:54 step 167400 loss=3.1956 lr=2.39e-05 tok=22.033B dt=1190 wait=218 +18:08:10 pushed ckpt_w160.pt,log_w160.txt +18:09:10 waited 61s for global r=677 +18:09:16 global r=677 load 4.274 -> 4.280 +18:10:23 step 167420 loss=3.1411 lr=2.39e-05 tok=22.036B dt=1339 wait=218 +18:11:28 step 167440 loss=3.1986 lr=2.39e-05 tok=22.038B dt=1404 wait=218 +18:12:33 step 167460 loss=3.1680 lr=2.39e-05 tok=22.041B dt=1469 wait=218 +18:13:39 step 167480 loss=3.0237 lr=2.39e-05 tok=22.044B dt=1535 wait=218 +18:14:43 step 167500 loss=3.1149 lr=2.39e-05 tok=22.046B dt=1599 wait=218 +18:15:48 step 167520 loss=3.2005 lr=2.39e-05 tok=22.049B dt=1664 wait=218 +18:16:53 step 167540 loss=3.0394 lr=2.38e-05 tok=22.052B dt=1729 wait=219 +18:17:37 pushed ckpt_w160.pt,log_w160.txt +18:19:08 waited 91s for global r=678 +18:19:16 global r=678 load 4.278 -> 4.281 +18:19:50 step 167560 loss=3.1173 lr=2.38e-05 tok=22.054B dt=1906 wait=219 +18:20:54 step 167580 loss=3.1052 lr=2.38e-05 tok=22.057B dt=1970 wait=219 +18:21:59 step 167600 loss=3.0516 lr=2.38e-05 tok=22.059B dt=2035 wait=219 +18:23:04 step 167620 loss=3.1309 lr=2.38e-05 tok=22.062B dt=2100 wait=219 +18:24:10 step 167640 loss=3.0309 lr=2.38e-05 tok=22.065B dt=2166 wait=219 +18:25:15 step 167660 loss=3.1471 lr=2.38e-05 tok=22.067B dt=2231 wait=220 +18:26:20 step 167680 loss=3.0317 lr=2.38e-05 tok=22.070B dt=2296 wait=220 +18:27:25 step 167700 loss=3.0283 lr=2.38e-05 tok=22.073B dt=2361 wait=220 +18:27:38 pushed ckpt_w160.pt,log_w160.txt +18:28:54 waited 76s for global r=679 +18:29:01 global r=679 load 4.280 -> 4.283 +18:30:07 step 167720 loss=3.1782 lr=2.38e-05 tok=22.075B dt=2523 wait=220 +18:31:12 step 167740 loss=3.1301 lr=2.37e-05 tok=22.078B dt=2588 wait=220 +18:32:18 step 167760 loss=3.0209 lr=2.37e-05 tok=22.080B dt=2654 wait=220 +18:33:23 step 167780 loss=3.0302 lr=2.37e-05 tok=22.083B dt=2719 wait=220 +18:34:28 step 167800 loss=3.1195 lr=2.37e-05 tok=22.086B dt=2784 wait=220 +18:35:34 step 167820 loss=3.1641 lr=2.37e-05 tok=22.088B dt=2850 wait=221 +18:36:40 step 167840 loss=3.0370 lr=2.37e-05 tok=22.091B dt=2916 wait=221 +18:37:27 pushed ckpt_w160.pt,log_w160.txt +18:38:43 waited 76s for global r=680 +18:38:51 global r=680 load 4.280 -> 4.282 +18:39:25 step 167860 loss=3.2338 lr=2.37e-05 tok=22.093B dt=3081 wait=221 +18:40:30 step 167880 loss=3.1495 lr=2.37e-05 tok=22.096B dt=3146 wait=221 +18:41:35 step 167900 loss=3.0723 lr=2.37e-05 tok=22.099B dt=3211 wait=221 +18:42:41 step 167920 loss=3.1911 lr=2.36e-05 tok=22.101B dt=3277 wait=221 +18:43:45 step 167940 loss=3.1140 lr=2.36e-05 tok=22.104B dt=3341 wait=222 +18:44:51 step 167960 loss=3.0011 lr=2.36e-05 tok=22.107B dt=3407 wait=222 +18:45:55 step 167980 loss=3.0661 lr=2.36e-05 tok=22.109B dt=3472 wait=222 +18:47:01 step 168000 loss=2.9847 lr=2.36e-05 tok=22.112B dt=3537 wait=222 +18:47:15 pushed ckpt_w160.pt,log_w160.txt +18:48:45 waited 91s for global r=681 +18:48:52 global r=681 load 4.283 -> 4.285 +18:49:58 step 168020 loss=3.0856 lr=2.36e-05 tok=22.114B dt=3714 wait=222 +18:51:03 step 168040 loss=3.0132 lr=2.36e-05 tok=22.117B dt=3779 wait=222 +18:52:08 step 168060 loss=3.0588 lr=2.36e-05 tok=22.120B dt=3844 wait=222 +18:53:14 step 168080 loss=3.0143 lr=2.36e-05 tok=22.122B dt=3910 wait=223 +18:54:19 step 168100 loss=3.2017 lr=2.36e-05 tok=22.125B dt=3975 wait=223 +18:55:24 step 168120 loss=3.2949 lr=2.35e-05 tok=22.128B dt=4040 wait=223 +18:56:30 step 168140 loss=3.1230 lr=2.35e-05 tok=22.130B dt=4106 wait=223 +18:57:16 pushed ckpt_w160.pt,log_w160.txt +18:58:32 waited 76s for global r=682 +18:58:41 global r=682 load 4.278 -> 4.283 +18:59:14 step 168160 loss=3.1155 lr=2.35e-05 tok=22.133B dt=4270 wait=223 +19:00:20 step 168180 loss=3.1816 lr=2.35e-05 tok=22.135B dt=4336 wait=223 +19:01:25 step 168200 loss=3.1246 lr=2.35e-05 tok=22.138B dt=4401 wait=224 +19:02:30 step 168220 loss=3.2176 lr=2.35e-05 tok=22.141B dt=4466 wait=224 +19:03:35 step 168240 loss=3.0078 lr=2.35e-05 tok=22.143B dt=4531 wait=224 +19:04:42 step 168260 loss=2.4019 lr=2.35e-05 tok=22.146B dt=4598 wait=224 +19:05:48 step 168280 loss=3.1197 lr=2.35e-05 tok=22.149B dt=4664 wait=224 +19:06:53 step 168300 loss=3.2038 lr=2.35e-05 tok=22.151B dt=4729 wait=224 +19:07:07 pushed ckpt_w160.pt,log_w160.txt +19:08:22 waited 76s for global r=683 +19:08:29 global r=683 load 4.281 -> 4.281 +19:09:36 step 168320 loss=3.0948 lr=2.34e-05 tok=22.154B dt=4892 wait=224 +19:10:40 step 168340 loss=3.0812 lr=2.34e-05 tok=22.156B dt=4956 wait=225 +19:11:45 step 168360 loss=2.9872 lr=2.34e-05 tok=22.159B dt=5021 wait=225 +19:12:50 step 168380 loss=3.1331 lr=2.34e-05 tok=22.162B dt=5086 wait=225 +19:13:56 step 168400 loss=3.0385 lr=2.34e-05 tok=22.164B dt=5152 wait=225 +19:15:00 step 168420 loss=3.0886 lr=2.34e-05 tok=22.167B dt=5216 wait=225 +19:16:05 step 168440 loss=2.9050 lr=2.34e-05 tok=22.170B dt=5281 wait=225 +19:16:49 pushed ckpt_w160.pt,log_w160.txt +19:18:20 waited 91s for global r=684 +19:18:26 global r=684 load 4.291 -> 4.282 +19:19:00 step 168460 loss=3.1504 lr=2.34e-05 tok=22.172B dt=5456 wait=225 +19:20:05 step 168480 loss=3.0093 lr=2.34e-05 tok=22.175B dt=5521 wait=226 +19:21:11 step 168500 loss=3.1824 lr=2.33e-05 tok=22.177B dt=5587 wait=226 +19:22:16 step 168520 loss=3.1767 lr=2.33e-05 tok=22.180B dt=5652 wait=226 +19:23:21 step 168540 loss=3.1019 lr=2.33e-05 tok=22.183B dt=5717 wait=226 +19:24:26 step 168560 loss=3.1564 lr=2.33e-05 tok=22.185B dt=5782 wait=226 +19:25:31 step 168580 loss=3.1103 lr=2.33e-05 tok=22.188B dt=5847 wait=226 +19:26:36 step 168600 loss=3.1552 lr=2.33e-05 tok=22.190B dt=5912 wait=226 +19:26:48 pushed ckpt_w160.pt,log_w160.txt +19:27:49 waited 61s for global r=685 +19:27:57 global r=685 load 4.282 -> 4.283 +19:29:03 step 168620 loss=3.1330 lr=2.33e-05 tok=22.193B dt=6060 wait=227 +19:30:08 step 168640 loss=3.1536 lr=2.33e-05 tok=22.196B dt=6124 wait=227 +19:31:13 step 168660 loss=3.1134 lr=2.33e-05 tok=22.198B dt=6189 wait=227 +19:32:17 step 168680 loss=3.1933 lr=2.33e-05 tok=22.201B dt=6253 wait=227 +19:33:22 step 168700 loss=3.1216 lr=2.32e-05 tok=22.204B dt=6318 wait=227 +19:34:27 step 168720 loss=3.1823 lr=2.32e-05 tok=22.206B dt=6383 wait=227 +19:35:31 step 168740 loss=3.0098 lr=2.32e-05 tok=22.209B dt=6447 wait=227 +19:36:16 pushed ckpt_w160.pt,log_w160.txt +19:37:47 waited 91s for global r=686 +19:37:59 global r=686 load 4.287 -> 4.286 +19:38:33 step 168760 loss=3.1674 lr=2.32e-05 tok=22.211B dt=6629 wait=228 +19:39:37 step 168780 loss=3.0250 lr=2.32e-05 tok=22.214B dt=6693 wait=228 +19:40:41 step 168800 loss=3.0376 lr=2.32e-05 tok=22.217B dt=6757 wait=228 +19:41:46 step 168820 loss=3.0895 lr=2.32e-05 tok=22.219B dt=6822 wait=228 +19:42:51 step 168840 loss=2.9928 lr=2.32e-05 tok=22.222B dt=6887 wait=228 +19:43:56 step 168860 loss=3.0177 lr=2.32e-05 tok=22.225B dt=6952 wait=228 +19:45:00 step 168880 loss=2.9515 lr=2.32e-05 tok=22.227B dt=7016 wait=228 +19:46:05 step 168900 loss=3.0231 lr=2.31e-05 tok=22.230B dt=7081 wait=229 diff --git a/log_w161.txt b/log_w161.txt new file mode 100644 index 0000000000000000000000000000000000000000..e47dc40fa0abc73420ccbd3dbbec267ccffd4c0a --- /dev/null +++ b/log_w161.txt @@ -0,0 +1,194 @@ +launch +17:04:23 name=w161 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 108/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:04:30 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:04:30 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:04:39 params 1.08B +17:04:41 resumed step=166500 seen=21.95B round=671 +17:04:41 stream salt=166500 +17:06:31 blocks compiled +17:06:31 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:09:20 mix source 4 paused 30 min ((Request ID: Root=1-6abbf0bf-4e6277294f6a420422ee3d07;9979f7c0-96f0-4d71-8b99-72) +17:09:20 mix source 7 paused 30 min ((Request ID: Root=1-6abbf0c0-6958dffd2f21a14a56cd9955;50ce8661-3e37-475b-bfbb-8d) +17:09:20 mix source 3 paused 30 min ((Request ID: Root=1-6abbf0c0-39664fc946523eb07e698c3c;73f6d851-8858-4f8f-b0e3-f1) +17:09:20 mix source 5 paused 30 min ((Request ID: Root=1-6abbf0c0-0cabb5ff46868fc70faeb720;a2acb6d2-2c25-4942-8f9c-e6) +17:09:20 mix source 6 paused 30 min ((Request ID: Root=1-6abbf0c0-2a29e32b41ae4e241a5ce36f;f3d5fd01-9f83-43d0-9981-56) +17:09:46 step 166501 loss=3.3639 lr=2.44e-05 tok=21.955B dt=195 wait=169 +17:10:01 step 166502 loss=3.3673 lr=2.44e-05 tok=21.955B dt=210 wait=169 +17:12:00 global r=672 load 4.265 -> 4.280 +17:12:32 step 166660 loss=3.7286 lr=2.43e-05 tok=21.957B dt=361 wait=169 +17:13:36 step 166680 loss=3.5948 lr=2.43e-05 tok=21.960B dt=425 wait=169 +17:14:40 step 166700 loss=3.6126 lr=2.43e-05 tok=21.962B dt=489 wait=170 +17:15:43 step 166720 loss=3.6470 lr=2.43e-05 tok=21.965B dt=552 wait=170 +17:16:47 step 166740 loss=3.5669 lr=2.43e-05 tok=21.968B dt=616 wait=170 +17:16:55 global r=673 load 4.266 -> 4.268 +17:17:59 step 166820 loss=3.4876 lr=2.42e-05 tok=21.970B dt=688 wait=171 +17:19:04 step 166840 loss=3.5431 lr=2.42e-05 tok=21.973B dt=753 wait=171 +launch +17:19:51 name=w161 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 108/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:19:55 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:19:55 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:20:05 params 1.08B +17:20:06 resumed step=166800 seen=21.98B round=673 +17:20:06 stream salt=166800 +17:21:51 blocks compiled +17:21:51 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:23:42 mix source 6 paused 30 min ((Request ID: Root=1-6abbf41e-5ff3cf470e2f7ea85d636479;b5d0d7ac-3a73-4ece-908b-0a) +17:23:42 mix source 0 paused 30 min ((Request ID: Root=1-6abbf41e-4f3eaff85c26cf0270fd3e46;ce7508c3-8d6c-462b-8eca-9d) +17:23:42 mix source 5 paused 30 min ((Request ID: Root=1-6abbf41e-4c003b3f75e3b0cf7315dc58;dffc4a5f-9420-49c3-8c52-dd) +17:23:42 mix source 8 paused 30 min ((Request ID: Root=1-6abbf41e-1bccdc5f1f76cfab5753a518;1fb72ccf-c975-4787-9eeb-78) +17:23:42 mix source 3 paused 30 min ((Request ID: Root=1-6abbf41e-38f2f892303fa194365182b9;2f6c86f9-3efb-4948-a695-c7) +17:23:42 mix source 7 paused 30 min ((Request ID: Root=1-6abbf41e-72de51007ded07c963d767ef;ab55a7b5-d68a-47b6-bb97-f6) +17:24:03 step 166801 loss=3.0620 lr=2.42e-05 tok=21.976B dt=132 wait=111 +17:24:13 step 166802 loss=3.1708 lr=2.42e-05 tok=21.976B dt=143 wait=111 +17:25:16 step 166820 loss=3.4261 lr=2.42e-05 tok=21.978B dt=205 wait=112 +17:26:24 step 166840 loss=2.9974 lr=2.42e-05 tok=21.981B dt=273 wait=112 +17:27:28 step 166860 loss=3.3913 lr=2.42e-05 tok=21.983B dt=337 wait=113 +17:28:32 step 166880 loss=3.5298 lr=2.42e-05 tok=21.986B dt=402 wait=113 +17:29:38 step 166900 loss=3.4981 lr=2.42e-05 tok=21.989B dt=467 wait=113 +17:30:43 step 166920 loss=3.4474 lr=2.42e-05 tok=21.991B dt=532 wait=113 +17:31:48 step 166940 loss=3.3913 lr=2.42e-05 tok=21.994B dt=597 wait=114 +17:32:34 pushed ckpt_w161.pt,log_w161.txt +17:35:35 waited 181s for global r=674 +17:35:51 global r=674 load 4.258 -> 4.258 +17:36:26 step 166960 loss=3.5190 lr=2.42e-05 tok=21.997B dt=875 wait=114 +17:37:32 step 166980 loss=3.2976 lr=2.41e-05 tok=21.999B dt=941 wait=114 +17:38:36 step 167000 loss=3.3665 lr=2.41e-05 tok=22.002B dt=1005 wait=115 +17:39:41 step 167020 loss=3.4002 lr=2.41e-05 tok=22.004B dt=1070 wait=115 +17:40:45 step 167040 loss=3.3466 lr=2.41e-05 tok=22.007B dt=1134 wait=115 +17:41:50 step 167060 loss=3.4225 lr=2.41e-05 tok=22.010B dt=1199 wait=116 +17:42:54 step 167080 loss=3.3047 lr=2.41e-05 tok=22.012B dt=1264 wait=116 +17:43:58 step 167100 loss=3.3741 lr=2.41e-05 tok=22.015B dt=1327 wait=116 +17:44:16 pushed ckpt_w161.pt,log_w161.txt +17:47:17 waited 181s for global r=675 +17:47:24 global r=675 load 4.267 -> 4.260 +17:48:29 step 167120 loss=3.4047 lr=2.41e-05 tok=22.017B dt=1598 wait=117 +launch +17:48:42 name=w161 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 108/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:48:47 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:48:48 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:48:57 params 1.08B +17:48:59 resumed step=167100 seen=22.02B round=675 +17:48:59 stream salt=167100 +17:50:44 blocks compiled +17:50:44 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:54:18 step 167101 loss=3.3159 lr=2.41e-05 tok=22.016B dt=215 wait=204 +17:54:26 step 167102 loss=3.5009 lr=2.41e-05 tok=22.016B dt=222 wait=204 +17:55:23 step 167120 loss=3.2867 lr=2.41e-05 tok=22.019B dt=279 wait=204 +17:56:28 step 167140 loss=3.1656 lr=2.41e-05 tok=22.021B dt=344 wait=204 +17:57:32 step 167160 loss=3.2019 lr=2.40e-05 tok=22.024B dt=408 wait=204 +17:58:36 step 167180 loss=3.2050 lr=2.40e-05 tok=22.027B dt=472 wait=204 +17:59:40 step 167200 loss=3.2945 lr=2.40e-05 tok=22.029B dt=537 wait=204 +17:59:51 global r=676 load 4.282 -> 4.285 +18:00:23 step 167260 loss=3.3240 lr=2.40e-05 tok=22.031B dt=579 wait=204 +18:01:27 step 167280 loss=3.1609 lr=2.40e-05 tok=22.033B dt=644 wait=204 +18:02:31 step 167300 loss=3.1944 lr=2.40e-05 tok=22.036B dt=708 wait=205 +18:03:36 step 167320 loss=3.2736 lr=2.40e-05 tok=22.038B dt=772 wait=205 +18:04:40 step 167340 loss=3.0273 lr=2.40e-05 tok=22.041B dt=837 wait=205 +18:05:45 step 167360 loss=3.2370 lr=2.39e-05 tok=22.044B dt=901 wait=205 +18:06:49 step 167380 loss=3.1094 lr=2.39e-05 tok=22.046B dt=966 wait=205 +18:07:54 step 167400 loss=3.1622 lr=2.39e-05 tok=22.049B dt=1031 wait=205 +18:08:07 pushed ckpt_w161.pt,log_w161.txt +18:09:22 waited 76s for global r=677 +18:09:29 global r=677 load 4.283 -> 4.280 +18:10:36 step 167420 loss=3.1849 lr=2.39e-05 tok=22.052B dt=1192 wait=206 +18:11:41 step 167440 loss=3.2241 lr=2.39e-05 tok=22.054B dt=1257 wait=206 +18:12:45 step 167460 loss=3.2871 lr=2.39e-05 tok=22.057B dt=1321 wait=206 +18:13:50 step 167480 loss=3.1275 lr=2.39e-05 tok=22.059B dt=1386 wait=206 +18:14:54 step 167500 loss=3.1060 lr=2.39e-05 tok=22.062B dt=1450 wait=206 +18:15:58 step 167520 loss=3.1793 lr=2.39e-05 tok=22.065B dt=1514 wait=206 +18:17:02 step 167540 loss=3.1237 lr=2.38e-05 tok=22.067B dt=1578 wait=206 +18:17:45 pushed ckpt_w161.pt,log_w161.txt +18:19:01 waited 76s for global r=678 +18:19:08 global r=678 load 4.287 -> 4.281 +18:19:42 step 167560 loss=3.1225 lr=2.38e-05 tok=22.070B dt=1738 wait=207 +18:20:47 step 167580 loss=2.9297 lr=2.38e-05 tok=22.073B dt=1804 wait=207 +18:21:52 step 167600 loss=3.0164 lr=2.38e-05 tok=22.075B dt=1869 wait=207 +18:22:57 step 167620 loss=3.0830 lr=2.38e-05 tok=22.078B dt=1933 wait=207 +18:24:03 step 167640 loss=2.9097 lr=2.38e-05 tok=22.080B dt=1999 wait=207 +18:25:09 step 167660 loss=3.0712 lr=2.38e-05 tok=22.083B dt=2066 wait=207 +18:26:15 step 167680 loss=2.9666 lr=2.38e-05 tok=22.086B dt=2131 wait=208 +18:27:21 step 167700 loss=2.9941 lr=2.38e-05 tok=22.088B dt=2197 wait=208 +18:27:34 pushed ckpt_w161.pt,log_w161.txt +18:28:49 waited 76s for global r=679 +18:28:57 global r=679 load 4.292 -> 4.283 +18:30:03 step 167720 loss=3.0056 lr=2.38e-05 tok=22.091B dt=2360 wait=208 +18:31:09 step 167740 loss=3.0930 lr=2.37e-05 tok=22.093B dt=2425 wait=208 +18:32:14 step 167760 loss=3.0946 lr=2.37e-05 tok=22.096B dt=2490 wait=208 +18:33:20 step 167780 loss=2.9476 lr=2.37e-05 tok=22.099B dt=2556 wait=208 +18:34:25 step 167800 loss=2.9441 lr=2.37e-05 tok=22.101B dt=2622 wait=208 +18:35:31 step 167820 loss=3.0473 lr=2.37e-05 tok=22.104B dt=2688 wait=209 +18:36:36 step 167840 loss=2.9335 lr=2.37e-05 tok=22.107B dt=2753 wait=209 +18:37:21 pushed ckpt_w161.pt,log_w161.txt +18:38:37 waited 76s for global r=680 +18:38:43 global r=680 load 4.294 -> 4.282 +18:39:18 step 167860 loss=2.9504 lr=2.37e-05 tok=22.109B dt=2914 wait=209 +18:40:24 step 167880 loss=2.8572 lr=2.37e-05 tok=22.112B dt=2980 wait=209 +18:41:29 step 167900 loss=2.8598 lr=2.37e-05 tok=22.114B dt=3046 wait=209 +18:42:35 step 167920 loss=2.6776 lr=2.36e-05 tok=22.117B dt=3111 wait=209 +18:43:40 step 167940 loss=2.8629 lr=2.36e-05 tok=22.120B dt=3177 wait=210 +18:44:47 step 167960 loss=2.8878 lr=2.36e-05 tok=22.122B dt=3243 wait=210 +18:45:52 step 167980 loss=2.8375 lr=2.36e-05 tok=22.125B dt=3309 wait=210 +18:46:58 step 168000 loss=2.9613 lr=2.36e-05 tok=22.128B dt=3374 wait=210 +18:47:12 pushed ckpt_w161.pt,log_w161.txt +18:48:43 waited 91s for global r=681 +18:48:51 global r=681 load 4.295 -> 4.285 +18:49:58 step 168020 loss=2.8799 lr=2.36e-05 tok=22.130B dt=3554 wait=210 +18:51:03 step 168040 loss=2.9935 lr=2.36e-05 tok=22.133B dt=3619 wait=210 +18:52:09 step 168060 loss=2.9660 lr=2.36e-05 tok=22.135B dt=3685 wait=210 +18:53:14 step 168080 loss=2.9185 lr=2.36e-05 tok=22.138B dt=3750 wait=211 +18:54:20 step 168100 loss=2.9411 lr=2.36e-05 tok=22.141B dt=3816 wait=211 +18:55:26 step 168120 loss=2.9971 lr=2.35e-05 tok=22.143B dt=3882 wait=211 +18:56:31 step 168140 loss=2.8846 lr=2.35e-05 tok=22.146B dt=3948 wait=211 +18:57:17 pushed ckpt_w161.pt,log_w161.txt +18:58:32 waited 76s for global r=682 +18:58:42 global r=682 load 4.296 -> 4.283 +18:59:15 step 168160 loss=3.0987 lr=2.35e-05 tok=22.149B dt=4111 wait=211 +19:00:20 step 168180 loss=2.8397 lr=2.35e-05 tok=22.151B dt=4177 wait=211 +19:01:26 step 168200 loss=2.9600 lr=2.35e-05 tok=22.154B dt=4242 wait=212 +19:02:31 step 168220 loss=3.0381 lr=2.35e-05 tok=22.156B dt=4307 wait=212 +19:03:35 step 168240 loss=2.8697 lr=2.35e-05 tok=22.159B dt=4372 wait=212 +19:04:39 step 168260 loss=3.0172 lr=2.35e-05 tok=22.162B dt=4436 wait=212 +19:05:44 step 168280 loss=3.0024 lr=2.35e-05 tok=22.164B dt=4500 wait=212 +19:06:49 step 168300 loss=3.0210 lr=2.35e-05 tok=22.167B dt=4565 wait=212 +19:07:01 pushed ckpt_w161.pt,log_w161.txt +19:08:16 waited 76s for global r=683 +19:08:25 global r=683 load 4.283 -> 4.281 +19:09:31 step 168320 loss=2.9823 lr=2.34e-05 tok=22.170B dt=4727 wait=213 +19:10:35 step 168340 loss=3.0059 lr=2.34e-05 tok=22.172B dt=4791 wait=213 +19:11:39 step 168360 loss=2.9368 lr=2.34e-05 tok=22.175B dt=4855 wait=213 +19:12:43 step 168380 loss=3.1587 lr=2.34e-05 tok=22.177B dt=4919 wait=213 +19:13:47 step 168400 loss=3.1524 lr=2.34e-05 tok=22.180B dt=4984 wait=213 +19:14:53 step 168420 loss=3.2658 lr=2.34e-05 tok=22.183B dt=5049 wait=213 +19:15:58 step 168440 loss=3.1969 lr=2.34e-05 tok=22.185B dt=5114 wait=213 +19:16:45 pushed ckpt_w161.pt,log_w161.txt +19:18:15 waited 91s for global r=684 +19:18:23 global r=684 load 4.283 -> 4.282 +19:18:56 step 168460 loss=3.2870 lr=2.34e-05 tok=22.188B dt=5292 wait=214 +19:20:00 step 168480 loss=3.2447 lr=2.34e-05 tok=22.190B dt=5356 wait=214 +19:21:04 step 168500 loss=3.1781 lr=2.33e-05 tok=22.193B dt=5420 wait=214 +19:22:08 step 168520 loss=3.2390 lr=2.33e-05 tok=22.196B dt=5485 wait=214 +19:23:14 step 168540 loss=3.1833 lr=2.33e-05 tok=22.198B dt=5550 wait=214 +19:24:18 step 168560 loss=3.1200 lr=2.33e-05 tok=22.201B dt=5615 wait=214 +19:25:23 step 168580 loss=3.1423 lr=2.33e-05 tok=22.204B dt=5679 wait=215 +19:26:27 step 168600 loss=3.1607 lr=2.33e-05 tok=22.206B dt=5743 wait=215 +19:26:39 pushed ckpt_w161.pt,log_w161.txt +19:27:55 waited 76s for global r=685 +19:28:00 global r=685 load 4.283 -> 4.283 +19:29:06 step 168620 loss=3.1190 lr=2.33e-05 tok=22.209B dt=5902 wait=215 +19:30:10 step 168640 loss=3.2408 lr=2.33e-05 tok=22.211B dt=5967 wait=215 +19:31:15 step 168660 loss=3.1552 lr=2.33e-05 tok=22.214B dt=6031 wait=215 +19:32:19 step 168680 loss=3.0803 lr=2.33e-05 tok=22.217B dt=6095 wait=215 +19:33:22 step 168700 loss=3.0511 lr=2.32e-05 tok=22.219B dt=6159 wait=215 +19:34:27 step 168720 loss=3.0431 lr=2.32e-05 tok=22.222B dt=6223 wait=216 +19:35:31 step 168740 loss=3.0588 lr=2.32e-05 tok=22.225B dt=6288 wait=216 +19:36:16 pushed ckpt_w161.pt,log_w161.txt +19:37:47 waited 91s for global r=686 +19:37:53 global r=686 load 4.285 -> 4.286 +19:38:26 step 168760 loss=3.1641 lr=2.32e-05 tok=22.227B dt=6462 wait=216 +19:39:30 step 168780 loss=3.0390 lr=2.32e-05 tok=22.230B dt=6527 wait=216 +19:40:35 step 168800 loss=2.9306 lr=2.32e-05 tok=22.232B dt=6591 wait=216 +19:41:38 step 168820 loss=3.0968 lr=2.32e-05 tok=22.235B dt=6655 wait=216 +19:42:43 step 168840 loss=2.8139 lr=2.32e-05 tok=22.238B dt=6720 wait=216 +19:43:48 step 168860 loss=3.0999 lr=2.32e-05 tok=22.240B dt=6784 wait=217 +19:44:53 step 168880 loss=2.9062 lr=2.32e-05 tok=22.243B dt=6850 wait=217 +19:45:57 step 168900 loss=3.0558 lr=2.31e-05 tok=22.246B dt=6914 wait=217 diff --git a/log_w162.txt b/log_w162.txt new file mode 100644 index 0000000000000000000000000000000000000000..de3d198c674ecda8de5e310a0fafdf950f6fa901 --- /dev/null +++ b/log_w162.txt @@ -0,0 +1,188 @@ +launch +17:04:26 name=w162 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 109/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:04:32 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:04:32 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:04:41 params 1.08B +17:04:43 resumed step=166500 seen=21.95B round=671 +17:04:43 stream salt=166500 +17:06:30 blocks compiled +17:06:30 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:08:55 mix source 2 paused 30 min ((Request ID: Root=1-6abbf0a7-0c35d5652712f8702e3e97b7;f2f4b7bf-51cd-424f-b002-ee) +17:09:32 step 166501 loss=2.8404 lr=2.44e-05 tok=21.955B dt=182 wait=145 +17:09:55 step 166502 loss=2.8766 lr=2.44e-05 tok=21.955B dt=205 wait=145 +17:12:25 global r=672 load 4.288 -> 4.280 +17:12:57 step 166660 loss=3.0775 lr=2.43e-05 tok=21.957B dt=387 wait=146 +17:14:00 step 166680 loss=3.0214 lr=2.43e-05 tok=21.960B dt=450 wait=146 +17:15:03 step 166700 loss=2.9532 lr=2.43e-05 tok=21.962B dt=513 wait=146 +17:16:06 step 166720 loss=3.1004 lr=2.43e-05 tok=21.965B dt=576 wait=146 +17:16:46 global r=673 load 4.274 -> 4.268 +17:17:49 step 166820 loss=2.9777 lr=2.42e-05 tok=21.969B dt=679 wait=147 +17:18:52 step 166840 loss=3.0355 lr=2.42e-05 tok=21.972B dt=742 wait=147 +launch +17:19:46 name=w162 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 109/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:19:50 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:19:50 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:20:00 params 1.08B +17:20:01 resumed step=166800 seen=21.98B round=673 +17:20:01 stream salt=166800 +17:21:50 blocks compiled +17:21:50 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:24:05 mix source 8 paused 30 min ((Request ID: Root=1-6abbf435-5865657629664add29c32946;36ac5147-4bd7-4ab7-b2da-da) +17:24:05 mix source 2 paused 30 min ((Request ID: Root=1-6abbf435-5789ddbb23ab5ade7c969aa1;4fc41cbd-eeaf-4510-9325-aa) +17:24:05 mix source 0 paused 30 min ((Request ID: Root=1-6abbf435-7860dfd201419ca42a196d56;d9608085-2bff-44d0-9e01-81) +17:24:05 mix source 4 paused 30 min ((Request ID: Root=1-6abbf435-30bf59e70274c44b5882bba5;23fc4c7f-32eb-4303-b0aa-7e) +17:24:05 mix source 1 paused 30 min ((Request ID: Root=1-6abbf435-074e7097653304417a74ed35;7bc5c3e5-e053-45fb-ac7b-0c) +17:24:06 mix source 5 paused 30 min ((Request ID: Root=1-6abbf436-6a9705eb7e70525328d5693e;c804e9bb-0fed-47ca-a178-0d) +17:24:21 step 166801 loss=2.8828 lr=2.42e-05 tok=21.976B dt=151 wait=136 +17:24:28 step 166802 loss=2.9954 lr=2.42e-05 tok=21.976B dt=158 wait=136 +17:25:24 step 166820 loss=3.1736 lr=2.42e-05 tok=21.978B dt=214 wait=136 +17:26:27 step 166840 loss=3.1858 lr=2.42e-05 tok=21.981B dt=277 wait=136 +17:27:30 step 166860 loss=3.1627 lr=2.42e-05 tok=21.983B dt=339 wait=137 +17:28:31 step 166880 loss=3.2932 lr=2.42e-05 tok=21.986B dt=401 wait=137 +17:29:34 step 166900 loss=3.3096 lr=2.42e-05 tok=21.989B dt=463 wait=138 +17:30:36 step 166920 loss=3.1864 lr=2.42e-05 tok=21.991B dt=525 wait=138 +17:31:38 step 166940 loss=3.1360 lr=2.42e-05 tok=21.994B dt=588 wait=138 +17:32:26 pushed ckpt_w162.pt,log_w162.txt +17:36:02 step 166960 loss=3.3374 lr=2.42e-05 tok=21.997B dt=851 wait=139 +17:36:12 global r=674 load 4.257 -> 4.258 +17:36:43 step 166960 loss=3.3076 lr=2.42e-05 tok=21.998B dt=892 wait=139 +17:37:44 step 166980 loss=3.1331 lr=2.41e-05 tok=22.000B dt=954 wait=139 +17:38:46 step 167000 loss=3.2299 lr=2.41e-05 tok=22.003B dt=1016 wait=140 +17:39:48 step 167020 loss=3.2920 lr=2.41e-05 tok=22.006B dt=1078 wait=140 +17:40:50 step 167040 loss=3.1946 lr=2.41e-05 tok=22.008B dt=1140 wait=141 +17:41:52 step 167060 loss=3.3223 lr=2.41e-05 tok=22.011B dt=1202 wait=141 +17:42:54 step 167080 loss=3.2696 lr=2.41e-05 tok=22.014B dt=1263 wait=141 +17:43:55 step 167100 loss=3.2395 lr=2.41e-05 tok=22.016B dt=1325 wait=142 +17:44:07 pushed ckpt_w162.pt,log_w162.txt +17:47:09 waited 181s for global r=675 +17:47:18 global r=675 load 4.255 -> 4.260 +17:48:21 step 167120 loss=3.2119 lr=2.41e-05 tok=22.019B dt=1591 wait=142 +17:49:22 step 167140 loss=3.2157 lr=2.41e-05 tok=22.021B dt=1652 wait=143 +17:50:24 step 167160 loss=3.2595 lr=2.40e-05 tok=22.024B dt=1714 wait=143 +launch +17:51:28 name=w162 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 109/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:51:33 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:51:33 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:51:42 params 1.08B +17:51:44 resumed step=167100 seen=22.02B round=675 +17:51:44 stream salt=167100 +17:53:31 blocks compiled +17:53:31 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +17:57:22 step 167101 loss=2.8792 lr=2.41e-05 tok=22.016B dt=231 wait=220 +17:57:29 step 167102 loss=2.9378 lr=2.41e-05 tok=22.016B dt=238 wait=220 +17:58:27 step 167120 loss=3.1283 lr=2.41e-05 tok=22.019B dt=296 wait=220 +17:59:31 step 167140 loss=3.0694 lr=2.41e-05 tok=22.021B dt=361 wait=220 +17:59:43 global r=676 load 4.277 -> 4.285 +18:00:15 step 167260 loss=3.0732 lr=2.40e-05 tok=22.023B dt=404 wait=220 +18:01:19 step 167280 loss=3.0827 lr=2.40e-05 tok=22.025B dt=468 wait=220 +18:02:23 step 167300 loss=3.0682 lr=2.40e-05 tok=22.028B dt=532 wait=220 +18:03:27 step 167320 loss=3.2388 lr=2.40e-05 tok=22.031B dt=597 wait=221 +18:04:32 step 167340 loss=3.1915 lr=2.40e-05 tok=22.033B dt=661 wait=221 +18:05:36 step 167360 loss=3.1822 lr=2.39e-05 tok=22.036B dt=725 wait=221 +18:06:40 step 167380 loss=3.0908 lr=2.39e-05 tok=22.038B dt=789 wait=221 +18:07:44 step 167400 loss=3.1238 lr=2.39e-05 tok=22.041B dt=854 wait=221 +18:07:58 pushed ckpt_w162.pt,log_w162.txt +18:09:13 waited 76s for global r=677 +18:09:23 global r=677 load 4.286 -> 4.280 +18:10:29 step 167420 loss=3.0846 lr=2.39e-05 tok=22.044B dt=1018 wait=221 +18:11:33 step 167440 loss=3.1088 lr=2.39e-05 tok=22.046B dt=1083 wait=221 +18:12:38 step 167460 loss=3.1327 lr=2.39e-05 tok=22.049B dt=1147 wait=222 +18:13:43 step 167480 loss=3.1580 lr=2.39e-05 tok=22.052B dt=1212 wait=222 +18:14:47 step 167500 loss=3.0958 lr=2.39e-05 tok=22.054B dt=1277 wait=222 +18:15:51 step 167520 loss=3.1723 lr=2.39e-05 tok=22.057B dt=1340 wait=222 +18:16:56 step 167540 loss=3.0333 lr=2.38e-05 tok=22.059B dt=1405 wait=222 +18:17:43 pushed ckpt_w162.pt,log_w162.txt +18:18:59 waited 76s for global r=678 +18:19:05 global r=678 load 4.287 -> 4.281 +18:19:39 step 167560 loss=3.1603 lr=2.38e-05 tok=22.062B dt=1568 wait=222 +18:20:43 step 167580 loss=3.0858 lr=2.38e-05 tok=22.065B dt=1633 wait=222 +18:21:47 step 167600 loss=3.0076 lr=2.38e-05 tok=22.067B dt=1696 wait=223 +18:22:51 step 167620 loss=3.1367 lr=2.38e-05 tok=22.070B dt=1760 wait=223 +18:23:56 step 167640 loss=3.0162 lr=2.38e-05 tok=22.073B dt=1825 wait=223 +18:25:01 step 167660 loss=3.0987 lr=2.38e-05 tok=22.075B dt=1890 wait=223 +18:26:04 step 167680 loss=3.0044 lr=2.38e-05 tok=22.078B dt=1954 wait=223 +18:27:10 step 167700 loss=3.0940 lr=2.38e-05 tok=22.080B dt=2019 wait=223 +18:27:23 pushed ckpt_w162.pt,log_w162.txt +18:28:54 waited 91s for global r=679 +18:29:02 global r=679 load 4.287 -> 4.283 +18:30:09 step 167720 loss=3.0207 lr=2.38e-05 tok=22.083B dt=2198 wait=223 +18:31:13 step 167740 loss=3.0643 lr=2.37e-05 tok=22.086B dt=2263 wait=223 +18:32:18 step 167760 loss=3.0719 lr=2.37e-05 tok=22.088B dt=2327 wait=224 +18:33:22 step 167780 loss=2.9768 lr=2.37e-05 tok=22.091B dt=2391 wait=224 +18:34:26 step 167800 loss=3.0873 lr=2.37e-05 tok=22.093B dt=2455 wait=224 +18:35:30 step 167820 loss=3.0297 lr=2.37e-05 tok=22.096B dt=2520 wait=224 +18:36:34 step 167840 loss=2.9621 lr=2.37e-05 tok=22.099B dt=2583 wait=224 +18:37:21 pushed ckpt_w162.pt,log_w162.txt +18:38:36 waited 76s for global r=680 +18:38:42 global r=680 load 4.280 -> 4.282 +18:39:15 step 167860 loss=3.0182 lr=2.37e-05 tok=22.101B dt=2745 wait=224 +18:40:19 step 167880 loss=2.9824 lr=2.37e-05 tok=22.104B dt=2809 wait=224 +18:41:23 step 167900 loss=2.8594 lr=2.37e-05 tok=22.107B dt=2872 wait=225 +18:42:27 step 167920 loss=2.9238 lr=2.36e-05 tok=22.109B dt=2936 wait=225 +18:43:31 step 167940 loss=2.8587 lr=2.36e-05 tok=22.112B dt=3000 wait=225 +18:44:35 step 167960 loss=2.9609 lr=2.36e-05 tok=22.114B dt=3064 wait=225 +18:45:38 step 167980 loss=2.9374 lr=2.36e-05 tok=22.117B dt=3128 wait=225 +18:46:43 step 168000 loss=2.6931 lr=2.36e-05 tok=22.120B dt=3193 wait=225 +18:46:55 pushed ckpt_w162.pt,log_w162.txt +18:48:41 waited 106s for global r=681 +18:48:49 global r=681 load 4.285 -> 4.285 +18:49:55 step 168020 loss=2.7065 lr=2.36e-05 tok=22.122B dt=3384 wait=225 +18:50:59 step 168040 loss=3.0106 lr=2.36e-05 tok=22.125B dt=3448 wait=226 +18:52:03 step 168060 loss=2.9490 lr=2.36e-05 tok=22.128B dt=3512 wait=226 +18:53:07 step 168080 loss=3.0054 lr=2.36e-05 tok=22.130B dt=3576 wait=226 +18:54:12 step 168100 loss=3.1055 lr=2.36e-05 tok=22.133B dt=3641 wait=226 +18:55:16 step 168120 loss=3.0832 lr=2.35e-05 tok=22.135B dt=3705 wait=226 +18:56:20 step 168140 loss=2.9260 lr=2.35e-05 tok=22.138B dt=3769 wait=226 +18:57:06 pushed ckpt_w162.pt,log_w162.txt +18:58:37 waited 91s for global r=682 +18:58:47 global r=682 load 4.273 -> 4.283 +18:59:20 step 168160 loss=3.1324 lr=2.35e-05 tok=22.141B dt=3950 wait=226 +19:00:24 step 168180 loss=2.9455 lr=2.35e-05 tok=22.143B dt=4014 wait=226 +19:01:28 step 168200 loss=2.8743 lr=2.35e-05 tok=22.146B dt=4078 wait=227 +19:02:32 step 168220 loss=3.1520 lr=2.35e-05 tok=22.149B dt=4141 wait=227 +19:03:37 step 168240 loss=2.9832 lr=2.35e-05 tok=22.151B dt=4206 wait=227 +19:04:41 step 168260 loss=3.0820 lr=2.35e-05 tok=22.154B dt=4270 wait=227 +19:05:45 step 168280 loss=3.0109 lr=2.35e-05 tok=22.156B dt=4334 wait=227 +19:06:49 step 168300 loss=2.9949 lr=2.35e-05 tok=22.159B dt=4399 wait=227 +19:07:02 pushed ckpt_w162.pt,log_w162.txt +19:08:17 waited 76s for global r=683 +19:08:23 global r=683 load 4.279 -> 4.281 +19:09:28 step 168320 loss=2.8898 lr=2.34e-05 tok=22.162B dt=4557 wait=227 +19:10:32 step 168340 loss=2.9599 lr=2.34e-05 tok=22.164B dt=4622 wait=228 +19:11:37 step 168360 loss=2.9591 lr=2.34e-05 tok=22.167B dt=4687 wait=228 +19:12:41 step 168380 loss=2.9231 lr=2.34e-05 tok=22.170B dt=4751 wait=228 +19:13:46 step 168400 loss=2.9598 lr=2.34e-05 tok=22.172B dt=4815 wait=228 +19:14:50 step 168420 loss=3.0596 lr=2.34e-05 tok=22.175B dt=4880 wait=228 +19:15:54 step 168440 loss=2.8860 lr=2.34e-05 tok=22.177B dt=4944 wait=228 +19:16:41 pushed ckpt_w162.pt,log_w162.txt +19:18:12 waited 91s for global r=684 +19:18:19 global r=684 load 4.281 -> 4.282 +19:18:52 step 168460 loss=3.0006 lr=2.34e-05 tok=22.180B dt=5122 wait=228 +19:19:57 step 168480 loss=2.9544 lr=2.34e-05 tok=22.183B dt=5187 wait=229 +19:21:02 step 168500 loss=2.9374 lr=2.33e-05 tok=22.185B dt=5252 wait=229 +19:22:06 step 168520 loss=2.9857 lr=2.33e-05 tok=22.188B dt=5316 wait=229 +19:23:11 step 168540 loss=2.9972 lr=2.33e-05 tok=22.190B dt=5380 wait=229 +19:24:15 step 168560 loss=2.9519 lr=2.33e-05 tok=22.193B dt=5444 wait=229 +19:25:19 step 168580 loss=2.9442 lr=2.33e-05 tok=22.196B dt=5508 wait=229 +19:26:23 step 168600 loss=2.9506 lr=2.33e-05 tok=22.198B dt=5572 wait=229 +19:26:37 pushed ckpt_w162.pt,log_w162.txt +19:27:52 waited 76s for global r=685 +19:27:58 global r=685 load 4.297 -> 4.283 +19:29:03 step 168620 loss=2.9168 lr=2.33e-05 tok=22.201B dt=5732 wait=230 +19:30:06 step 168640 loss=2.9488 lr=2.33e-05 tok=22.204B dt=5796 wait=230 +19:31:10 step 168660 loss=2.9928 lr=2.33e-05 tok=22.206B dt=5860 wait=230 +19:32:14 step 168680 loss=2.9040 lr=2.33e-05 tok=22.209B dt=5924 wait=230 +19:33:19 step 168700 loss=2.8748 lr=2.32e-05 tok=22.211B dt=5988 wait=230 +19:34:22 step 168720 loss=2.9476 lr=2.32e-05 tok=22.214B dt=6052 wait=230 +19:35:26 step 168740 loss=2.8831 lr=2.32e-05 tok=22.217B dt=6115 wait=230 +19:36:11 pushed ckpt_w162.pt,log_w162.txt +19:37:42 waited 91s for global r=686 +19:37:48 global r=686 load 4.288 -> 4.286 +19:38:22 step 168760 loss=2.9152 lr=2.32e-05 tok=22.219B dt=6291 wait=231 +19:39:25 step 168780 loss=2.8490 lr=2.32e-05 tok=22.222B dt=6355 wait=231 +19:40:29 step 168800 loss=2.8042 lr=2.32e-05 tok=22.225B dt=6419 wait=231 +19:41:33 step 168820 loss=2.9486 lr=2.32e-05 tok=22.227B dt=6483 wait=231 +19:42:37 step 168840 loss=2.8054 lr=2.32e-05 tok=22.230B dt=6546 wait=231 +19:43:40 step 168860 loss=2.8024 lr=2.32e-05 tok=22.232B dt=6610 wait=231 +19:44:44 step 168880 loss=2.7943 lr=2.32e-05 tok=22.235B dt=6673 wait=231 +19:45:47 step 168900 loss=2.8444 lr=2.31e-05 tok=22.238B dt=6737 wait=232 diff --git a/log_w163.txt b/log_w163.txt new file mode 100644 index 0000000000000000000000000000000000000000..9e03ad58ce781679ed0282a56940670bfc492fba --- /dev/null +++ b/log_w163.txt @@ -0,0 +1,240 @@ +launch +20:00:22 name=w163 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 110/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:00:29 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:00:29 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +20:00:39 params 1.08B +20:00:40 resumed step=169050 seen=22.26B round=688 +20:00:40 stream salt=169050 +20:02:25 blocks compiled +20:02:25 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +20:06:31 step 169051 loss=2.9949 lr=2.31e-05 tok=22.260B dt=245 wait=205 +20:06:55 step 169052 loss=3.1919 lr=2.31e-05 tok=22.260B dt=270 wait=205 +20:07:22 step 169060 loss=3.2994 lr=2.31e-05 tok=22.261B dt=296 wait=205 +20:08:28 step 169080 loss=3.2468 lr=2.31e-05 tok=22.264B dt=362 wait=205 +20:09:15 global r=689 load 4.296 -> 4.280 +20:10:20 step 169220 loss=3.1759 lr=2.30e-05 tok=22.268B dt=475 wait=205 +20:11:24 step 169240 loss=3.2327 lr=2.30e-05 tok=22.270B dt=539 wait=205 +20:12:29 step 169260 loss=3.2298 lr=2.30e-05 tok=22.273B dt=604 wait=206 +20:13:34 step 169280 loss=3.1435 lr=2.30e-05 tok=22.276B dt=669 wait=206 +20:14:40 step 169300 loss=3.2108 lr=2.29e-05 tok=22.278B dt=734 wait=206 +20:15:44 step 169320 loss=3.2815 lr=2.29e-05 tok=22.281B dt=799 wait=206 +20:16:50 step 169340 loss=3.1263 lr=2.29e-05 tok=22.284B dt=864 wait=206 +20:17:37 pushed ckpt_w163.pt,log_w163.txt +20:18:53 waited 76s for global r=690 +20:19:03 global r=690 load 4.293 -> 4.284 +20:19:37 step 169360 loss=3.2908 lr=2.29e-05 tok=22.286B dt=1032 wait=206 +20:20:43 step 169380 loss=3.1680 lr=2.29e-05 tok=22.289B dt=1097 wait=206 +20:21:48 step 169400 loss=3.2397 lr=2.29e-05 tok=22.291B dt=1163 wait=207 +20:22:54 step 169420 loss=3.2269 lr=2.29e-05 tok=22.294B dt=1229 wait=207 +20:24:00 step 169440 loss=3.1650 lr=2.29e-05 tok=22.297B dt=1294 wait=207 +20:25:04 step 169460 loss=3.2296 lr=2.29e-05 tok=22.299B dt=1359 wait=207 +20:26:09 step 169480 loss=3.1367 lr=2.29e-05 tok=22.302B dt=1423 wait=207 +20:27:14 step 169500 loss=3.3295 lr=2.28e-05 tok=22.305B dt=1488 wait=207 +20:27:26 pushed ckpt_w163.pt,log_w163.txt +20:28:26 waited 61s for global r=691 +20:28:36 global r=691 load 4.291 -> 4.287 +20:29:42 step 169520 loss=3.2735 lr=2.28e-05 tok=22.307B dt=1637 wait=208 +20:30:47 step 169540 loss=3.2080 lr=2.28e-05 tok=22.310B dt=1702 wait=208 +20:31:51 step 169560 loss=3.2771 lr=2.28e-05 tok=22.312B dt=1766 wait=208 +20:32:56 step 169580 loss=3.1557 lr=2.28e-05 tok=22.315B dt=1830 wait=208 +20:34:00 step 169600 loss=3.0983 lr=2.28e-05 tok=22.318B dt=1895 wait=208 +20:35:05 step 169620 loss=3.2558 lr=2.28e-05 tok=22.320B dt=1959 wait=208 +20:36:09 step 169640 loss=2.7560 lr=2.28e-05 tok=22.323B dt=2024 wait=208 +20:36:54 pushed ckpt_w163.pt,log_w163.txt +20:38:10 waited 76s for global r=692 +20:38:15 global r=692 load 4.286 -> 4.284 +20:38:49 step 169660 loss=3.0319 lr=2.28e-05 tok=22.325B dt=2184 wait=209 +20:39:54 step 169680 loss=2.9910 lr=2.28e-05 tok=22.328B dt=2248 wait=209 +20:40:58 step 169700 loss=2.9964 lr=2.27e-05 tok=22.331B dt=2312 wait=209 +20:42:02 step 169720 loss=3.1006 lr=2.27e-05 tok=22.333B dt=2377 wait=209 +20:43:06 step 169740 loss=2.9677 lr=2.27e-05 tok=22.336B dt=2441 wait=209 +20:44:10 step 169760 loss=3.0719 lr=2.27e-05 tok=22.339B dt=2505 wait=209 +20:45:13 step 169780 loss=3.0491 lr=2.27e-05 tok=22.341B dt=2568 wait=209 +20:46:17 step 169800 loss=3.0365 lr=2.27e-05 tok=22.344B dt=2632 wait=210 +20:46:30 pushed ckpt_w163.pt,log_w163.txt +20:47:46 waited 76s for global r=693 +20:47:53 global r=693 load 4.283 -> 4.282 +20:48:58 step 169820 loss=3.0775 lr=2.27e-05 tok=22.346B dt=2792 wait=210 +20:50:02 step 169840 loss=3.0158 lr=2.27e-05 tok=22.349B dt=2856 wait=210 +20:51:05 step 169860 loss=2.9588 lr=2.27e-05 tok=22.352B dt=2920 wait=210 +20:52:10 step 169880 loss=2.8949 lr=2.27e-05 tok=22.354B dt=2984 wait=210 +20:53:13 step 169900 loss=3.0037 lr=2.26e-05 tok=22.357B dt=3048 wait=210 +20:54:16 step 169920 loss=3.0711 lr=2.26e-05 tok=22.360B dt=3111 wait=211 +20:55:20 step 169940 loss=2.9900 lr=2.26e-05 tok=22.362B dt=3174 wait=211 +20:56:03 pushed ckpt_w163.pt,log_w163.txt +20:57:33 waited 91s for global r=694 +20:57:41 global r=694 load 4.278 -> 4.282 +20:58:14 step 169960 loss=3.0580 lr=2.26e-05 tok=22.365B dt=3349 wait=211 +20:59:18 step 169980 loss=2.9406 lr=2.26e-05 tok=22.367B dt=3413 wait=211 +21:00:22 step 170000 loss=3.0068 lr=2.26e-05 tok=22.370B dt=3477 wait=211 +21:01:26 step 170020 loss=3.0736 lr=2.26e-05 tok=22.373B dt=3541 wait=211 +21:02:30 step 170040 loss=2.9927 lr=2.26e-05 tok=22.375B dt=3605 wait=211 +21:03:35 step 170060 loss=3.1827 lr=2.26e-05 tok=22.378B dt=3669 wait=212 +21:04:39 step 170080 loss=3.1050 lr=2.26e-05 tok=22.381B dt=3734 wait=212 +21:05:43 step 170100 loss=3.0639 lr=2.26e-05 tok=22.383B dt=3798 wait=212 +21:05:54 pushed ckpt_w163.pt,log_w163.txt +21:07:09 waited 76s for global r=695 +21:07:18 global r=695 load 4.285 -> 4.282 +21:08:24 step 170120 loss=2.9700 lr=2.25e-05 tok=22.386B dt=3959 wait=212 +21:09:28 step 170140 loss=3.0825 lr=2.25e-05 tok=22.388B dt=4023 wait=212 +21:10:33 step 170160 loss=3.0514 lr=2.25e-05 tok=22.391B dt=4087 wait=212 +21:11:37 step 170180 loss=3.0720 lr=2.25e-05 tok=22.394B dt=4151 wait=212 +21:12:41 step 170200 loss=3.0627 lr=2.25e-05 tok=22.396B dt=4216 wait=213 +21:13:46 step 170220 loss=3.0825 lr=2.25e-05 tok=22.399B dt=4281 wait=213 +21:14:51 step 170240 loss=3.0433 lr=2.25e-05 tok=22.402B dt=4346 wait=213 +21:15:38 pushed ckpt_w163.pt,log_w163.txt +21:19:14 step 170260 loss=3.2980 lr=2.25e-05 tok=22.404B dt=4609 wait=213 +21:19:20 global r=696 load 4.280 -> 4.281 +21:19:52 step 170260 loss=3.2244 lr=2.25e-05 tok=22.405B dt=4647 wait=213 +21:20:57 step 170280 loss=3.1283 lr=2.25e-05 tok=22.408B dt=4712 wait=213 +21:22:02 step 170300 loss=3.0217 lr=2.25e-05 tok=22.411B dt=4777 wait=213 +21:23:07 step 170320 loss=3.1638 lr=2.24e-05 tok=22.413B dt=4841 wait=214 +21:24:11 step 170340 loss=3.0815 lr=2.24e-05 tok=22.416B dt=4906 wait=214 +21:25:16 step 170360 loss=3.2891 lr=2.24e-05 tok=22.419B dt=4971 wait=214 +21:26:21 step 170380 loss=3.1757 lr=2.24e-05 tok=22.421B dt=5036 wait=214 +21:27:25 step 170400 loss=3.1835 lr=2.24e-05 tok=22.424B dt=5100 wait=214 +21:27:36 pushed ckpt_w163.pt,log_w163.txt +21:31:17 global r=697 load 4.286 -> 4.282 +21:32:22 step 170420 loss=3.0876 lr=2.24e-05 tok=22.428B dt=5397 wait=214 +21:33:26 step 170440 loss=3.3217 lr=2.24e-05 tok=22.430B dt=5461 wait=215 +21:34:30 step 170460 loss=3.1284 lr=2.24e-05 tok=22.433B dt=5525 wait=215 +21:35:34 step 170480 loss=3.1224 lr=2.24e-05 tok=22.436B dt=5589 wait=215 +21:36:38 step 170500 loss=3.0499 lr=2.24e-05 tok=22.438B dt=5653 wait=215 +21:37:42 step 170520 loss=3.1232 lr=2.23e-05 tok=22.441B dt=5717 wait=215 +21:38:47 step 170540 loss=3.0271 lr=2.23e-05 tok=22.443B dt=5782 wait=215 +21:39:32 pushed ckpt_w163.pt,log_w163.txt +21:42:34 waited 181s for global r=698 +21:42:40 global r=698 load 4.289 -> 4.285 +21:43:14 step 170560 loss=3.0152 lr=2.23e-05 tok=22.446B dt=6048 wait=215 +21:44:17 step 170580 loss=2.9720 lr=2.23e-05 tok=22.449B dt=6112 wait=216 +21:45:22 step 170600 loss=3.0974 lr=2.23e-05 tok=22.451B dt=6176 wait=216 +21:46:27 step 170620 loss=3.3973 lr=2.23e-05 tok=22.454B dt=6242 wait=216 +21:47:33 step 170640 loss=3.4501 lr=2.23e-05 tok=22.457B dt=6307 wait=216 +21:48:38 step 170660 loss=3.3941 lr=2.23e-05 tok=22.459B dt=6373 wait=216 +21:49:43 step 170680 loss=3.2650 lr=2.23e-05 tok=22.462B dt=6437 wait=216 +21:50:47 step 170700 loss=3.3253 lr=2.23e-05 tok=22.464B dt=6502 wait=216 +21:50:58 pushed ckpt_w163.pt,log_w163.txt +21:54:00 waited 182s for global r=699 +21:54:09 global r=699 load 4.288 -> 4.282 +21:55:15 step 170720 loss=3.1668 lr=2.23e-05 tok=22.467B dt=6769 wait=217 +21:56:20 step 170740 loss=3.2668 lr=2.22e-05 tok=22.470B dt=6835 wait=217 +21:57:24 step 170760 loss=3.2651 lr=2.22e-05 tok=22.472B dt=6899 wait=217 +21:58:30 step 170780 loss=3.2250 lr=2.22e-05 tok=22.475B dt=6964 wait=217 +21:59:34 step 170800 loss=3.1189 lr=2.22e-05 tok=22.478B dt=7029 wait=217 +22:00:39 step 170820 loss=3.0238 lr=2.22e-05 tok=22.480B dt=7094 wait=217 +22:01:43 step 170840 loss=2.9281 lr=2.22e-05 tok=22.483B dt=7158 wait=218 +22:02:28 pushed ckpt_w163.pt,log_w163.txt +22:06:04 step 170860 loss=3.1669 lr=2.22e-05 tok=22.485B dt=7418 wait=218 +22:06:11 global r=700 load 4.285 -> 4.283 +22:06:45 step 170860 loss=3.0849 lr=2.22e-05 tok=22.487B dt=7459 wait=218 +22:07:50 step 170880 loss=2.8375 lr=2.22e-05 tok=22.489B dt=7524 wait=218 +22:08:55 step 170900 loss=2.8846 lr=2.22e-05 tok=22.492B dt=7590 wait=218 +22:10:01 step 170920 loss=2.9156 lr=2.22e-05 tok=22.495B dt=7656 wait=218 +22:11:07 step 170940 loss=2.7760 lr=2.21e-05 tok=22.497B dt=7721 wait=218 +22:12:13 step 170960 loss=2.8627 lr=2.21e-05 tok=22.500B dt=7787 wait=218 +22:13:18 step 170980 loss=2.8335 lr=2.21e-05 tok=22.502B dt=7852 wait=219 +22:14:23 step 171000 loss=2.8294 lr=2.21e-05 tok=22.505B dt=7918 wait=219 +22:14:41 pushed ckpt_w163.pt,log_w163.txt +22:17:27 waited 167s for global r=701 +22:17:35 global r=701 load 4.295 -> 4.287 +22:18:42 step 171020 loss=2.8267 lr=2.21e-05 tok=22.508B dt=8176 wait=219 +22:19:47 step 171040 loss=2.7980 lr=2.21e-05 tok=22.510B dt=8242 wait=219 +22:20:52 step 171060 loss=2.8461 lr=2.21e-05 tok=22.513B dt=8307 wait=219 +22:21:58 step 171080 loss=2.7948 lr=2.21e-05 tok=22.516B dt=8372 wait=219 +22:23:03 step 171100 loss=2.9020 lr=2.21e-05 tok=22.518B dt=8437 wait=220 +22:24:08 step 171120 loss=2.9556 lr=2.21e-05 tok=22.521B dt=8503 wait=220 +22:25:14 step 171140 loss=2.7993 lr=2.21e-05 tok=22.523B dt=8569 wait=220 +22:26:00 pushed ckpt_w163.pt,log_w163.txt +22:29:02 waited 181s for global r=702 +22:29:12 global r=702 load 4.291 -> 4.284 +22:29:47 step 171160 loss=2.9369 lr=2.20e-05 tok=22.526B dt=8842 wait=220 +22:30:52 step 171180 loss=2.9630 lr=2.20e-05 tok=22.529B dt=8907 wait=220 +22:31:58 step 171200 loss=2.9629 lr=2.20e-05 tok=22.531B dt=8973 wait=220 +22:33:03 step 171220 loss=3.0433 lr=2.20e-05 tok=22.534B dt=9038 wait=221 +22:34:10 step 171240 loss=3.0750 lr=2.20e-05 tok=22.537B dt=9104 wait=221 +22:35:15 step 171260 loss=2.8742 lr=2.20e-05 tok=22.539B dt=9169 wait=221 +22:36:21 step 171280 loss=2.8982 lr=2.20e-05 tok=22.542B dt=9235 wait=221 +22:37:26 step 171300 loss=2.8581 lr=2.20e-05 tok=22.544B dt=9301 wait=221 +22:37:39 pushed ckpt_w163.pt,log_w163.txt +22:40:41 waited 182s for global r=703 +22:40:46 global r=703 load 4.286 -> 4.279 +22:41:53 step 171320 loss=2.9236 lr=2.20e-05 tok=22.547B dt=9568 wait=221 +22:42:59 step 171340 loss=2.9478 lr=2.20e-05 tok=22.550B dt=9633 wait=222 +22:44:05 step 171360 loss=2.8421 lr=2.19e-05 tok=22.552B dt=9699 wait=222 +22:45:11 step 171380 loss=2.7620 lr=2.19e-05 tok=22.555B dt=9766 wait=222 +22:46:16 step 171400 loss=2.8343 lr=2.19e-05 tok=22.557B dt=9831 wait=222 +22:47:22 step 171420 loss=2.9272 lr=2.19e-05 tok=22.560B dt=9897 wait=222 +22:48:27 step 171440 loss=2.8034 lr=2.19e-05 tok=22.563B dt=9961 wait=222 +22:49:13 pushed ckpt_w163.pt,log_w163.txt +22:52:14 waited 181s for global r=704 +22:52:22 global r=704 load 4.295 -> 4.282 +22:52:56 step 171460 loss=3.0429 lr=2.19e-05 tok=22.565B dt=10230 wait=222 +22:54:01 step 171480 loss=3.0277 lr=2.19e-05 tok=22.568B dt=10296 wait=223 +22:55:05 step 171500 loss=3.0429 lr=2.19e-05 tok=22.571B dt=10360 wait=223 +22:56:11 step 171520 loss=3.1686 lr=2.19e-05 tok=22.573B dt=10425 wait=223 +22:57:15 step 171540 loss=3.0286 lr=2.19e-05 tok=22.576B dt=10490 wait=223 +22:58:19 step 171560 loss=3.1393 lr=2.19e-05 tok=22.578B dt=10554 wait=223 +22:59:23 step 171580 loss=3.1433 lr=2.18e-05 tok=22.581B dt=10618 wait=223 +23:00:29 step 171600 loss=3.1657 lr=2.18e-05 tok=22.584B dt=10683 wait=224 +23:00:39 pushed ckpt_w163.pt,log_w163.txt +23:03:41 waited 181s for global r=705 +23:03:46 global r=705 load 4.281 -> 4.282 +23:04:52 step 171620 loss=3.1986 lr=2.18e-05 tok=22.586B dt=10947 wait=224 +23:05:57 step 171640 loss=3.2028 lr=2.18e-05 tok=22.589B dt=11012 wait=224 +23:07:01 step 171660 loss=3.0991 lr=2.18e-05 tok=22.592B dt=11076 wait=224 +23:08:05 step 171680 loss=3.0369 lr=2.18e-05 tok=22.594B dt=11140 wait=224 +23:09:09 step 171700 loss=3.0132 lr=2.18e-05 tok=22.597B dt=11204 wait=224 +23:10:13 step 171720 loss=3.1011 lr=2.18e-05 tok=22.599B dt=11267 wait=225 +23:11:16 step 171740 loss=2.8839 lr=2.18e-05 tok=22.602B dt=11331 wait=225 +23:11:58 pushed ckpt_w163.pt,log_w163.txt +23:15:33 step 171760 loss=3.0644 lr=2.18e-05 tok=22.605B dt=11588 wait=225 +23:15:39 global r=706 load 4.269 -> 4.279 +23:16:11 step 171760 loss=3.0121 lr=2.18e-05 tok=22.606B dt=11626 wait=225 +23:17:14 step 171780 loss=2.9444 lr=2.18e-05 tok=22.609B dt=11689 wait=225 +23:18:17 step 171800 loss=3.0037 lr=2.17e-05 tok=22.611B dt=11751 wait=225 +23:19:20 step 171820 loss=3.0265 lr=2.17e-05 tok=22.614B dt=11814 wait=226 +23:20:23 step 171840 loss=2.9856 lr=2.17e-05 tok=22.616B dt=11878 wait=226 +23:21:28 step 171860 loss=3.1324 lr=2.17e-05 tok=22.619B dt=11942 wait=226 +23:22:32 step 171880 loss=3.1077 lr=2.17e-05 tok=22.622B dt=12006 wait=226 +23:23:35 step 171900 loss=3.0808 lr=2.17e-05 tok=22.624B dt=12070 wait=226 +23:23:46 pushed ckpt_w163.pt,log_w163.txt +23:27:28 global r=707 load 4.280 -> 4.277 +23:28:33 step 171920 loss=2.9644 lr=2.17e-05 tok=22.628B dt=12367 wait=226 +23:29:36 step 171940 loss=3.0921 lr=2.17e-05 tok=22.631B dt=12431 wait=227 +23:30:40 step 171960 loss=3.0869 lr=2.17e-05 tok=22.634B dt=12494 wait=227 +23:31:44 step 171980 loss=3.0577 lr=2.17e-05 tok=22.636B dt=12558 wait=227 +23:32:47 step 172000 loss=3.1116 lr=2.17e-05 tok=22.639B dt=12622 wait=227 +23:33:51 step 172020 loss=3.1968 lr=2.16e-05 tok=22.641B dt=12685 wait=227 +23:34:55 step 172040 loss=3.0939 lr=2.16e-05 tok=22.644B dt=12750 wait=227 +23:35:38 pushed ckpt_w163.pt,log_w163.txt +23:38:39 waited 181s for global r=708 +23:38:44 global r=708 load 4.284 -> 4.281 +23:39:18 step 172060 loss=3.1318 lr=2.16e-05 tok=22.647B dt=13013 wait=228 +23:40:21 step 172080 loss=3.0544 lr=2.16e-05 tok=22.649B dt=13076 wait=228 +23:41:25 step 172100 loss=3.0182 lr=2.16e-05 tok=22.652B dt=13140 wait=228 +23:42:29 step 172120 loss=2.9577 lr=2.16e-05 tok=22.654B dt=13204 wait=228 +23:43:33 step 172140 loss=2.9736 lr=2.16e-05 tok=22.657B dt=13267 wait=228 +23:44:36 step 172160 loss=3.0378 lr=2.16e-05 tok=22.660B dt=13330 wait=228 +23:45:39 step 172180 loss=3.0166 lr=2.16e-05 tok=22.662B dt=13393 wait=228 +23:46:42 step 172200 loss=3.0210 lr=2.16e-05 tok=22.665B dt=13457 wait=229 +23:46:54 pushed ckpt_w163.pt,log_w163.txt +23:49:55 waited 181s for global r=709 +23:50:03 global r=709 load 4.282 -> 4.280 +23:51:07 step 172220 loss=3.0171 lr=2.16e-05 tok=22.668B dt=13721 wait=229 +23:52:10 step 172240 loss=3.0522 lr=2.15e-05 tok=22.670B dt=13784 wait=229 +23:53:13 step 172260 loss=2.9654 lr=2.15e-05 tok=22.673B dt=13848 wait=229 +23:54:17 step 172280 loss=2.9628 lr=2.15e-05 tok=22.675B dt=13912 wait=229 +23:55:20 step 172300 loss=3.0822 lr=2.15e-05 tok=22.678B dt=13975 wait=229 +23:56:23 step 172320 loss=3.0419 lr=2.15e-05 tok=22.681B dt=14038 wait=229 +23:57:27 step 172340 loss=2.8900 lr=2.15e-05 tok=22.683B dt=14102 wait=230 +23:58:10 pushed ckpt_w163.pt,log_w163.txt +00:01:44 step 172360 loss=3.0586 lr=2.15e-05 tok=22.686B dt=14359 wait=230 +00:01:51 global r=710 load 4.279 -> 4.280 +00:02:23 step 172360 loss=3.1169 lr=2.15e-05 tok=22.687B dt=14397 wait=230 +00:03:25 step 172380 loss=2.9621 lr=2.15e-05 tok=22.690B dt=14460 wait=230 +00:04:30 step 172400 loss=2.7040 lr=2.15e-05 tok=22.692B dt=14524 wait=230 +00:05:33 step 172420 loss=2.9265 lr=2.15e-05 tok=22.695B dt=14587 wait=230 +00:06:36 step 172440 loss=2.9054 lr=2.15e-05 tok=22.698B dt=14651 wait=230 +00:07:39 step 172460 loss=3.0200 lr=2.14e-05 tok=22.700B dt=14714 wait=231 +00:08:42 step 172480 loss=2.9656 lr=2.14e-05 tok=22.703B dt=14777 wait=231 +00:09:47 step 172500 loss=2.9588 lr=2.14e-05 tok=22.706B dt=14842 wait=231 diff --git a/log_w164.txt b/log_w164.txt new file mode 100644 index 0000000000000000000000000000000000000000..12afe69c6db55299f7a4c7a5edfd372a8904ecb0 --- /dev/null +++ b/log_w164.txt @@ -0,0 +1,238 @@ +launch +20:00:17 name=w164 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 111/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:00:23 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:00:23 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +20:00:33 params 1.08B +20:00:34 resumed step=169050 seen=22.26B round=688 +20:00:34 stream salt=169050 +20:02:18 blocks compiled +20:02:18 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +20:06:19 step 169051 loss=2.6434 lr=2.31e-05 tok=22.260B dt=241 wait=214 +20:06:35 step 169052 loss=2.8159 lr=2.31e-05 tok=22.260B dt=257 wait=214 +20:07:00 step 169060 loss=3.3247 lr=2.31e-05 tok=22.261B dt=283 wait=214 +20:08:06 step 169080 loss=3.2591 lr=2.31e-05 tok=22.264B dt=348 wait=214 +20:09:12 step 169100 loss=3.1125 lr=2.30e-05 tok=22.267B dt=414 wait=214 +20:09:25 global r=689 load 4.286 -> 4.280 +20:10:30 step 169220 loss=3.1528 lr=2.30e-05 tok=22.269B dt=492 wait=215 +20:11:34 step 169240 loss=3.1397 lr=2.30e-05 tok=22.272B dt=557 wait=215 +20:12:39 step 169260 loss=3.1107 lr=2.30e-05 tok=22.274B dt=621 wait=215 +20:13:43 step 169280 loss=3.1215 lr=2.30e-05 tok=22.277B dt=686 wait=215 +20:14:48 step 169300 loss=3.1006 lr=2.29e-05 tok=22.280B dt=750 wait=215 +20:15:53 step 169320 loss=3.0547 lr=2.29e-05 tok=22.282B dt=816 wait=215 +20:16:57 step 169340 loss=3.0188 lr=2.29e-05 tok=22.285B dt=880 wait=215 +20:17:44 pushed ckpt_w164.pt,log_w164.txt +20:18:59 waited 76s for global r=690 +20:19:07 global r=690 load 4.295 -> 4.284 +20:19:41 step 169360 loss=3.0943 lr=2.29e-05 tok=22.287B dt=1044 wait=216 +20:20:46 step 169380 loss=3.0566 lr=2.29e-05 tok=22.290B dt=1108 wait=216 +20:21:50 step 169400 loss=2.9437 lr=2.29e-05 tok=22.293B dt=1173 wait=216 +20:22:55 step 169420 loss=3.0882 lr=2.29e-05 tok=22.295B dt=1237 wait=216 +20:23:59 step 169440 loss=2.9789 lr=2.29e-05 tok=22.298B dt=1302 wait=216 +20:25:03 step 169460 loss=3.0472 lr=2.29e-05 tok=22.301B dt=1365 wait=216 +20:26:07 step 169480 loss=3.0045 lr=2.29e-05 tok=22.303B dt=1430 wait=216 +20:27:12 step 169500 loss=3.1242 lr=2.28e-05 tok=22.306B dt=1494 wait=216 +20:27:24 pushed ckpt_w164.pt,log_w164.txt +20:28:25 waited 61s for global r=691 +20:28:33 global r=691 load 4.287 -> 4.287 +20:29:40 step 169520 loss=2.7854 lr=2.28e-05 tok=22.308B dt=1642 wait=217 +20:30:44 step 169540 loss=3.0170 lr=2.28e-05 tok=22.311B dt=1706 wait=217 +20:31:47 step 169560 loss=3.1002 lr=2.28e-05 tok=22.314B dt=1770 wait=217 +20:32:52 step 169580 loss=2.8538 lr=2.28e-05 tok=22.316B dt=1834 wait=217 +20:33:56 step 169600 loss=2.9465 lr=2.28e-05 tok=22.319B dt=1899 wait=217 +20:35:00 step 169620 loss=2.9914 lr=2.28e-05 tok=22.322B dt=1963 wait=217 +20:36:05 step 169640 loss=2.9328 lr=2.28e-05 tok=22.324B dt=2027 wait=217 +20:36:50 pushed ckpt_w164.pt,log_w164.txt +20:38:06 waited 76s for global r=692 +20:38:13 global r=692 load 4.287 -> 4.284 +20:38:47 step 169660 loss=2.9517 lr=2.28e-05 tok=22.327B dt=2190 wait=218 +20:39:52 step 169680 loss=2.6859 lr=2.28e-05 tok=22.329B dt=2254 wait=218 +20:40:57 step 169700 loss=2.9416 lr=2.27e-05 tok=22.332B dt=2319 wait=218 +20:42:01 step 169720 loss=3.0372 lr=2.27e-05 tok=22.335B dt=2383 wait=218 +20:43:06 step 169740 loss=2.9710 lr=2.27e-05 tok=22.337B dt=2448 wait=218 +20:44:09 step 169760 loss=3.0456 lr=2.27e-05 tok=22.340B dt=2512 wait=218 +20:45:13 step 169780 loss=2.9422 lr=2.27e-05 tok=22.343B dt=2575 wait=218 +20:46:17 step 169800 loss=2.9273 lr=2.27e-05 tok=22.345B dt=2640 wait=219 +20:46:30 pushed ckpt_w164.pt,log_w164.txt +20:47:46 waited 76s for global r=693 +20:47:51 global r=693 load 4.285 -> 4.282 +20:48:57 step 169820 loss=2.9386 lr=2.27e-05 tok=22.348B dt=2799 wait=219 +20:50:01 step 169840 loss=3.0051 lr=2.27e-05 tok=22.350B dt=2863 wait=219 +20:51:05 step 169860 loss=2.9262 lr=2.27e-05 tok=22.353B dt=2928 wait=219 +20:52:10 step 169880 loss=2.9546 lr=2.27e-05 tok=22.356B dt=2992 wait=219 +20:53:14 step 169900 loss=2.9877 lr=2.26e-05 tok=22.358B dt=3056 wait=219 +20:54:18 step 169920 loss=3.0174 lr=2.26e-05 tok=22.361B dt=3121 wait=219 +20:55:23 step 169940 loss=3.0241 lr=2.26e-05 tok=22.364B dt=3185 wait=220 +20:56:09 pushed ckpt_w164.pt,log_w164.txt +20:57:25 waited 76s for global r=694 +20:57:33 global r=694 load 4.285 -> 4.282 +20:58:08 step 169960 loss=3.0630 lr=2.26e-05 tok=22.366B dt=3350 wait=220 +20:59:12 step 169980 loss=2.9755 lr=2.26e-05 tok=22.369B dt=3414 wait=220 +21:00:16 step 170000 loss=2.8759 lr=2.26e-05 tok=22.371B dt=3479 wait=220 +21:01:20 step 170020 loss=2.9175 lr=2.26e-05 tok=22.374B dt=3543 wait=220 +21:02:25 step 170040 loss=2.8910 lr=2.26e-05 tok=22.377B dt=3607 wait=220 +21:03:29 step 170060 loss=2.7938 lr=2.26e-05 tok=22.379B dt=3671 wait=221 +21:04:32 step 170080 loss=2.8137 lr=2.26e-05 tok=22.382B dt=3735 wait=221 +21:05:36 step 170100 loss=2.8347 lr=2.26e-05 tok=22.384B dt=3799 wait=221 +21:05:48 pushed ckpt_w164.pt,log_w164.txt +21:07:19 waited 91s for global r=695 +21:07:27 global r=695 load 4.283 -> 4.282 +21:08:32 step 170120 loss=2.8367 lr=2.25e-05 tok=22.387B dt=3974 wait=221 +21:09:36 step 170140 loss=2.8568 lr=2.25e-05 tok=22.390B dt=4038 wait=221 +21:10:39 step 170160 loss=2.8871 lr=2.25e-05 tok=22.392B dt=4101 wait=221 +21:11:42 step 170180 loss=2.8873 lr=2.25e-05 tok=22.395B dt=4165 wait=221 +21:12:46 step 170200 loss=2.8751 lr=2.25e-05 tok=22.398B dt=4228 wait=222 +21:13:50 step 170220 loss=2.9717 lr=2.25e-05 tok=22.400B dt=4292 wait=222 +21:14:53 step 170240 loss=2.8411 lr=2.25e-05 tok=22.403B dt=4355 wait=222 +21:15:40 pushed ckpt_w164.pt,log_w164.txt +21:19:15 step 170260 loss=2.9653 lr=2.25e-05 tok=22.405B dt=4618 wait=222 +21:19:21 global r=696 load 4.283 -> 4.281 +21:19:53 step 170260 loss=2.8945 lr=2.25e-05 tok=22.407B dt=4655 wait=222 +21:20:56 step 170280 loss=2.8318 lr=2.25e-05 tok=22.409B dt=4719 wait=222 +21:22:00 step 170300 loss=2.8791 lr=2.25e-05 tok=22.412B dt=4783 wait=222 +21:23:05 step 170320 loss=3.0884 lr=2.24e-05 tok=22.415B dt=4847 wait=223 +21:24:08 step 170340 loss=3.0288 lr=2.24e-05 tok=22.417B dt=4911 wait=223 +21:25:12 step 170360 loss=3.0920 lr=2.24e-05 tok=22.420B dt=4975 wait=223 +21:26:16 step 170380 loss=3.0861 lr=2.24e-05 tok=22.422B dt=5038 wait=223 +21:27:20 step 170400 loss=3.0023 lr=2.24e-05 tok=22.425B dt=5103 wait=223 +21:27:33 pushed ckpt_w164.pt,log_w164.txt +21:31:15 global r=697 load 4.276 -> 4.282 +21:32:20 step 170420 loss=3.0103 lr=2.24e-05 tok=22.429B dt=5402 wait=223 +21:33:23 step 170440 loss=3.0033 lr=2.24e-05 tok=22.432B dt=5466 wait=224 +21:34:27 step 170460 loss=2.9269 lr=2.24e-05 tok=22.434B dt=5529 wait=224 +21:35:30 step 170480 loss=2.9460 lr=2.24e-05 tok=22.437B dt=5592 wait=224 +21:36:33 step 170500 loss=2.9695 lr=2.24e-05 tok=22.440B dt=5655 wait=224 +21:37:37 step 170520 loss=3.0009 lr=2.23e-05 tok=22.442B dt=5719 wait=224 +21:38:41 step 170540 loss=2.7737 lr=2.23e-05 tok=22.445B dt=5783 wait=224 +21:39:25 pushed ckpt_w164.pt,log_w164.txt +21:42:26 waited 181s for global r=698 +21:42:34 global r=698 load 4.279 -> 4.285 +21:43:08 step 170560 loss=2.9287 lr=2.23e-05 tok=22.447B dt=6050 wait=225 +21:44:10 step 170580 loss=2.9266 lr=2.23e-05 tok=22.450B dt=6113 wait=225 +21:45:14 step 170600 loss=2.9391 lr=2.23e-05 tok=22.453B dt=6176 wait=225 +21:46:17 step 170620 loss=3.0018 lr=2.23e-05 tok=22.455B dt=6239 wait=225 +21:47:21 step 170640 loss=2.9250 lr=2.23e-05 tok=22.458B dt=6303 wait=225 +21:48:24 step 170660 loss=2.9646 lr=2.23e-05 tok=22.460B dt=6366 wait=225 +21:49:27 step 170680 loss=2.9223 lr=2.23e-05 tok=22.463B dt=6429 wait=225 +21:50:30 step 170700 loss=3.0004 lr=2.23e-05 tok=22.466B dt=6492 wait=226 +21:50:43 pushed ckpt_w164.pt,log_w164.txt +21:54:27 global r=699 load 4.283 -> 4.282 +21:55:30 step 170720 loss=2.9525 lr=2.23e-05 tok=22.470B dt=6793 wait=226 +21:56:33 step 170740 loss=3.0205 lr=2.22e-05 tok=22.472B dt=6855 wait=226 +21:57:37 step 170760 loss=2.8527 lr=2.22e-05 tok=22.475B dt=6919 wait=226 +21:58:39 step 170780 loss=2.9425 lr=2.22e-05 tok=22.478B dt=6982 wait=226 +21:59:43 step 170800 loss=2.8535 lr=2.22e-05 tok=22.480B dt=7045 wait=226 +22:00:46 step 170820 loss=3.0656 lr=2.22e-05 tok=22.483B dt=7109 wait=227 +22:01:50 step 170840 loss=2.9024 lr=2.22e-05 tok=22.485B dt=7172 wait=227 +22:02:39 pushed ckpt_w164.pt,log_w164.txt +22:05:40 waited 182s for global r=700 +22:05:48 global r=700 load 4.281 -> 4.283 +22:06:22 step 170860 loss=2.9525 lr=2.22e-05 tok=22.488B dt=7444 wait=227 +22:07:25 step 170880 loss=3.0460 lr=2.22e-05 tok=22.491B dt=7507 wait=227 +22:08:30 step 170900 loss=2.6181 lr=2.22e-05 tok=22.493B dt=7572 wait=227 +22:09:34 step 170920 loss=2.9146 lr=2.22e-05 tok=22.496B dt=7637 wait=227 +22:10:38 step 170940 loss=2.9695 lr=2.21e-05 tok=22.499B dt=7701 wait=228 +22:11:42 step 170960 loss=3.0722 lr=2.21e-05 tok=22.501B dt=7764 wait=228 +22:12:45 step 170980 loss=3.0116 lr=2.21e-05 tok=22.504B dt=7827 wait=228 +22:13:49 step 171000 loss=2.9528 lr=2.21e-05 tok=22.506B dt=7891 wait=228 +22:14:06 pushed ckpt_w164.pt,log_w164.txt +22:17:50 global r=701 load 4.282 -> 4.287 +22:18:55 step 171020 loss=2.9338 lr=2.21e-05 tok=22.510B dt=8197 wait=228 +22:19:59 step 171040 loss=3.0482 lr=2.21e-05 tok=22.513B dt=8261 wait=228 +22:21:03 step 171060 loss=2.9951 lr=2.21e-05 tok=22.516B dt=8325 wait=228 +22:22:06 step 171080 loss=2.9298 lr=2.21e-05 tok=22.518B dt=8389 wait=229 +22:23:10 step 171100 loss=3.0352 lr=2.21e-05 tok=22.521B dt=8452 wait=229 +22:24:14 step 171120 loss=3.0651 lr=2.21e-05 tok=22.523B dt=8516 wait=229 +22:25:18 step 171140 loss=2.8583 lr=2.21e-05 tok=22.526B dt=8580 wait=229 +22:26:06 pushed ckpt_w164.pt,log_w164.txt +22:29:07 waited 181s for global r=702 +22:29:15 global r=702 load 4.278 -> 4.284 +22:29:49 step 171160 loss=3.0708 lr=2.20e-05 tok=22.529B dt=8851 wait=229 +22:30:52 step 171180 loss=2.9438 lr=2.20e-05 tok=22.531B dt=8915 wait=229 +22:31:56 step 171200 loss=2.9788 lr=2.20e-05 tok=22.534B dt=8979 wait=230 +22:33:00 step 171220 loss=3.0413 lr=2.20e-05 tok=22.537B dt=9043 wait=230 +22:34:04 step 171240 loss=2.9970 lr=2.20e-05 tok=22.539B dt=9106 wait=230 +22:35:08 step 171260 loss=3.0544 lr=2.20e-05 tok=22.542B dt=9171 wait=230 +22:36:12 step 171280 loss=3.0007 lr=2.20e-05 tok=22.544B dt=9235 wait=230 +22:37:17 step 171300 loss=3.1168 lr=2.20e-05 tok=22.547B dt=9299 wait=230 +22:37:28 pushed ckpt_w164.pt,log_w164.txt +22:41:12 global r=703 load 4.279 -> 4.279 +22:42:15 step 171320 loss=3.0651 lr=2.20e-05 tok=22.551B dt=9598 wait=230 +22:43:19 step 171340 loss=3.1079 lr=2.20e-05 tok=22.554B dt=9661 wait=231 +22:44:23 step 171360 loss=3.0119 lr=2.19e-05 tok=22.556B dt=9725 wait=231 +22:45:26 step 171380 loss=2.9499 lr=2.19e-05 tok=22.559B dt=9788 wait=231 +22:46:30 step 171400 loss=2.9520 lr=2.19e-05 tok=22.561B dt=9852 wait=231 +22:47:33 step 171420 loss=3.0305 lr=2.19e-05 tok=22.564B dt=9916 wait=231 +22:48:37 step 171440 loss=2.9722 lr=2.19e-05 tok=22.567B dt=9979 wait=231 +22:49:25 pushed ckpt_w164.pt,log_w164.txt +22:52:11 waited 166s for global r=704 +22:52:18 global r=704 load 4.282 -> 4.282 +22:52:51 step 171460 loss=3.1096 lr=2.19e-05 tok=22.569B dt=10233 wait=231 +22:53:54 step 171480 loss=2.9486 lr=2.19e-05 tok=22.572B dt=10296 wait=232 +22:54:58 step 171500 loss=2.8899 lr=2.19e-05 tok=22.575B dt=10360 wait=232 +22:56:02 step 171520 loss=2.9648 lr=2.19e-05 tok=22.577B dt=10424 wait=232 +22:57:05 step 171540 loss=2.8802 lr=2.19e-05 tok=22.580B dt=10487 wait=232 +22:58:09 step 171560 loss=2.9324 lr=2.19e-05 tok=22.582B dt=10551 wait=232 +22:59:12 step 171580 loss=2.9393 lr=2.18e-05 tok=22.585B dt=10614 wait=232 +23:00:15 step 171600 loss=2.9233 lr=2.18e-05 tok=22.588B dt=10678 wait=233 +23:00:28 pushed ckpt_w164.pt,log_w164.txt +23:04:08 global r=705 load 4.291 -> 4.282 +23:05:11 step 171620 loss=2.9377 lr=2.18e-05 tok=22.592B dt=10974 wait=233 +23:06:15 step 171640 loss=2.9520 lr=2.18e-05 tok=22.594B dt=11037 wait=233 +23:07:18 step 171660 loss=2.8779 lr=2.18e-05 tok=22.597B dt=11101 wait=233 +23:08:21 step 171680 loss=2.8705 lr=2.18e-05 tok=22.599B dt=11164 wait=233 +23:09:24 step 171700 loss=2.8721 lr=2.18e-05 tok=22.602B dt=11226 wait=234 +23:10:28 step 171720 loss=2.9177 lr=2.18e-05 tok=22.605B dt=11290 wait=234 +23:11:32 step 171740 loss=2.7936 lr=2.18e-05 tok=22.607B dt=11354 wait=234 +23:12:16 pushed ckpt_w164.pt,log_w164.txt +23:15:50 step 171760 loss=2.9717 lr=2.18e-05 tok=22.610B dt=11613 wait=234 +23:15:57 global r=706 load 4.281 -> 4.279 +23:16:29 step 171760 loss=3.0209 lr=2.18e-05 tok=22.611B dt=11651 wait=234 +23:17:32 step 171780 loss=2.9004 lr=2.18e-05 tok=22.614B dt=11715 wait=234 +23:18:36 step 171800 loss=2.8860 lr=2.17e-05 tok=22.616B dt=11778 wait=234 +23:19:40 step 171820 loss=2.8730 lr=2.17e-05 tok=22.619B dt=11843 wait=235 +23:20:44 step 171840 loss=2.8937 lr=2.17e-05 tok=22.622B dt=11907 wait=235 +23:21:49 step 171860 loss=2.9491 lr=2.17e-05 tok=22.624B dt=11971 wait=235 +23:22:52 step 171880 loss=2.9193 lr=2.17e-05 tok=22.627B dt=12035 wait=235 +23:23:56 step 171900 loss=2.9643 lr=2.17e-05 tok=22.630B dt=12098 wait=235 +23:24:08 pushed ckpt_w164.pt,log_w164.txt +23:26:54 waited 166s for global r=707 +23:27:02 global r=707 load 4.279 -> 4.277 +23:28:07 step 171920 loss=2.9200 lr=2.17e-05 tok=22.632B dt=12349 wait=235 +23:29:11 step 171940 loss=2.9740 lr=2.17e-05 tok=22.635B dt=12413 wait=236 +23:30:14 step 171960 loss=2.9001 lr=2.17e-05 tok=22.637B dt=12477 wait=236 +23:31:18 step 171980 loss=2.8229 lr=2.17e-05 tok=22.640B dt=12541 wait=236 +23:32:22 step 172000 loss=2.8646 lr=2.17e-05 tok=22.643B dt=12604 wait=236 +23:33:25 step 172020 loss=2.9844 lr=2.16e-05 tok=22.645B dt=12668 wait=236 +23:34:30 step 172040 loss=2.7596 lr=2.16e-05 tok=22.648B dt=12732 wait=236 +23:35:16 pushed ckpt_w164.pt,log_w164.txt +23:38:51 step 172060 loss=3.0693 lr=2.16e-05 tok=22.651B dt=12993 wait=236 +23:38:58 global r=708 load 4.280 -> 4.281 +23:39:30 step 172060 loss=3.0242 lr=2.16e-05 tok=22.652B dt=13032 wait=237 +23:40:34 step 172080 loss=2.8360 lr=2.16e-05 tok=22.654B dt=13096 wait=237 +23:41:37 step 172100 loss=2.8755 lr=2.16e-05 tok=22.657B dt=13160 wait=237 +23:42:41 step 172120 loss=2.6781 lr=2.16e-05 tok=22.660B dt=13224 wait=237 +23:43:45 step 172140 loss=2.8760 lr=2.16e-05 tok=22.662B dt=13287 wait=237 +23:44:49 step 172160 loss=3.0033 lr=2.16e-05 tok=22.665B dt=13351 wait=237 +23:45:52 step 172180 loss=2.9420 lr=2.16e-05 tok=22.668B dt=13415 wait=237 +23:46:56 step 172200 loss=2.9039 lr=2.16e-05 tok=22.670B dt=13478 wait=238 +23:47:08 pushed ckpt_w164.pt,log_w164.txt +23:49:54 waited 166s for global r=709 +23:50:01 global r=709 load 4.278 -> 4.280 +23:51:05 step 172220 loss=2.8365 lr=2.16e-05 tok=22.673B dt=13728 wait=238 +23:52:09 step 172240 loss=2.8845 lr=2.15e-05 tok=22.675B dt=13791 wait=238 +23:53:12 step 172260 loss=2.9501 lr=2.15e-05 tok=22.678B dt=13854 wait=238 +23:54:15 step 172280 loss=2.9296 lr=2.15e-05 tok=22.681B dt=13917 wait=238 +23:55:18 step 172300 loss=2.8766 lr=2.15e-05 tok=22.683B dt=13981 wait=238 +23:56:22 step 172320 loss=2.9506 lr=2.15e-05 tok=22.686B dt=14044 wait=238 +23:57:25 step 172340 loss=2.8740 lr=2.15e-05 tok=22.689B dt=14107 wait=239 +23:58:14 pushed ckpt_w164.pt,log_w164.txt +00:01:50 step 172360 loss=3.0166 lr=2.15e-05 tok=22.691B dt=14372 wait=239 +00:01:55 global r=710 load 4.285 -> 4.280 +00:02:27 step 172360 loss=3.0133 lr=2.15e-05 tok=22.692B dt=14409 wait=239 +00:03:30 step 172380 loss=2.8679 lr=2.15e-05 tok=22.695B dt=14473 wait=239 +00:04:34 step 172400 loss=2.8744 lr=2.15e-05 tok=22.698B dt=14536 wait=239 +00:05:37 step 172420 loss=3.0435 lr=2.15e-05 tok=22.700B dt=14600 wait=239 +00:06:41 step 172440 loss=2.9350 lr=2.15e-05 tok=22.703B dt=14664 wait=239 +00:07:46 step 172460 loss=2.9596 lr=2.14e-05 tok=22.706B dt=14728 wait=240 +00:08:50 step 172480 loss=2.9291 lr=2.14e-05 tok=22.708B dt=14792 wait=240 +00:09:55 step 172500 loss=2.9810 lr=2.14e-05 tok=22.711B dt=14857 wait=240 diff --git a/log_w165.txt b/log_w165.txt new file mode 100644 index 0000000000000000000000000000000000000000..48e17d1a9dce997d2779047fd148346ba6ab6ead --- /dev/null +++ b/log_w165.txt @@ -0,0 +1,218 @@ +launch +20:00:18 name=w165 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 112/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:00:25 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:00:25 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +20:00:34 params 1.08B +20:00:36 resumed step=169050 seen=22.26B round=688 +20:00:36 stream salt=169050 +20:02:17 blocks compiled +20:02:17 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +20:05:59 step 169051 loss=3.5583 lr=2.31e-05 tok=22.260B dt=221 wait=194 +20:06:15 step 169052 loss=3.5429 lr=2.31e-05 tok=22.260B dt=237 wait=194 +20:06:40 step 169060 loss=3.2555 lr=2.31e-05 tok=22.261B dt=263 wait=194 +20:07:44 step 169080 loss=2.9032 lr=2.31e-05 tok=22.264B dt=327 wait=194 +20:08:49 step 169100 loss=2.8877 lr=2.30e-05 tok=22.267B dt=391 wait=194 +20:09:33 global r=689 load 4.278 -> 4.280 +20:10:36 step 169220 loss=2.9938 lr=2.30e-05 tok=22.270B dt=499 wait=195 +20:11:41 step 169240 loss=3.0824 lr=2.30e-05 tok=22.273B dt=563 wait=195 +20:12:45 step 169260 loss=3.0751 lr=2.30e-05 tok=22.276B dt=628 wait=195 +20:13:49 step 169280 loss=3.0131 lr=2.30e-05 tok=22.278B dt=692 wait=195 +20:14:53 step 169300 loss=3.0917 lr=2.29e-05 tok=22.281B dt=756 wait=195 +20:15:58 step 169320 loss=3.2724 lr=2.29e-05 tok=22.284B dt=821 wait=195 +20:17:02 step 169340 loss=3.1301 lr=2.29e-05 tok=22.286B dt=885 wait=196 +20:17:48 pushed ckpt_w165.pt,log_w165.txt +20:18:49 waited 61s for global r=690 +20:18:59 global r=690 load 4.274 -> 4.284 +20:19:33 step 169360 loss=3.0242 lr=2.29e-05 tok=22.289B dt=1036 wait=196 +20:20:38 step 169380 loss=3.0090 lr=2.29e-05 tok=22.291B dt=1100 wait=196 +20:21:42 step 169400 loss=2.9450 lr=2.29e-05 tok=22.294B dt=1164 wait=196 +20:22:46 step 169420 loss=2.9716 lr=2.29e-05 tok=22.297B dt=1229 wait=196 +20:23:49 step 169440 loss=2.9725 lr=2.29e-05 tok=22.299B dt=1292 wait=196 +20:24:53 step 169460 loss=3.0517 lr=2.29e-05 tok=22.302B dt=1356 wait=197 +20:25:58 step 169480 loss=2.9732 lr=2.29e-05 tok=22.305B dt=1420 wait=197 +20:27:02 step 169500 loss=2.9477 lr=2.28e-05 tok=22.307B dt=1484 wait=197 +20:27:14 pushed ckpt_w165.pt,log_w165.txt +20:28:29 waited 76s for global r=691 +20:28:40 global r=691 load 4.283 -> 4.287 +20:29:45 step 169520 loss=2.9180 lr=2.28e-05 tok=22.310B dt=1648 wait=197 +20:30:49 step 169540 loss=2.9495 lr=2.28e-05 tok=22.312B dt=1711 wait=197 +20:31:54 step 169560 loss=2.9691 lr=2.28e-05 tok=22.315B dt=1776 wait=197 +20:32:58 step 169580 loss=2.9228 lr=2.28e-05 tok=22.318B dt=1841 wait=197 +20:34:03 step 169600 loss=2.9375 lr=2.28e-05 tok=22.320B dt=1905 wait=197 +20:35:07 step 169620 loss=3.1235 lr=2.28e-05 tok=22.323B dt=1969 wait=198 +20:36:12 step 169640 loss=2.9337 lr=2.28e-05 tok=22.325B dt=2034 wait=198 +20:36:57 pushed ckpt_w165.pt,log_w165.txt +20:37:57 waited 60s for global r=692 +20:38:03 global r=692 load 4.287 -> 4.284 +20:38:37 step 169660 loss=3.0068 lr=2.28e-05 tok=22.328B dt=2180 wait=198 +20:39:42 step 169680 loss=3.0469 lr=2.28e-05 tok=22.331B dt=2245 wait=198 +20:40:47 step 169700 loss=2.9333 lr=2.27e-05 tok=22.333B dt=2309 wait=198 +20:41:51 step 169720 loss=2.9947 lr=2.27e-05 tok=22.336B dt=2374 wait=198 +20:42:55 step 169740 loss=2.9562 lr=2.27e-05 tok=22.339B dt=2438 wait=198 +20:44:00 step 169760 loss=2.7320 lr=2.27e-05 tok=22.341B dt=2502 wait=199 +20:45:04 step 169780 loss=2.9071 lr=2.27e-05 tok=22.344B dt=2566 wait=199 +20:46:07 step 169800 loss=2.9491 lr=2.27e-05 tok=22.346B dt=2630 wait=199 +20:46:19 pushed ckpt_w165.pt,log_w165.txt +20:47:50 waited 91s for global r=693 +20:47:57 global r=693 load 4.277 -> 4.282 +20:49:02 step 169820 loss=2.8777 lr=2.27e-05 tok=22.349B dt=2805 wait=199 +20:50:06 step 169840 loss=2.9548 lr=2.27e-05 tok=22.352B dt=2868 wait=199 +20:51:10 step 169860 loss=2.8415 lr=2.27e-05 tok=22.354B dt=2933 wait=199 +20:52:14 step 169880 loss=2.8411 lr=2.27e-05 tok=22.357B dt=2997 wait=200 +20:53:18 step 169900 loss=2.8552 lr=2.26e-05 tok=22.360B dt=3061 wait=200 +20:54:22 step 169920 loss=3.0115 lr=2.26e-05 tok=22.362B dt=3125 wait=200 +20:55:26 step 169940 loss=2.9656 lr=2.26e-05 tok=22.365B dt=3189 wait=200 +20:56:14 pushed ckpt_w165.pt,log_w165.txt +20:57:29 waited 76s for global r=694 +20:57:34 global r=694 load 4.280 -> 4.282 +20:58:07 step 169960 loss=2.9441 lr=2.26e-05 tok=22.367B dt=3350 wait=200 +20:59:10 step 169980 loss=2.9195 lr=2.26e-05 tok=22.370B dt=3413 wait=200 +21:00:14 step 170000 loss=2.8981 lr=2.26e-05 tok=22.373B dt=3477 wait=200 +21:01:18 step 170020 loss=2.9460 lr=2.26e-05 tok=22.375B dt=3541 wait=201 +21:02:22 step 170040 loss=2.9326 lr=2.26e-05 tok=22.378B dt=3604 wait=201 +21:03:26 step 170060 loss=3.0251 lr=2.26e-05 tok=22.381B dt=3668 wait=201 +21:04:29 step 170080 loss=3.0019 lr=2.26e-05 tok=22.383B dt=3732 wait=201 +21:05:34 step 170100 loss=2.9434 lr=2.26e-05 tok=22.386B dt=3796 wait=201 +21:05:46 pushed ckpt_w165.pt,log_w165.txt +21:07:16 waited 91s for global r=695 +21:07:22 global r=695 load 4.276 -> 4.282 +21:08:28 step 170120 loss=2.9115 lr=2.25e-05 tok=22.388B dt=3970 wait=201 +21:09:32 step 170140 loss=2.9103 lr=2.25e-05 tok=22.391B dt=4034 wait=202 +21:10:35 step 170160 loss=2.9209 lr=2.25e-05 tok=22.394B dt=4098 wait=202 +21:11:40 step 170180 loss=2.9987 lr=2.25e-05 tok=22.396B dt=4162 wait=202 +21:12:44 step 170200 loss=2.8822 lr=2.25e-05 tok=22.399B dt=4227 wait=202 +21:13:48 step 170220 loss=3.0148 lr=2.25e-05 tok=22.402B dt=4290 wait=202 +21:14:52 step 170240 loss=2.9086 lr=2.25e-05 tok=22.404B dt=4354 wait=202 +21:15:53 pushed ckpt_w165.pt,log_w165.txt +21:18:55 waited 181s for global r=696 +21:19:02 global r=696 load 4.287 -> 4.281 +21:19:44 step 170260 loss=2.9733 lr=2.25e-05 tok=22.407B dt=4646 wait=202 +21:21:37 step 170280 loss=2.8090 lr=2.25e-05 tok=22.409B dt=4759 wait=203 +21:23:50 step 170300 loss=2.9612 lr=2.25e-05 tok=22.412B dt=4893 wait=203 +21:25:38 step 170320 loss=3.0976 lr=2.24e-05 tok=22.415B dt=5001 wait=203 +21:27:29 step 170340 loss=2.9701 lr=2.24e-05 tok=22.417B dt=5112 wait=203 +21:29:42 step 170360 loss=3.0220 lr=2.24e-05 tok=22.420B dt=5245 wait=203 +21:30:53 global r=697 load 4.278 -> 4.282 +21:32:21 step 170420 loss=2.9501 lr=2.24e-05 tok=22.424B dt=5404 wait=203 +21:34:32 step 170440 loss=2.9655 lr=2.24e-05 tok=22.426B dt=5535 wait=204 +21:36:44 step 170460 loss=2.9448 lr=2.24e-05 tok=22.429B dt=5667 wait=204 +21:38:32 step 170480 loss=3.0026 lr=2.24e-05 tok=22.432B dt=5774 wait=204 +21:40:51 step 170500 loss=2.9659 lr=2.24e-05 tok=22.434B dt=5914 wait=204 +21:42:42 step 170520 loss=3.0997 lr=2.23e-05 tok=22.437B dt=6024 wait=204 +21:42:48 global r=698 load 4.277 -> 4.285 +21:43:20 step 170560 loss=2.9550 lr=2.23e-05 tok=22.438B dt=6062 wait=204 +21:44:24 step 170580 loss=2.9073 lr=2.23e-05 tok=22.441B dt=6126 wait=204 +21:45:28 step 170600 loss=2.8745 lr=2.23e-05 tok=22.443B dt=6190 wait=204 +21:46:32 step 170620 loss=3.0708 lr=2.23e-05 tok=22.446B dt=6254 wait=205 +21:47:36 step 170640 loss=2.8813 lr=2.23e-05 tok=22.449B dt=6318 wait=205 +21:48:40 step 170660 loss=3.0016 lr=2.23e-05 tok=22.451B dt=6382 wait=205 +21:49:43 step 170680 loss=3.0154 lr=2.23e-05 tok=22.454B dt=6446 wait=205 +21:50:46 step 170700 loss=2.9615 lr=2.23e-05 tok=22.457B dt=6509 wait=205 +21:50:59 pushed ckpt_w165.pt,log_w165.txt +21:54:00 waited 181s for global r=699 +21:54:07 global r=699 load 4.277 -> 4.282 +21:55:12 step 170720 loss=2.9818 lr=2.23e-05 tok=22.459B dt=6774 wait=205 +21:56:15 step 170740 loss=2.9550 lr=2.22e-05 tok=22.462B dt=6838 wait=205 +21:57:19 step 170760 loss=3.0329 lr=2.22e-05 tok=22.464B dt=6901 wait=206 +21:58:22 step 170780 loss=2.9458 lr=2.22e-05 tok=22.467B dt=6965 wait=206 +21:59:26 step 170800 loss=2.9457 lr=2.22e-05 tok=22.470B dt=7029 wait=206 +22:00:30 step 170820 loss=3.0260 lr=2.22e-05 tok=22.472B dt=7092 wait=206 +22:01:34 step 170840 loss=2.9344 lr=2.22e-05 tok=22.475B dt=7156 wait=206 +22:02:24 pushed ckpt_w165.pt,log_w165.txt +22:05:58 step 170860 loss=3.1339 lr=2.22e-05 tok=22.478B dt=7420 wait=206 +22:06:08 global r=700 load 4.286 -> 4.283 +22:06:40 step 170860 loss=3.1222 lr=2.22e-05 tok=22.479B dt=7462 wait=206 +22:07:47 step 170880 loss=2.9910 lr=2.22e-05 tok=22.481B dt=7529 wait=207 +22:09:57 step 170900 loss=2.9335 lr=2.22e-05 tok=22.484B dt=7659 wait=207 +22:12:11 step 170920 loss=3.0448 lr=2.22e-05 tok=22.487B dt=7794 wait=207 +22:14:24 step 170940 loss=2.9579 lr=2.21e-05 tok=22.489B dt=7926 wait=207 +22:15:59 step 170960 loss=2.9859 lr=2.21e-05 tok=22.492B dt=8021 wait=207 +22:17:54 step 170980 loss=2.9120 lr=2.21e-05 tok=22.495B dt=8136 wait=207 +22:18:00 global r=701 load 4.278 -> 4.287 +22:20:09 step 171020 loss=2.9020 lr=2.21e-05 tok=22.497B dt=8272 wait=207 +22:22:20 step 171040 loss=3.0142 lr=2.21e-05 tok=22.500B dt=8402 wait=208 +22:24:19 step 171060 loss=3.0529 lr=2.21e-05 tok=22.502B dt=8521 wait=208 +22:26:20 step 171080 loss=2.9256 lr=2.21e-05 tok=22.505B dt=8642 wait=208 +22:28:38 step 171100 loss=2.9812 lr=2.21e-05 tok=22.508B dt=8781 wait=208 +22:30:02 global r=702 load 4.280 -> 4.284 +22:31:11 step 171160 loss=2.9993 lr=2.20e-05 tok=22.510B dt=8934 wait=208 +22:32:37 step 171180 loss=2.9574 lr=2.20e-05 tok=22.513B dt=9020 wait=208 +22:34:50 step 171200 loss=2.8327 lr=2.20e-05 tok=22.516B dt=9153 wait=208 +22:37:01 step 171220 loss=2.9760 lr=2.20e-05 tok=22.518B dt=9284 wait=208 +22:39:13 step 171240 loss=2.8578 lr=2.20e-05 tok=22.521B dt=9416 wait=209 +22:40:42 step 171260 loss=2.9662 lr=2.20e-05 tok=22.523B dt=9504 wait=209 +22:40:49 global r=703 load 4.289 -> 4.279 +22:42:57 step 171320 loss=2.9721 lr=2.20e-05 tok=22.526B dt=9640 wait=209 +22:45:04 step 171340 loss=2.9943 lr=2.20e-05 tok=22.529B dt=9767 wait=209 +22:47:13 step 171360 loss=2.9253 lr=2.19e-05 tok=22.531B dt=9896 wait=209 +22:48:51 step 171380 loss=2.9526 lr=2.19e-05 tok=22.534B dt=9993 wait=209 +22:50:38 step 171400 loss=2.8513 lr=2.19e-05 tok=22.537B dt=10101 wait=209 +22:52:50 step 171420 loss=3.0572 lr=2.19e-05 tok=22.539B dt=10232 wait=209 +22:52:56 global r=704 load 4.284 -> 4.282 +22:54:03 step 171460 loss=2.9551 lr=2.19e-05 tok=22.540B dt=10306 wait=209 +22:56:15 step 171480 loss=2.8338 lr=2.19e-05 tok=22.543B dt=10438 wait=210 +22:57:43 step 171500 loss=2.6356 lr=2.19e-05 tok=22.546B dt=10526 wait=210 +22:59:51 step 171520 loss=2.9797 lr=2.19e-05 tok=22.548B dt=10654 wait=210 +23:02:01 step 171540 loss=2.7677 lr=2.19e-05 tok=22.551B dt=10784 wait=210 +23:04:11 step 171560 loss=2.9070 lr=2.19e-05 tok=22.554B dt=10914 wait=210 +23:04:18 global r=705 load 4.289 -> 4.282 +23:05:58 step 171620 loss=2.8256 lr=2.18e-05 tok=22.556B dt=11021 wait=210 +23:07:02 step 171640 loss=2.8675 lr=2.18e-05 tok=22.559B dt=11084 wait=210 +23:08:06 step 171660 loss=2.7958 lr=2.18e-05 tok=22.561B dt=11149 wait=211 +23:09:10 step 171680 loss=2.7608 lr=2.18e-05 tok=22.564B dt=11213 wait=211 +23:10:14 step 171700 loss=2.8211 lr=2.18e-05 tok=22.567B dt=11276 wait=211 +23:11:17 step 171720 loss=2.8765 lr=2.18e-05 tok=22.569B dt=11340 wait=211 +23:12:21 step 171740 loss=2.7878 lr=2.18e-05 tok=22.572B dt=11404 wait=211 +23:13:06 pushed ckpt_w165.pt,log_w165.txt +23:15:37 waited 151s for global r=706 +23:15:47 global r=706 load 4.282 -> 4.279 +23:16:21 step 171760 loss=2.8663 lr=2.18e-05 tok=22.575B dt=11643 wait=211 +23:17:24 step 171780 loss=2.8038 lr=2.18e-05 tok=22.577B dt=11707 wait=211 +23:18:27 step 171800 loss=2.8314 lr=2.17e-05 tok=22.580B dt=11770 wait=212 +23:19:32 step 171820 loss=2.8892 lr=2.17e-05 tok=22.582B dt=11835 wait=212 +23:20:35 step 171840 loss=2.9283 lr=2.17e-05 tok=22.585B dt=11898 wait=212 +23:21:40 step 171860 loss=3.0459 lr=2.17e-05 tok=22.588B dt=11962 wait=212 +23:22:43 step 171880 loss=2.9618 lr=2.17e-05 tok=22.590B dt=12026 wait=212 +23:23:48 step 171900 loss=3.0273 lr=2.17e-05 tok=22.593B dt=12090 wait=212 +23:24:00 pushed ckpt_w165.pt,log_w165.txt +23:27:01 waited 181s for global r=707 +23:27:08 global r=707 load 4.276 -> 4.277 +23:28:13 step 171920 loss=3.0320 lr=2.17e-05 tok=22.596B dt=12355 wait=212 +23:29:17 step 171940 loss=3.1336 lr=2.17e-05 tok=22.598B dt=12419 wait=213 +23:30:21 step 171960 loss=3.0886 lr=2.17e-05 tok=22.601B dt=12483 wait=213 +23:31:24 step 171980 loss=3.0132 lr=2.17e-05 tok=22.603B dt=12547 wait=213 +23:32:29 step 172000 loss=2.9906 lr=2.17e-05 tok=22.606B dt=12611 wait=213 +23:33:33 step 172020 loss=3.0977 lr=2.16e-05 tok=22.609B dt=12675 wait=213 +23:34:36 step 172040 loss=3.0474 lr=2.16e-05 tok=22.611B dt=12739 wait=213 +23:35:21 pushed ckpt_w165.pt,log_w165.txt +23:38:56 step 172060 loss=3.1248 lr=2.16e-05 tok=22.614B dt=12998 wait=213 +23:39:02 global r=708 load 4.277 -> 4.281 +23:39:34 step 172060 loss=3.0340 lr=2.16e-05 tok=22.615B dt=13037 wait=214 +23:40:38 step 172080 loss=2.9778 lr=2.16e-05 tok=22.618B dt=13101 wait=214 +23:41:43 step 172100 loss=2.9936 lr=2.16e-05 tok=22.620B dt=13165 wait=214 +23:42:48 step 172120 loss=3.0605 lr=2.16e-05 tok=22.623B dt=13230 wait=214 +23:43:52 step 172140 loss=3.0575 lr=2.16e-05 tok=22.626B dt=13295 wait=214 +23:44:58 step 172160 loss=3.0904 lr=2.16e-05 tok=22.628B dt=13360 wait=214 +23:46:02 step 172180 loss=2.9820 lr=2.16e-05 tok=22.631B dt=13425 wait=215 +23:47:07 step 172200 loss=2.9364 lr=2.16e-05 tok=22.634B dt=13490 wait=215 +23:47:22 pushed ckpt_w165.pt,log_w165.txt +23:49:53 waited 151s for global r=709 +23:50:01 global r=709 load 4.276 -> 4.280 +23:51:07 step 172220 loss=2.9360 lr=2.16e-05 tok=22.636B dt=13729 wait=215 +23:52:11 step 172240 loss=2.9716 lr=2.15e-05 tok=22.639B dt=13793 wait=215 +23:53:14 step 172260 loss=2.8963 lr=2.15e-05 tok=22.641B dt=13857 wait=215 +23:54:18 step 172280 loss=2.8959 lr=2.15e-05 tok=22.644B dt=13920 wait=215 +23:55:21 step 172300 loss=2.8485 lr=2.15e-05 tok=22.647B dt=13984 wait=216 +23:56:25 step 172320 loss=2.9604 lr=2.15e-05 tok=22.649B dt=14047 wait=216 +23:57:28 step 172340 loss=2.8576 lr=2.15e-05 tok=22.652B dt=14110 wait=216 +23:58:14 pushed ckpt_w165.pt,log_w165.txt +00:01:48 step 172360 loss=3.0021 lr=2.15e-05 tok=22.654B dt=14371 wait=216 +00:01:54 global r=710 load 4.276 -> 4.280 +00:02:26 step 172360 loss=2.9649 lr=2.15e-05 tok=22.656B dt=14408 wait=216 +00:03:29 step 172380 loss=2.8929 lr=2.15e-05 tok=22.658B dt=14472 wait=216 +00:04:33 step 172400 loss=2.8789 lr=2.15e-05 tok=22.661B dt=14535 wait=216 +00:05:37 step 172420 loss=2.9650 lr=2.15e-05 tok=22.664B dt=14599 wait=217 +00:06:40 step 172440 loss=2.8978 lr=2.15e-05 tok=22.666B dt=14662 wait=217 +00:07:43 step 172460 loss=3.0117 lr=2.14e-05 tok=22.669B dt=14726 wait=217 +00:08:48 step 172480 loss=2.9241 lr=2.14e-05 tok=22.672B dt=14790 wait=217 +00:09:51 step 172500 loss=2.9511 lr=2.14e-05 tok=22.674B dt=14854 wait=217 diff --git a/log_w166.txt b/log_w166.txt new file mode 100644 index 0000000000000000000000000000000000000000..89e55eee4c1bbfb8d75b76eab67b3ba148a2ac8f --- /dev/null +++ b/log_w166.txt @@ -0,0 +1,189 @@ +launch +05:29:07 name=w166 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 113/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +05:29:13 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +05:29:13 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +05:29:24 params 1.08B +05:29:26 resumed step=173250 seen=22.82B round=711 +05:29:26 stream salt=173250 +05:31:15 blocks compiled +05:31:15 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:34:16 mix source 0 paused 376s ((Request ID: Root=1-6abc9f58-6c0faa78268dc1193f407818;4f6fbc09-5050-4682-8d68-53) +05:34:16 mix source 7 paused 328s ((Request ID: Root=1-6abc9f58-0e0c9f771cbbbff13225baad;dcf3690f-91a3-4854-8bfd-72) +05:34:17 mix source 6 paused 385s ((Request ID: Root=1-6abc9f59-4033b9924f5e048c71d6cf49;688c9e11-cfb1-4576-9203-91) +05:34:43 step 173251 loss=2.9383 lr=2.11e-05 tok=22.821B dt=208 wait=182 +05:34:59 step 173252 loss=2.9534 lr=2.11e-05 tok=22.821B dt=225 wait=182 +05:35:25 step 173260 loss=3.2809 lr=2.11e-05 tok=22.822B dt=250 wait=182 +05:36:30 step 173280 loss=3.1320 lr=2.11e-05 tok=22.825B dt=315 wait=182 +05:37:35 step 173300 loss=3.0533 lr=2.11e-05 tok=22.827B dt=380 wait=182 +05:38:38 step 173320 loss=3.1321 lr=2.11e-05 tok=22.830B dt=444 wait=183 +05:39:42 step 173340 loss=3.1477 lr=2.11e-05 tok=22.833B dt=508 wait=183 +05:40:48 step 173360 loss=3.2601 lr=2.10e-05 tok=22.835B dt=573 wait=183 +05:41:53 step 173380 loss=3.0349 lr=2.10e-05 tok=22.838B dt=638 wait=183 +05:42:57 step 173400 loss=3.1461 lr=2.10e-05 tok=22.841B dt=703 wait=183 +05:43:12 pushed ckpt_w166.pt,log_w166.txt +05:47:05 global r=712 load 4.281 -> 4.280 +05:48:11 step 173420 loss=2.9548 lr=2.10e-05 tok=22.845B dt=1016 wait=184 +05:49:15 step 173440 loss=3.0981 lr=2.10e-05 tok=22.847B dt=1081 wait=184 +05:50:20 step 173460 loss=3.1861 lr=2.10e-05 tok=22.850B dt=1146 wait=184 +05:51:25 step 173480 loss=3.1348 lr=2.10e-05 tok=22.852B dt=1211 wait=184 +05:52:30 step 173500 loss=3.0963 lr=2.10e-05 tok=22.855B dt=1276 wait=184 +05:53:36 step 173520 loss=3.3025 lr=2.10e-05 tok=22.858B dt=1341 wait=184 +05:54:40 step 173540 loss=3.1337 lr=2.10e-05 tok=22.860B dt=1406 wait=185 +05:55:27 pushed ckpt_w166.pt,log_w166.txt +05:56:13 waited 46s for global r=713 +05:56:24 global r=713 load 4.286 -> 4.280 +05:56:59 step 173560 loss=3.2154 lr=2.10e-05 tok=22.863B dt=1544 wait=185 +05:58:03 step 173580 loss=3.1117 lr=2.09e-05 tok=22.866B dt=1609 wait=185 +05:59:08 step 173600 loss=3.2278 lr=2.09e-05 tok=22.868B dt=1673 wait=185 +06:00:13 step 173620 loss=3.3376 lr=2.09e-05 tok=22.871B dt=1738 wait=185 +06:01:16 step 173640 loss=3.2629 lr=2.09e-05 tok=22.873B dt=1802 wait=185 +06:02:29 step 173660 loss=3.0942 lr=2.09e-05 tok=22.876B dt=1875 wait=193 +06:03:34 step 173680 loss=3.1069 lr=2.09e-05 tok=22.879B dt=1939 wait=194 +06:04:37 step 173700 loss=3.2010 lr=2.09e-05 tok=22.881B dt=2003 wait=194 +06:04:54 pushed ckpt_w166.pt,log_w166.txt +06:05:55 waited 61s for global r=714 +06:06:02 global r=714 load 4.280 -> 4.281 +06:07:07 step 173720 loss=3.0729 lr=2.09e-05 tok=22.884B dt=2152 wait=194 +06:08:12 step 173740 loss=3.0673 lr=2.09e-05 tok=22.886B dt=2217 wait=194 +06:09:16 step 173760 loss=3.0590 lr=2.09e-05 tok=22.889B dt=2282 wait=194 +06:10:21 step 173780 loss=3.0759 lr=2.09e-05 tok=22.892B dt=2347 wait=194 +06:11:26 step 173800 loss=3.0918 lr=2.09e-05 tok=22.894B dt=2411 wait=195 +06:12:30 step 173820 loss=3.0925 lr=2.08e-05 tok=22.897B dt=2475 wait=195 +06:13:34 step 173840 loss=2.9500 lr=2.08e-05 tok=22.900B dt=2539 wait=195 +06:14:24 pushed ckpt_w166.pt,log_w166.txt +06:15:39 waited 76s for global r=715 +06:15:48 global r=715 load 4.288 -> 4.281 +06:16:21 step 173860 loss=3.0903 lr=2.08e-05 tok=22.902B dt=2707 wait=195 +06:17:26 step 173880 loss=3.0189 lr=2.08e-05 tok=22.905B dt=2771 wait=195 +06:18:29 step 173900 loss=2.8929 lr=2.08e-05 tok=22.907B dt=2834 wait=195 +06:19:33 step 173920 loss=3.0579 lr=2.08e-05 tok=22.910B dt=2898 wait=195 +06:20:36 step 173940 loss=2.9309 lr=2.08e-05 tok=22.913B dt=2962 wait=196 +06:21:41 step 173960 loss=2.9933 lr=2.08e-05 tok=22.915B dt=3026 wait=196 +06:22:45 step 173980 loss=2.9145 lr=2.08e-05 tok=22.918B dt=3090 wait=196 +06:23:48 step 174000 loss=2.8969 lr=2.08e-05 tok=22.921B dt=3154 wait=196 +06:24:07 pushed ckpt_w166.pt,log_w166.txt +06:25:08 waited 61s for global r=716 +06:25:18 global r=716 load 4.285 -> 4.282 +06:26:25 step 174020 loss=2.8613 lr=2.08e-05 tok=22.923B dt=3311 wait=196 +06:27:29 step 174040 loss=2.8966 lr=2.07e-05 tok=22.926B dt=3375 wait=196 +06:28:33 step 174060 loss=2.9327 lr=2.07e-05 tok=22.928B dt=3439 wait=196 +06:29:37 step 174080 loss=2.9092 lr=2.07e-05 tok=22.931B dt=3503 wait=197 +06:30:41 step 174100 loss=2.9805 lr=2.07e-05 tok=22.934B dt=3566 wait=197 +06:31:46 step 174120 loss=3.0021 lr=2.07e-05 tok=22.936B dt=3632 wait=197 +06:32:51 step 174140 loss=2.9754 lr=2.07e-05 tok=22.939B dt=3696 wait=197 +06:33:37 pushed ckpt_w166.pt,log_w166.txt +06:34:39 waited 61s for global r=717 +06:34:47 global r=717 load 4.283 -> 4.279 +06:35:21 step 174160 loss=2.9177 lr=2.07e-05 tok=22.942B dt=3847 wait=197 +06:36:26 step 174180 loss=2.9383 lr=2.07e-05 tok=22.944B dt=3911 wait=197 +06:37:31 step 174200 loss=2.8339 lr=2.07e-05 tok=22.947B dt=3976 wait=197 +06:38:36 step 174220 loss=2.9493 lr=2.07e-05 tok=22.949B dt=4042 wait=198 +06:39:41 step 174240 loss=2.8573 lr=2.07e-05 tok=22.952B dt=4107 wait=198 +06:40:46 step 174260 loss=3.1246 lr=2.07e-05 tok=22.955B dt=4171 wait=198 +06:41:51 step 174280 loss=2.9709 lr=2.06e-05 tok=22.957B dt=4237 wait=198 +06:42:56 step 174300 loss=2.9646 lr=2.06e-05 tok=22.960B dt=4302 wait=198 +06:43:20 pushed ckpt_w166.pt,log_w166.txt +06:44:20 waited 61s for global r=718 +06:44:28 global r=718 load 4.280 -> 4.281 +06:45:34 step 174320 loss=2.8601 lr=2.06e-05 tok=22.963B dt=4459 wait=198 +06:46:38 step 174340 loss=2.8439 lr=2.06e-05 tok=22.965B dt=4524 wait=199 +06:47:43 step 174360 loss=2.9135 lr=2.06e-05 tok=22.968B dt=4588 wait=199 +06:48:47 step 174380 loss=2.9555 lr=2.06e-05 tok=22.970B dt=4653 wait=199 +06:49:52 step 174400 loss=3.0362 lr=2.06e-05 tok=22.973B dt=4717 wait=199 +06:50:57 step 174420 loss=2.7677 lr=2.06e-05 tok=22.976B dt=4782 wait=199 +06:52:01 step 174440 loss=3.0615 lr=2.06e-05 tok=22.978B dt=4846 wait=199 +06:52:50 pushed ckpt_w166.pt,log_w166.txt +06:53:51 waited 61s for global r=719 +06:54:02 global r=719 load 4.279 -> 4.278 +06:54:36 step 174460 loss=3.0619 lr=2.06e-05 tok=22.981B dt=5001 wait=199 +06:55:40 step 174480 loss=3.0220 lr=2.06e-05 tok=22.983B dt=5065 wait=200 +06:56:44 step 174500 loss=2.9435 lr=2.06e-05 tok=22.986B dt=5129 wait=200 +06:57:49 step 174520 loss=3.2045 lr=2.05e-05 tok=22.989B dt=5195 wait=200 +06:58:53 step 174540 loss=2.9456 lr=2.05e-05 tok=22.991B dt=5259 wait=200 +06:59:57 step 174560 loss=3.0503 lr=2.05e-05 tok=22.994B dt=5323 wait=200 +07:01:01 step 174580 loss=2.9453 lr=2.05e-05 tok=22.997B dt=5386 wait=200 +07:02:05 step 174600 loss=2.9334 lr=2.05e-05 tok=22.999B dt=5450 wait=200 +07:02:16 pushed ckpt_w166.pt,log_w166.txt +07:03:48 waited 93s for global r=720 +07:03:55 global r=720 load 4.283 -> 4.282 +07:05:02 step 174620 loss=2.8678 lr=2.05e-05 tok=23.002B dt=5627 wait=201 +07:06:06 step 174640 loss=2.9025 lr=2.05e-05 tok=23.004B dt=5691 wait=201 +07:07:10 step 174660 loss=2.9713 lr=2.05e-05 tok=23.007B dt=5756 wait=201 +07:08:14 step 174680 loss=2.9360 lr=2.05e-05 tok=23.010B dt=5820 wait=201 +07:09:18 step 174700 loss=2.9553 lr=2.05e-05 tok=23.012B dt=5884 wait=201 +07:10:23 step 174720 loss=3.0134 lr=2.05e-05 tok=23.015B dt=5948 wait=201 +07:11:27 step 174740 loss=2.8866 lr=2.05e-05 tok=23.018B dt=6013 wait=201 +07:12:19 pushed ckpt_w166.pt,log_w166.txt +07:13:20 waited 61s for global r=721 +07:13:33 global r=721 load 4.284 -> 4.282 +07:14:06 step 174760 loss=3.1317 lr=2.04e-05 tok=23.020B dt=6172 wait=201 +07:15:10 step 174780 loss=3.0088 lr=2.04e-05 tok=23.023B dt=6236 wait=202 +07:16:14 step 174800 loss=3.0923 lr=2.04e-05 tok=23.025B dt=6300 wait=202 +07:17:23 step 174820 loss=2.9549 lr=2.04e-05 tok=23.028B dt=6369 wait=202 +07:18:28 step 174840 loss=2.6181 lr=2.04e-05 tok=23.031B dt=6434 wait=202 +07:19:32 step 174860 loss=2.8222 lr=2.04e-05 tok=23.033B dt=6498 wait=202 +07:20:36 step 174880 loss=2.9290 lr=2.04e-05 tok=23.036B dt=6562 wait=202 +07:21:40 step 174900 loss=2.8761 lr=2.04e-05 tok=23.039B dt=6626 wait=202 +07:21:58 pushed ckpt_w166.pt,log_w166.txt +07:22:58 waited 61s for global r=722 +07:23:08 global r=722 load 4.290 -> 4.284 +07:24:19 step 174920 loss=2.9568 lr=2.04e-05 tok=23.041B dt=6785 wait=203 +07:25:23 step 174940 loss=3.0039 lr=2.04e-05 tok=23.044B dt=6848 wait=203 +07:26:28 step 174960 loss=2.8982 lr=2.04e-05 tok=23.046B dt=6914 wait=203 +07:27:33 step 174980 loss=2.8717 lr=2.04e-05 tok=23.049B dt=6979 wait=203 +07:28:37 step 175000 loss=2.9794 lr=2.03e-05 tok=23.052B dt=7043 wait=203 +07:29:41 step 175020 loss=2.9664 lr=2.03e-05 tok=23.054B dt=7107 wait=203 +07:30:46 step 175040 loss=2.8767 lr=2.03e-05 tok=23.057B dt=7171 wait=204 +07:31:32 pushed ckpt_w166.pt,log_w166.txt +07:32:48 waited 76s for global r=723 +07:32:56 global r=723 load 4.288 -> 4.282 +07:33:30 step 175060 loss=2.9829 lr=2.03e-05 tok=23.059B dt=7335 wait=204 +07:34:34 step 175080 loss=2.9165 lr=2.03e-05 tok=23.062B dt=7400 wait=204 +07:35:38 step 175100 loss=2.9245 lr=2.03e-05 tok=23.065B dt=7463 wait=204 +07:36:42 step 175120 loss=2.9899 lr=2.03e-05 tok=23.067B dt=7528 wait=204 +07:37:47 step 175140 loss=2.8622 lr=2.03e-05 tok=23.070B dt=7593 wait=204 +07:38:52 step 175160 loss=2.9755 lr=2.03e-05 tok=23.073B dt=7657 wait=204 +07:39:56 step 175180 loss=2.9576 lr=2.03e-05 tok=23.075B dt=7722 wait=205 +07:41:02 step 175200 loss=2.7924 lr=2.03e-05 tok=23.078B dt=7788 wait=205 +07:41:24 pushed ckpt_w166.pt,log_w166.txt +07:42:25 waited 61s for global r=724 +07:42:33 global r=724 load 4.279 -> 4.283 +07:43:39 step 175220 loss=2.7882 lr=2.03e-05 tok=23.080B dt=7944 wait=205 +07:44:43 step 175240 loss=2.9745 lr=2.02e-05 tok=23.083B dt=8008 wait=205 +07:45:48 step 175260 loss=2.9563 lr=2.02e-05 tok=23.086B dt=8073 wait=205 +07:46:53 step 175280 loss=2.8901 lr=2.02e-05 tok=23.088B dt=8138 wait=205 +07:47:57 step 175300 loss=3.0041 lr=2.02e-05 tok=23.091B dt=8202 wait=205 +07:49:01 step 175320 loss=3.0463 lr=2.02e-05 tok=23.094B dt=8266 wait=206 +07:50:06 step 175340 loss=2.7448 lr=2.02e-05 tok=23.096B dt=8331 wait=206 +07:50:56 pushed ckpt_w166.pt,log_w166.txt +07:51:57 waited 61s for global r=725 +07:52:05 global r=725 load 4.301 -> 4.286 +07:52:39 step 175360 loss=2.8528 lr=2.02e-05 tok=23.099B dt=8485 wait=206 +07:53:44 step 175380 loss=3.0495 lr=2.02e-05 tok=23.101B dt=8549 wait=206 +07:54:48 step 175400 loss=2.9619 lr=2.02e-05 tok=23.104B dt=8613 wait=206 +07:55:52 step 175420 loss=2.9922 lr=2.02e-05 tok=23.107B dt=8678 wait=206 +07:56:56 step 175440 loss=2.9671 lr=2.02e-05 tok=23.109B dt=8742 wait=207 +07:58:01 step 175460 loss=3.1040 lr=2.02e-05 tok=23.112B dt=8806 wait=207 +07:59:05 step 175480 loss=2.9753 lr=2.01e-05 tok=23.115B dt=8870 wait=207 +08:00:09 step 175500 loss=2.9270 lr=2.01e-05 tok=23.117B dt=8935 wait=207 +08:00:22 pushed ckpt_w166.pt,log_w166.txt +08:01:38 waited 76s for global r=726 +08:01:47 global r=726 load 4.289 -> 4.283 +08:02:52 step 175520 loss=2.9037 lr=2.01e-05 tok=23.120B dt=9097 wait=207 +08:03:55 step 175540 loss=2.9837 lr=2.01e-05 tok=23.122B dt=9161 wait=207 +08:04:59 step 175560 loss=2.9150 lr=2.01e-05 tok=23.125B dt=9224 wait=207 +08:06:02 step 175580 loss=2.8562 lr=2.01e-05 tok=23.128B dt=9288 wait=207 +08:07:06 step 175600 loss=2.9358 lr=2.01e-05 tok=23.130B dt=9351 wait=208 +08:08:09 step 175620 loss=2.9487 lr=2.01e-05 tok=23.133B dt=9415 wait=208 +08:09:13 step 175640 loss=2.8068 lr=2.01e-05 tok=23.136B dt=9479 wait=208 +08:09:57 pushed ckpt_w166.pt,log_w166.txt +08:11:16 waited 79s for global r=727 +08:11:25 global r=727 load 4.287 -> 4.284 +08:11:58 step 175660 loss=2.9493 lr=2.01e-05 tok=23.138B dt=9644 wait=208 +08:13:01 step 175680 loss=2.9174 lr=2.01e-05 tok=23.141B dt=9707 wait=208 +08:14:05 step 175700 loss=2.8023 lr=2.01e-05 tok=23.143B dt=9770 wait=208 +08:15:08 step 175720 loss=2.9966 lr=2.00e-05 tok=23.146B dt=9834 wait=208 +08:16:12 step 175740 loss=2.8875 lr=2.00e-05 tok=23.149B dt=9897 wait=209 +08:17:16 step 175760 loss=2.9764 lr=2.00e-05 tok=23.151B dt=9961 wait=209 +08:18:21 step 175780 loss=3.0244 lr=2.00e-05 tok=23.154B dt=10026 wait=209 +08:19:25 step 175800 loss=2.9840 lr=2.00e-05 tok=23.156B dt=10090 wait=209 diff --git a/log_w167.txt b/log_w167.txt new file mode 100644 index 0000000000000000000000000000000000000000..b8d06b55a1cf319b5dc64f2fa6d9c8ce4b5c6277 --- /dev/null +++ b/log_w167.txt @@ -0,0 +1,777 @@ +launch +05:29:38 name=w167 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 114/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +05:29:45 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +05:29:45 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +05:29:54 params 1.08B +05:29:56 resumed step=173250 seen=22.82B round=711 +05:29:56 stream salt=173250 +05:31:40 blocks compiled +05:31:40 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:34:10 dolma file skipped data/ingredient1-common_crawl-high-quality_20_education_and_jobs/shard_00000180.jsonl.zst: (Request ID: Root=1-6abc9f52-741bd53d39cb5e236e18ae26;c54242e4-e8b6-4922-b22c-297dac1ee3f1) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00053_f278.jsonl.zst: (Request ID: Root=1-6abc9f52-00670e5019c1e2c80de045d4;385d329c-ef18-4da1-8b8a-0c08917f46c1) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-common_crawl-high-quality_20_games/shard_00000103.jsonl.zst: (Request ID: Root=1-6abc9f52-061973cf4658d3bc45ebb577;0da9c5ef-d127-4eed-9404-cb91466b29c7) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-common_crawl-high-quality_19_transportation/shard_00000386.jsonl.zst: (Request ID: Root=1-6abc9f52-743da57303e0a8bd684d278b;9a7bff85-29a1-44fe-8826-e17df3a1f1f4) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-cranemath/shard_00000172-withid.jsonl.zst: (Request ID: Root=1-6abc9f52-4435a2417c54534e69c929a5;e0429485-b22b-4daf-8aac-3bd9f83138fd) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-wiki_to_rcqa-part1/00023_f247.jsonl.zst: (Request ID: Root=1-6abc9f52-40f50dee03c2e58643e1b20a;be7e24de-fe2c-4be7-9db0-c9983e47cdf3) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-common_crawl-high-quality_19_literature/shard_00000617.jsonl.zst: (Request ID: Root=1-6abc9f52-1c003fda2b2bd54b7ccb8814;761db735-7e99-40bb-be6d-9bdce500e142) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-common_crawl-high-quality_19_sports_and_fitness/shard_00000470.jsonl.zst: (Request ID: Root=1-6abc9f52-658ede202390dea6012964fd;aff03094-cf39-4497-b5a4-2a9ae2d727d3) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-finance_business-2e13/finance_business_p070_shard_00005005.jsonl.zst: (Request ID: Root=1-6abc9f52-4ea3f74a4aa50d186a53d23e;209ce3ef-d696-4290-9364-96107f8b382a) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-common_crawl-high-quality_20_software/shard_00000178.jsonl.zst: (Request ID: Root=1-6abc9f52-28e167c57a64626c0baaba79;d820924a-2233-48b5-abc7-3cfa8077b7a6) + +429 Too Many Requests: you +05:34:10 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-education_jobs-2e12/education_jobs_p080_shard_00002444.jsonl.zst: (Request ID: Root=1-6abc9f52-3296cbdc0211be742368b38d;af870201-af91-428a-b08f-98a1f58e71ef) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part1/00008_f290.jsonl.zst: (Request ID: Root=1-6abc9f52-6b9e5359706597ed67deec97;b72d0c5b-f148-4b7a-9b6c-679ce769ed29) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00055_f282.jsonl.zst: (Request ID: Root=1-6abc9f53-7ab7bc1f33a112e64ed619d4;a235b22c-c54b-4cf5-b00b-ef971bd33669) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part1/00013_f148.jsonl.zst: (Request ID: Root=1-6abc9f53-15bd6e2a07de49c422f09ccc;0ca9e06e-7708-4ac4-a41f-38d6756de080) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_19_education_and_jobs/shard_00000509.jsonl.zst: (Request ID: Root=1-6abc9f53-28d34f012aa4c9a66ef04169;7a00cdd6-ce83-4e42-80cf-7aaf2e53ce5b) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00042_f65.jsonl.zst: (Request ID: Root=1-6abc9f53-7d3281ba0ec7e3fc244e394b;7ebbf209-f9e1-4c6c-a2b2-3190c883f675) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00037_f46.jsonl.zst: (Request ID: Root=1-6abc9f53-6a115b3011a7ed8964f99afb;a738a009-03fc-4eb3-9b5c-a0958877747b) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00036_f23.jsonl.zst: (Request ID: Root=1-6abc9f53-63d836152f98b48e11ec1103;642a4c5d-d2e6-4160-9c10-a093ff387a76) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_20_sports_and_fitness/shard_00000063.jsonl.zst: (Request ID: Root=1-6abc9f53-4016d7c37dd49a8360ab00c5;88e8f4c2-e680-40ae-9908-6d35cd2f1164) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part1/00014_f171.jsonl.zst: (Request ID: Root=1-6abc9f53-7fd3e3355bd6f6847f5d0bd2;b5c43033-234e-4fb8-8b42-1b94fbf51106) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00037_f22.jsonl.zst: (Request ID: Root=1-6abc9f53-3f32351a354a1a9f7ba36d9d;382cbfb8-5590-4d44-bfde-4775cf863103) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-dolmino-math/dolmino_math_mathcoder2-synthmath_m-a-p_Matrix_filtered-math_book_math.0001.0417.jsonl.jsonl.zst: (Request ID: Root=1-6abc9f53-62a5abdd1bb0a9be46e2eebb;37579fd1-61b1-4d9b-b3e1-f9963e7646ca) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_19_transportation/shard_00000099.jsonl.zst: (Request ID: Root=1-6abc9f53-543af1fa1390e30e43c3b9bb;fe38a1bb-5a85-4cc7-8a8a-5c525167b877) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-science_tech-2e12/science_tech_p040_shard_00001351.jsonl.zst: (Request ID: Root=1-6abc9f53-0bf32f192d37b2847c8d1371;1f9ff71d-dceb-4897-8783-c1610d12b430) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part1/00015_f188.jsonl.zst: (Request ID: Root=1-6abc9f53-6914e7417c217b1b0fb5e52a;b7497108-519a-475f-b80e-5ef13e3f5320) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00057_f252.jsonl.zst: (Request ID: Root=1-6abc9f53-15071cc77ae76143753b3d6f;fef7cf9d-7f6b-4f49-8aa3-bb1935b44f5b) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part1/00002_f156.jsonl.zst: (Request ID: Root=1-6abc9f53-1dc3da9c0ea79588762e72f2;20a4838f-4803-49b4-bc83-4fadf75761fe) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_20_health/shard_00000073.jsonl.zst: (Request ID: Root=1-6abc9f53-2e0beed1326d8caf2ee3e05b;76ba254d-4dce-4380-ac34-c99f22f65f4b) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-dolmino-math/dolmino_math_mathcoder2-synthmath_m-a-p_Matrix_filtered-math_book_math.0002.0262.jsonl.jsonl.zst: (Request ID: Root=1-6abc9f53-74efdcbb41ad4cb20f16305e;438353d6-83c0-4987-8402-9ee7615075a1) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00034_f199.jsonl.zst: (Request ID: Root=1-6abc9f53-6c0a5f8e445a42e60295033f;0435da4e-0929-4780-b56f-66b5b9dc6585) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_19_social_life/shard_00000208.jsonl.zst: (Request ID: Root=1-6abc9f53-4c682c38025c12aa10f97edd;6174d8d6-0459-4695-b64b-325fd6cf5a56) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_19_religion/shard_00000459.jsonl.zst: (Request ID: Root=1-6abc9f53-342fd30e7fe635e5244767e6;762882c6-6e86-40a3-bc3b-e7b0218dcbf9) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_20_fashion_and_beauty/shard_00000291.jsonl.zst: (Request ID: Root=1-6abc9f53-7f78d229794c3df5400512cf;07ad51aa-b3a0-4f7a-850f-311ad834fde1) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00033_f59.jsonl.zst: (Request ID: Root=1-6abc9f53-42a3320178b7e4d400738e19;3fe5dca0-eacb-40b9-9b76-0b1aa8f1b1d8) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00036_f275.jsonl.zst: (Request ID: Root=1-6abc9f53-00772b2528962d6005a7f9da;75420010-4937-4f6c-92a6-24a9cc5a9a2f) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-health-2e12/health_p060_shard_00003649.jsonl.zst: (Request ID: Root=1-6abc9f53-15bc7daf7c0081430b37486f;4ef843c1-3dc4-4c7e-91d1-8dd339b528b5) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_19_travel_and_tourism/shard_00000008.jsonl.zst: (Request ID: Root=1-6abc9f53-1a6a542a4bc50bd5477ead72;1065540e-7f0c-425f-8cc7-bc4dacd3c5de) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_20_health/shard_00000690.jsonl.zst: (Request ID: Root=1-6abc9f53-678bc7cd493a43fc26724805;eaefc68a-1049-4bdf-a30b-8b25ee16cf12) + +429 Too Many Requests: you +05:34:11 dolma file skipped data/ingredient1-common_crawl-high-quality_19_finance_and_business/shard_00000109.jsonl.zst: (Request ID: Root=1-6abc9f53-061d906b0bc9875841de6249;ec15e545-4c43-4b3c-b82a-cf82573d6dbc) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-dolmino-math/dolmino_math_mathcoder2-synthmath_m-a-p_Matrix_filtered-math_book_math.0000.0667.jsonl.jsonl.zst: (Request ID: Root=1-6abc9f54-438717aa7511777b644f3fc6;5fc614dc-f2dd-4c2c-a2e2-d37754f92915) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-common_crawl-high-quality_19_home_and_hobbies/shard_00001144.jsonl.zst: (Request ID: Root=1-6abc9f54-1afb087665e59a5b12278220;87522f26-af49-4704-9920-f3c3b9997692) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-nemotron-synth-qa/CC-MAIN-2020-29-part-00005.jsonl.zst: (Request ID: Root=1-6abc9f54-656a6eda146347eb1acff511;e511d5e1-01db-4e5f-af02-3ab76bc30853) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-common_crawl-high-quality_19_home_and_hobbies/shard_00000256.jsonl.zst: (Request ID: Root=1-6abc9f54-068ca516074dd4b633ffc7b3;71caf3e0-8d32-4450-9432-48cb448da350) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-nemotron-synth-qa/CC-MAIN-2022-21-part-00026.jsonl.zst: (Request ID: Root=1-6abc9f54-02e676cb682814196d75919f;9e60df30-6345-4b5e-aa70-a636f22b3710) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-cranecode/shard_00000185.jsonl.zst: (Request ID: Root=1-6abc9f54-02df0311425f23a579fac583;6ed8b7bc-6f7c-4b1f-9b63-d96ee8890347) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00038_f108.jsonl.zst: (Request ID: Root=1-6abc9f54-1ec30b6c68753a7a770c33cc;8a162c09-9708-4cde-9f02-862148bcb8fa) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-stack_edu-fim_vigintile_17_Java/shard_00000001.jsonl.zst: (Request ID: Root=1-6abc9f54-5e1428d42a99456274fbfa53;1a23bfe6-4dae-4844-a48e-79c863a84ea4) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-common_crawl-high-quality_19_games/shard_00000599.jsonl.zst: (Request ID: Root=1-6abc9f54-6919875a0e35993a267b9446;389144cc-5988-49ef-961e-9c7573166ea4) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-dolmino-math/dolmino_math_mathcoder2-synthmath_m-a-p_Matrix_filtered-math_book_math.0000.0163.jsonl.jsonl.zst: (Request ID: Root=1-6abc9f54-75554e6947b9c75e029f80e2;64959e06-2c02-44fd-b568-b4354f3f4c8a) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-common_crawl-high-quality_20_politics/shard_00001059.jsonl.zst: (Request ID: Root=1-6abc9f54-3f9ccb8b69d3b0d4376e5ee2;d50e4f06-4a23-49f9-ad2a-64d41e7d60d8) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-dolmino-math/dolmino_math_mathcoder2-synthmath_m-a-p_Matrix_filtered-math_book_math.0001.0084.jsonl.jsonl.zst: (Request ID: Root=1-6abc9f54-5d503d285adb85ad2b36ae76;3797277e-8599-49ce-8f2a-9662a20324b2) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-health-2e12/health_p040_shard_00003664.jsonl.zst: (Request ID: Root=1-6abc9f54-13907d010e8d0ae24e6d5dca;90fa6586-24a5-4edc-b33d-d278e76e7af1) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-tinymath-mind/output_139.jsonl.zst: (Request ID: Root=1-6abc9f54-319b6dca4a8ce9ba2081fc0f;635ed12a-fb20-469b-af49-2a36e213a05a) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-wiki_to_rcqa-part1/00006_f305.jsonl.zst: (Request ID: Root=1-6abc9f54-306c5f3a7ac5ede3177d3370;0e8c9391-fff1-4731-a2a9-91ee48d454ff) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-common_crawl-high-quality_19_history_and_geography/shard_00000240.jsonl.zst: (Request ID: Root=1-6abc9f54-5716425f663d099a5e0ffa7d;aef001bf-765c-432b-a66b-c692392094bf) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-dolmino-math/dolmino_math_tulu_math_personahub_math_v5_train_0008.jsonl.jsonl.zst: (Request ID: Root=1-6abc9f54-09b61c522d0a66b71a7278ec;72d65c13-0e39-441b-922f-a5c2eed58ebe) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-wiki_to_rcqa-part1/00025_f111.jsonl.zst: (Request ID: Root=1-6abc9f54-729b37b00990390e4800707b;408334d2-0971-49dc-a378-e872fb6dfe11) + +429 Too Many Requests: you +05:34:12 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-crime_law-2e12/crime_law_p050_shard_00002605.jsonl.zst: (Request ID: Root=1-6abc9f54-36026a70162e2ad629c12cf0;53d61fb2-4cc8-4d7c-af4a-75472ed6ddda) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00051_f172.jsonl.zst: (Request ID: Root=1-6abc9f55-1d5115e0566a977822ea803f;673c3072-9318-40a6-ac5f-c286215756c1) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-wiki_to_rcqa-part1/00021_f161.jsonl.zst: (Request ID: Root=1-6abc9f55-67fcc735706d2f900dde7e3e;29a6178f-ab76-4c20-a578-83ad0f0f697e) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-common_crawl-high-quality_19_art_and_design/shard_00000202.jsonl.zst: (Request ID: Root=1-6abc9f55-59c2e31635876e3a12ece87c;0638cfb1-1040-4c00-984c-b5981ab1564c) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-common_crawl-high-quality_20_games/shard_00000426.jsonl.zst: (Request ID: Root=1-6abc9f55-5533990604dd380564e421a5;f5e139b5-ae2d-42c7-9756-4091bb85c618) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-common_crawl-high-quality_20_entertainment/shard_00000195.jsonl.zst: (Request ID: Root=1-6abc9f55-680185602d080e6949281054;185a30ae-0652-4123-8b70-6e308aa02a72) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-common_crawl-high-quality_20_home_and_hobbies/shard_00000151.jsonl.zst: (Request ID: Root=1-6abc9f55-19f115301b1565dd20eb1e35;2a0630df-0653-438c-904c-f486ea0028ac) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-dolmino-math/dolmino_math_mathcoder2-synthmath_m-a-p_Matrix_filtered-math_book_science.0000.0406.jsonl.jsonl.zst: (Request ID: Root=1-6abc9f55-4a6cee790a0b6405592b9d0f;e2baca05-6b8b-4399-a7c6-b41a7ea67499) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-dolmino-math/dolmino_math_tinyGSM-MIND_2students_tiny_gsm_inline_part004.000009.jsonl.jsonl.zst: (Request ID: Root=1-6abc9f55-28787f2872f798bf76b581f7;c0583ec2-47fe-47ac-83e7-146d0ab81700) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-stem-heavy-crawl/finance_and_business.00000205.jsonl.zst: (Request ID: Root=1-6abc9f55-57e5e4fb594f443a71b42a92;e7bdb4bc-cb70-45cf-b90a-58749fa33ed2) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-common_crawl-high-quality_20_transportation/shard_00000056.jsonl.zst: (Request ID: Root=1-6abc9f55-567e297a4932d8ff55f8c9d5;f786a3df-066d-4b49-9d9b-3ce950cac80e) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00034_f26.jsonl.zst: (Request ID: Root=1-6abc9f55-19beb6f15dcc82522fa30cab;c6a66421-ac62-46fb-b704-2432554c3f91) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-common_crawl-high-quality_20_politics/shard_00001010.jsonl.zst: (Request ID: Root=1-6abc9f55-054b393d26807f3b5722ae8c;37bbb15e-7ef3-42aa-a7c0-c94db7fa553a) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-wiki_to_rcqa-part2/00033_f122.jsonl.zst: (Request ID: Root=1-6abc9f55-3d421f7a575a60662aee3d69;57920b0e-79cc-46e4-b6e2-062767343fbb) + +429 Too Many Requests: you +05:34:13 dolma file skipped data/ingredient1-stem-heavy-crawl/software_development.00000260.jsonl.zst: (Request ID: Root=1-6abc9f55-7600fb4119e116591f87ee0f;7cf5d986-ba92-4750-80a2-c19f204338b2) + +429 Too Many Requests: you +05:38:08 step 173251 loss=2.7637 lr=2.11e-05 tok=22.821B dt=388 wait=360 +05:38:24 step 173252 loss=2.9044 lr=2.11e-05 tok=22.821B dt=404 wait=360 +05:38:50 step 173260 loss=3.2814 lr=2.11e-05 tok=22.822B dt=430 wait=360 +05:39:56 step 173280 loss=3.1337 lr=2.11e-05 tok=22.825B dt=496 wait=361 +05:41:03 step 173300 loss=3.1576 lr=2.11e-05 tok=22.827B dt=562 wait=361 +05:42:07 step 173320 loss=3.1150 lr=2.11e-05 tok=22.830B dt=627 wait=361 +05:43:12 step 173340 loss=3.2096 lr=2.11e-05 tok=22.833B dt=692 wait=361 +05:44:17 step 173360 loss=3.2900 lr=2.10e-05 tok=22.835B dt=757 wait=361 +05:45:22 step 173380 loss=3.1632 lr=2.10e-05 tok=22.838B dt=822 wait=361 +05:46:27 step 173400 loss=3.1569 lr=2.10e-05 tok=22.841B dt=887 wait=361 +05:46:44 pushed ckpt_w167.pt,log_w167.txt +05:46:45 waited 1s for global r=712 +05:47:04 global r=712 load 4.277 -> 4.280 +05:48:23 step 173420 loss=2.9979 lr=2.10e-05 tok=22.843B dt=1003 wait=362 +05:49:28 step 173440 loss=3.1342 lr=2.10e-05 tok=22.846B dt=1067 wait=362 +05:50:32 step 173460 loss=3.0833 lr=2.10e-05 tok=22.848B dt=1132 wait=362 +05:51:37 step 173480 loss=3.1144 lr=2.10e-05 tok=22.851B dt=1196 wait=362 +05:52:40 step 173500 loss=3.0249 lr=2.10e-05 tok=22.854B dt=1260 wait=362 +05:53:45 step 173520 loss=3.0972 lr=2.10e-05 tok=22.856B dt=1324 wait=362 +05:54:48 step 173540 loss=2.9879 lr=2.10e-05 tok=22.859B dt=1388 wait=363 +05:55:33 pushed ckpt_w167.pt,log_w167.txt +05:56:18 waited 45s for global r=713 +05:56:26 global r=713 load 4.286 -> 4.280 +05:57:00 step 173560 loss=2.9475 lr=2.10e-05 tok=22.862B dt=1520 wait=363 +05:58:04 step 173580 loss=2.9409 lr=2.09e-05 tok=22.864B dt=1584 wait=363 +05:59:32 step 173600 loss=3.0334 lr=2.09e-05 tok=22.867B dt=1672 wait=363 +06:00:36 step 173620 loss=3.0467 lr=2.09e-05 tok=22.869B dt=1736 wait=363 +06:01:40 step 173640 loss=2.8696 lr=2.09e-05 tok=22.872B dt=1800 wait=363 +06:02:44 step 173660 loss=2.9138 lr=2.09e-05 tok=22.875B dt=1863 wait=364 +06:03:47 step 173680 loss=2.9040 lr=2.09e-05 tok=22.877B dt=1927 wait=364 +06:04:51 step 173700 loss=2.9881 lr=2.09e-05 tok=22.880B dt=1991 wait=364 +06:05:06 pushed ckpt_w167.pt,log_w167.txt +06:05:51 waited 45s for global r=714 +06:05:57 global r=714 load 4.282 -> 4.281 +06:07:01 step 173720 loss=2.9499 lr=2.09e-05 tok=22.883B dt=2121 wait=364 +06:08:12 step 173740 loss=3.0087 lr=2.09e-05 tok=22.885B dt=2192 wait=364 +06:09:29 step 173760 loss=2.9864 lr=2.09e-05 tok=22.888B dt=2269 wait=364 +06:10:33 step 173780 loss=2.9449 lr=2.09e-05 tok=22.890B dt=2333 wait=364 +06:11:38 step 173800 loss=3.1434 lr=2.09e-05 tok=22.893B dt=2397 wait=365 +06:12:41 step 173820 loss=3.2926 lr=2.08e-05 tok=22.896B dt=2461 wait=365 +06:13:45 step 173840 loss=3.0891 lr=2.08e-05 tok=22.898B dt=2525 wait=365 +06:14:29 pushed ckpt_w167.pt,log_w167.txt +06:15:29 waited 61s for global r=715 +06:15:37 global r=715 load 4.286 -> 4.281 +06:16:10 step 173860 loss=3.0814 lr=2.08e-05 tok=22.901B dt=2669 wait=365 +06:17:13 step 173880 loss=3.0494 lr=2.08e-05 tok=22.904B dt=2733 wait=365 +06:18:36 step 173900 loss=3.0759 lr=2.08e-05 tok=22.906B dt=2816 wait=365 +06:19:43 step 173920 loss=3.0883 lr=2.08e-05 tok=22.909B dt=2883 wait=366 +06:20:47 step 173940 loss=2.9575 lr=2.08e-05 tok=22.911B dt=2946 wait=366 +06:21:51 step 173960 loss=3.1459 lr=2.08e-05 tok=22.914B dt=3011 wait=366 +06:22:55 step 173980 loss=2.9685 lr=2.08e-05 tok=22.917B dt=3075 wait=366 +06:23:59 step 174000 loss=3.0047 lr=2.08e-05 tok=22.919B dt=3138 wait=366 +06:24:11 pushed ckpt_w167.pt,log_w167.txt +06:25:11 waited 61s for global r=716 +06:25:18 global r=716 load 4.289 -> 4.282 +06:26:23 step 174020 loss=2.9100 lr=2.08e-05 tok=22.922B dt=3283 wait=366 +06:27:36 step 174040 loss=2.8496 lr=2.07e-05 tok=22.924B dt=3355 wait=366 +06:28:53 step 174060 loss=2.9282 lr=2.07e-05 tok=22.927B dt=3433 wait=367 +06:29:57 step 174080 loss=2.9037 lr=2.07e-05 tok=22.930B dt=3497 wait=367 +06:31:02 step 174100 loss=2.8225 lr=2.07e-05 tok=22.932B dt=3561 wait=367 +06:32:05 step 174120 loss=2.9839 lr=2.07e-05 tok=22.935B dt=3625 wait=367 +06:33:09 step 174140 loss=2.8708 lr=2.07e-05 tok=22.938B dt=3689 wait=367 +06:33:55 pushed ckpt_w167.pt,log_w167.txt +06:34:41 waited 45s for global r=717 +06:34:48 global r=717 load 4.282 -> 4.279 +06:35:22 step 174160 loss=3.0072 lr=2.07e-05 tok=22.940B dt=3821 wait=367 +06:36:25 step 174180 loss=2.9367 lr=2.07e-05 tok=22.943B dt=3885 wait=368 +06:37:45 step 174200 loss=2.9226 lr=2.07e-05 tok=22.945B dt=3964 wait=368 +06:38:53 step 174220 loss=3.0410 lr=2.07e-05 tok=22.948B dt=4033 wait=368 +06:39:57 step 174240 loss=2.8903 lr=2.07e-05 tok=22.951B dt=4097 wait=368 +06:41:01 step 174260 loss=2.9561 lr=2.07e-05 tok=22.953B dt=4160 wait=368 +06:42:04 step 174280 loss=2.9077 lr=2.06e-05 tok=22.956B dt=4224 wait=368 +06:43:08 step 174300 loss=2.9912 lr=2.06e-05 tok=22.959B dt=4288 wait=369 +06:43:20 pushed ckpt_w167.pt,log_w167.txt +06:44:20 waited 61s for global r=718 +06:44:26 global r=718 load 4.275 -> 4.281 +06:45:31 step 174320 loss=2.9414 lr=2.06e-05 tok=22.961B dt=4431 wait=369 +06:46:41 step 174340 loss=2.9929 lr=2.06e-05 tok=22.964B dt=4501 wait=369 +06:47:58 step 174360 loss=3.0305 lr=2.06e-05 tok=22.966B dt=4577 wait=369 +06:49:01 step 174380 loss=2.8677 lr=2.06e-05 tok=22.969B dt=4641 wait=369 +06:50:05 step 174400 loss=2.9042 lr=2.06e-05 tok=22.972B dt=4705 wait=369 +06:51:09 step 174420 loss=2.9760 lr=2.06e-05 tok=22.974B dt=4769 wait=370 +06:52:13 step 174440 loss=2.8580 lr=2.06e-05 tok=22.977B dt=4832 wait=370 +06:52:58 pushed ckpt_w167.pt,log_w167.txt +06:53:59 waited 61s for global r=719 +06:54:06 global r=719 load 4.281 -> 4.278 +06:54:41 step 174460 loss=2.9768 lr=2.06e-05 tok=22.980B dt=4981 wait=370 +06:55:44 step 174480 loss=2.9597 lr=2.06e-05 tok=22.982B dt=5044 wait=370 +06:57:03 step 174500 loss=2.9200 lr=2.06e-05 tok=22.985B dt=5123 wait=370 +06:58:13 step 174520 loss=2.9563 lr=2.05e-05 tok=22.987B dt=5193 wait=371 +06:59:17 step 174540 loss=2.9183 lr=2.05e-05 tok=22.990B dt=5257 wait=371 +07:00:20 step 174560 loss=3.0353 lr=2.05e-05 tok=22.993B dt=5320 wait=371 +07:01:24 step 174580 loss=2.5445 lr=2.05e-05 tok=22.995B dt=5384 wait=371 +07:02:28 step 174600 loss=2.9139 lr=2.05e-05 tok=22.998B dt=5447 wait=371 +07:02:41 pushed ckpt_w167.pt,log_w167.txt +07:03:42 waited 61s for global r=720 +07:03:49 global r=720 load 4.283 -> 4.282 +07:04:54 step 174620 loss=2.9025 lr=2.05e-05 tok=23.001B dt=5594 wait=371 +07:06:07 step 174640 loss=2.9282 lr=2.05e-05 tok=23.003B dt=5667 wait=371 +07:07:23 step 174660 loss=2.8791 lr=2.05e-05 tok=23.006B dt=5743 wait=372 +07:08:27 step 174680 loss=2.8972 lr=2.05e-05 tok=23.008B dt=5807 wait=372 +07:09:31 step 174700 loss=2.9048 lr=2.05e-05 tok=23.011B dt=5870 wait=372 +07:10:35 step 174720 loss=3.0312 lr=2.05e-05 tok=23.014B dt=5935 wait=372 +07:11:39 step 174740 loss=2.8907 lr=2.05e-05 tok=23.016B dt=5999 wait=372 +07:12:24 pushed ckpt_w167.pt,log_w167.txt +07:13:24 waited 61s for global r=721 +07:13:33 global r=721 load 4.283 -> 4.282 +07:14:06 step 174760 loss=2.9671 lr=2.04e-05 tok=23.019B dt=6146 wait=372 +07:15:09 step 174780 loss=2.8856 lr=2.04e-05 tok=23.021B dt=6209 wait=373 +07:16:29 step 174800 loss=2.8376 lr=2.04e-05 tok=23.024B dt=6289 wait=373 +07:17:36 step 174820 loss=2.9503 lr=2.04e-05 tok=23.027B dt=6356 wait=373 +07:18:41 step 174840 loss=2.5910 lr=2.04e-05 tok=23.029B dt=6421 wait=373 +07:19:44 step 174860 loss=2.9928 lr=2.04e-05 tok=23.032B dt=6484 wait=373 +07:20:48 step 174880 loss=2.8616 lr=2.04e-05 tok=23.035B dt=6548 wait=373 +07:21:52 step 174900 loss=2.8735 lr=2.04e-05 tok=23.037B dt=6611 wait=374 +07:22:05 pushed ckpt_w167.pt,log_w167.txt +07:23:06 waited 61s for global r=722 +07:23:12 global r=722 load 4.284 -> 4.284 +07:24:17 step 174920 loss=2.8584 lr=2.04e-05 tok=23.040B dt=6757 wait=374 +07:25:29 step 174940 loss=2.8500 lr=2.04e-05 tok=23.042B dt=6828 wait=374 +07:26:50 step 174960 loss=2.8190 lr=2.04e-05 tok=23.045B dt=6910 wait=374 +07:27:54 step 174980 loss=2.7410 lr=2.04e-05 tok=23.048B dt=6974 wait=374 +07:28:58 step 175000 loss=2.8645 lr=2.03e-05 tok=23.050B dt=7037 wait=374 +07:30:01 step 175020 loss=2.8962 lr=2.03e-05 tok=23.053B dt=7101 wait=375 +07:31:05 step 175040 loss=2.8045 lr=2.03e-05 tok=23.056B dt=7165 wait=375 +07:31:52 pushed ckpt_w167.pt,log_w167.txt +07:32:37 waited 45s for global r=723 +07:32:43 global r=723 load 4.284 -> 4.282 +07:33:16 step 175060 loss=2.9730 lr=2.03e-05 tok=23.058B dt=7296 wait=375 +07:34:20 step 175080 loss=2.9172 lr=2.03e-05 tok=23.061B dt=7360 wait=375 +07:35:40 step 175100 loss=2.6406 lr=2.03e-05 tok=23.063B dt=7440 wait=375 +07:36:50 step 175120 loss=3.0123 lr=2.03e-05 tok=23.066B dt=7510 wait=375 +07:37:54 step 175140 loss=2.9940 lr=2.03e-05 tok=23.069B dt=7574 wait=375 +07:38:59 step 175160 loss=3.0422 lr=2.03e-05 tok=23.071B dt=7639 wait=376 +07:40:03 step 175180 loss=2.9778 lr=2.03e-05 tok=23.074B dt=7703 wait=376 +07:41:08 step 175200 loss=3.1047 lr=2.03e-05 tok=23.077B dt=7767 wait=376 +07:41:20 pushed ckpt_w167.pt,log_w167.txt +07:42:21 waited 61s for global r=724 +07:42:27 global r=724 load 4.287 -> 4.283 +07:43:33 step 175220 loss=2.8452 lr=2.03e-05 tok=23.079B dt=7913 wait=376 +07:44:45 step 175240 loss=2.9558 lr=2.02e-05 tok=23.082B dt=7985 wait=376 +07:46:06 step 175260 loss=2.9310 lr=2.02e-05 tok=23.084B dt=8066 wait=376 +07:47:10 step 175280 loss=2.8653 lr=2.02e-05 tok=23.087B dt=8130 wait=377 +07:48:14 step 175300 loss=2.9374 lr=2.02e-05 tok=23.090B dt=8194 wait=377 +07:49:18 step 175320 loss=2.9553 lr=2.02e-05 tok=23.092B dt=8258 wait=377 +07:50:21 step 175340 loss=2.8481 lr=2.02e-05 tok=23.095B dt=8321 wait=377 +07:51:08 pushed ckpt_w167.pt,log_w167.txt +07:52:08 waited 61s for global r=725 +07:52:15 global r=725 load 4.280 -> 4.286 +07:52:49 step 175360 loss=2.8724 lr=2.02e-05 tok=23.098B dt=8468 wait=377 +07:53:52 step 175380 loss=2.7692 lr=2.02e-05 tok=23.100B dt=8532 wait=377 +07:55:17 step 175400 loss=2.7514 lr=2.02e-05 tok=23.103B dt=8616 wait=378 +07:56:22 step 175420 loss=2.6436 lr=2.02e-05 tok=23.105B dt=8681 wait=378 +07:57:26 step 175440 loss=2.8105 lr=2.02e-05 tok=23.108B dt=8745 wait=378 +07:58:29 step 175460 loss=3.0125 lr=2.02e-05 tok=23.111B dt=8809 wait=378 +07:59:33 step 175480 loss=2.9238 lr=2.01e-05 tok=23.113B dt=8873 wait=378 +08:00:37 step 175500 loss=2.9687 lr=2.01e-05 tok=23.116B dt=8937 wait=378 +08:00:49 pushed ckpt_w167.pt,log_w167.txt +08:01:35 waited 45s for global r=726 +08:01:44 global r=726 load 4.287 -> 4.283 +08:02:48 step 175520 loss=2.8912 lr=2.01e-05 tok=23.118B dt=9068 wait=379 +08:04:03 step 175540 loss=2.5350 lr=2.01e-05 tok=23.121B dt=9142 wait=379 +08:05:19 step 175560 loss=2.9057 lr=2.01e-05 tok=23.124B dt=9219 wait=379 +08:06:23 step 175580 loss=2.8631 lr=2.01e-05 tok=23.126B dt=9282 wait=379 +08:07:27 step 175600 loss=2.9526 lr=2.01e-05 tok=23.129B dt=9346 wait=379 +08:08:30 step 175620 loss=3.0081 lr=2.01e-05 tok=23.132B dt=9410 wait=379 +08:09:34 step 175640 loss=2.7158 lr=2.01e-05 tok=23.134B dt=9474 wait=380 +08:10:18 pushed ckpt_w167.pt,log_w167.txt +08:11:19 waited 61s for global r=727 +08:11:24 global r=727 load 4.286 -> 4.284 +08:11:58 step 175660 loss=2.9076 lr=2.01e-05 tok=23.137B dt=9618 wait=380 +08:13:01 step 175680 loss=2.9240 lr=2.01e-05 tok=23.139B dt=9681 wait=380 +08:14:29 step 175700 loss=2.8495 lr=2.01e-05 tok=23.142B dt=9769 wait=380 +08:15:35 step 175720 loss=2.9164 lr=2.00e-05 tok=23.145B dt=9834 wait=380 +08:16:38 step 175740 loss=2.9178 lr=2.00e-05 tok=23.147B dt=9898 wait=380 +08:17:42 step 175760 loss=3.0111 lr=2.00e-05 tok=23.150B dt=9962 wait=381 +08:18:46 step 175780 loss=2.9792 lr=2.00e-05 tok=23.153B dt=10026 wait=381 +08:19:50 step 175800 loss=3.0248 lr=2.00e-05 tok=23.155B dt=10090 wait=381 +08:20:05 pushed ckpt_w167.pt,log_w167.txt +08:21:05 waited 61s for global r=728 +08:21:19 global r=728 load 4.294 -> 4.285 +08:22:25 step 175820 loss=2.9956 lr=2.00e-05 tok=23.158B dt=10245 wait=381 +08:23:42 step 175840 loss=3.0931 lr=2.00e-05 tok=23.160B dt=10322 wait=381 +08:24:53 step 175860 loss=3.1342 lr=2.00e-05 tok=23.163B dt=10393 wait=381 +08:25:57 step 175880 loss=3.0902 lr=2.00e-05 tok=23.166B dt=10457 wait=382 +08:27:01 step 175900 loss=3.1278 lr=2.00e-05 tok=23.168B dt=10521 wait=382 +08:28:08 step 175920 loss=2.2915 lr=2.00e-05 tok=23.171B dt=10587 wait=382 +08:29:12 step 175940 loss=2.9663 lr=2.00e-05 tok=23.174B dt=10652 wait=382 +08:29:56 pushed ckpt_w167.pt,log_w167.txt +08:33:32 step 175960 loss=3.2103 lr=2.00e-05 tok=23.176B dt=10911 wait=382 +08:34:36 step 175980 loss=3.0140 lr=1.99e-05 tok=23.179B dt=10976 wait=382 +08:35:41 step 176000 loss=2.6201 lr=1.99e-05 tok=23.181B dt=11041 wait=383 +08:36:45 step 176020 loss=2.8622 lr=1.99e-05 tok=23.184B dt=11105 wait=383 +08:37:48 step 176040 loss=2.9929 lr=1.99e-05 tok=23.187B dt=11168 wait=383 +08:38:51 step 176060 loss=2.9327 lr=1.99e-05 tok=23.189B dt=11231 wait=383 +08:39:54 step 176080 loss=2.9187 lr=1.99e-05 tok=23.192B dt=11294 wait=383 +08:40:58 step 176100 loss=3.0484 lr=1.99e-05 tok=23.195B dt=11357 wait=383 +08:41:12 pushed ckpt_w167.pt,log_w167.txt +08:45:18 step 176120 loss=2.8773 lr=1.99e-05 tok=23.197B dt=11618 wait=384 +08:46:24 step 176140 loss=2.2707 lr=1.99e-05 tok=23.200B dt=11684 wait=384 +08:47:28 step 176160 loss=2.9546 lr=1.99e-05 tok=23.202B dt=11747 wait=384 +08:48:32 step 176180 loss=2.8770 lr=1.99e-05 tok=23.205B dt=11811 wait=384 +08:49:35 step 176200 loss=2.9047 lr=1.99e-05 tok=23.208B dt=11874 wait=384 +08:50:38 step 176220 loss=2.9021 lr=1.99e-05 tok=23.210B dt=11938 wait=384 +08:51:41 step 176240 loss=2.8572 lr=1.98e-05 tok=23.213B dt=12000 wait=385 +08:52:25 pushed ckpt_w167.pt,log_w167.txt +08:56:00 step 176260 loss=2.8482 lr=1.98e-05 tok=23.215B dt=12260 wait=385 +08:57:03 step 176280 loss=2.8708 lr=1.98e-05 tok=23.218B dt=12323 wait=385 +08:58:07 step 176300 loss=2.9520 lr=1.98e-05 tok=23.221B dt=12387 wait=385 +08:59:11 step 176320 loss=2.9036 lr=1.98e-05 tok=23.223B dt=12451 wait=385 +09:00:15 step 176340 loss=2.9487 lr=1.98e-05 tok=23.226B dt=12515 wait=385 +09:01:19 step 176360 loss=2.9204 lr=1.98e-05 tok=23.229B dt=12579 wait=386 +09:02:24 step 176380 loss=2.9978 lr=1.98e-05 tok=23.231B dt=12643 wait=386 +09:03:28 step 176400 loss=2.8636 lr=1.98e-05 tok=23.234B dt=12708 wait=386 +09:03:41 pushed ckpt_w167.pt,log_w167.txt +09:07:48 step 176420 loss=2.7972 lr=1.98e-05 tok=23.236B dt=12968 wait=386 +09:08:52 step 176440 loss=3.0189 lr=1.98e-05 tok=23.239B dt=13032 wait=386 +09:09:55 step 176460 loss=2.8617 lr=1.98e-05 tok=23.242B dt=13095 wait=386 +09:10:59 step 176480 loss=2.8643 lr=1.97e-05 tok=23.244B dt=13159 wait=387 +09:12:02 step 176500 loss=2.8883 lr=1.97e-05 tok=23.247B dt=13222 wait=387 +09:13:06 step 176520 loss=2.8891 lr=1.97e-05 tok=23.250B dt=13285 wait=387 +09:14:10 step 176540 loss=2.8127 lr=1.97e-05 tok=23.252B dt=13350 wait=387 +09:15:02 pushed ckpt_w167.pt,log_w167.txt +09:18:37 step 176560 loss=2.7998 lr=1.97e-05 tok=23.255B dt=13617 wait=387 +09:19:42 step 176580 loss=2.8181 lr=1.97e-05 tok=23.257B dt=13681 wait=388 +09:20:44 step 176600 loss=2.9495 lr=1.97e-05 tok=23.260B dt=13744 wait=388 +09:21:48 step 176620 loss=2.8869 lr=1.97e-05 tok=23.263B dt=13807 wait=388 +09:22:51 step 176640 loss=2.8389 lr=1.97e-05 tok=23.265B dt=13871 wait=388 +09:23:55 step 176660 loss=2.6890 lr=1.97e-05 tok=23.268B dt=13935 wait=388 +09:25:00 step 176680 loss=2.8475 lr=1.97e-05 tok=23.271B dt=13999 wait=388 +09:26:03 step 176700 loss=2.8994 lr=1.97e-05 tok=23.273B dt=14063 wait=388 +09:26:17 pushed ckpt_w167.pt,log_w167.txt +09:30:24 step 176720 loss=2.8145 lr=1.97e-05 tok=23.276B dt=14324 wait=389 +09:31:30 step 176740 loss=2.4832 lr=1.96e-05 tok=23.278B dt=14390 wait=389 +09:32:34 step 176760 loss=2.7791 lr=1.96e-05 tok=23.281B dt=14453 wait=389 +09:33:37 step 176780 loss=2.8550 lr=1.96e-05 tok=23.284B dt=14517 wait=389 +09:34:41 step 176800 loss=2.7676 lr=1.96e-05 tok=23.286B dt=14581 wait=389 +09:35:44 step 176820 loss=2.9346 lr=1.96e-05 tok=23.289B dt=14644 wait=389 +09:36:48 step 176840 loss=2.8745 lr=1.96e-05 tok=23.291B dt=14708 wait=390 +09:37:32 pushed ckpt_w167.pt,log_w167.txt +09:41:07 step 176860 loss=2.9026 lr=1.96e-05 tok=23.294B dt=14967 wait=390 +09:42:11 step 176880 loss=2.8964 lr=1.96e-05 tok=23.297B dt=15030 wait=390 +09:43:15 step 176900 loss=2.9561 lr=1.96e-05 tok=23.299B dt=15094 wait=390 +09:44:19 step 176920 loss=2.7464 lr=1.96e-05 tok=23.302B dt=15158 wait=390 +09:45:23 step 176940 loss=2.7864 lr=1.96e-05 tok=23.305B dt=15223 wait=390 +09:46:27 step 176960 loss=2.8062 lr=1.96e-05 tok=23.307B dt=15287 wait=391 +09:47:31 step 176980 loss=2.9006 lr=1.96e-05 tok=23.310B dt=15351 wait=391 +09:48:35 step 177000 loss=2.8684 lr=1.95e-05 tok=23.312B dt=15415 wait=391 +09:48:49 pushed ckpt_w167.pt,log_w167.txt +09:52:56 step 177020 loss=2.9230 lr=1.95e-05 tok=23.315B dt=15675 wait=391 +09:54:00 step 177040 loss=2.9472 lr=1.95e-05 tok=23.318B dt=15740 wait=391 +09:55:04 step 177060 loss=2.9830 lr=1.95e-05 tok=23.320B dt=15804 wait=391 +09:56:08 step 177080 loss=2.8909 lr=1.95e-05 tok=23.323B dt=15868 wait=391 +09:57:13 step 177100 loss=2.9971 lr=1.95e-05 tok=23.326B dt=15933 wait=392 +09:58:18 step 177120 loss=2.8954 lr=1.95e-05 tok=23.328B dt=15998 wait=392 +09:59:23 step 177140 loss=2.7224 lr=1.95e-05 tok=23.331B dt=16063 wait=392 +10:00:09 pushed ckpt_w167.pt,log_w167.txt +10:03:44 step 177160 loss=2.9750 lr=1.95e-05 tok=23.333B dt=16324 wait=392 +launch +12:20:45 name=w167 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 114/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:20:50 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:20:50 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:21:00 params 1.08B +12:21:01 resumed step=177150 seen=23.33B round=729 +12:21:01 stream salt=177150 +12:23:12 blocks compiled +12:23:12 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:29:11 step 177151 loss=2.8973 lr=1.95e-05 tok=23.332B dt=360 wait=350 +12:29:19 step 177152 loss=3.0087 lr=1.95e-05 tok=23.332B dt=367 wait=350 +12:29:44 step 177160 loss=3.2285 lr=1.95e-05 tok=23.333B dt=393 wait=350 +12:29:59 global r=730 load 4.283 -> 4.280 +12:31:03 step 177320 loss=3.0183 lr=1.94e-05 tok=23.336B dt=472 wait=351 +12:32:08 step 177340 loss=3.1114 lr=1.94e-05 tok=23.339B dt=537 wait=351 +12:33:12 step 177360 loss=3.1309 lr=1.94e-05 tok=23.341B dt=601 wait=351 +12:34:16 step 177380 loss=3.0222 lr=1.94e-05 tok=23.344B dt=665 wait=351 +12:35:22 step 177400 loss=2.7972 lr=1.94e-05 tok=23.347B dt=730 wait=351 +12:36:27 step 177420 loss=3.2272 lr=1.94e-05 tok=23.349B dt=795 wait=351 +12:37:07 global r=731 load 4.284 -> 4.278 +12:37:39 step 177460 loss=3.1184 lr=1.94e-05 tok=23.352B dt=867 wait=352 +12:38:51 step 177480 loss=3.0376 lr=1.94e-05 tok=23.354B dt=939 wait=352 +12:40:08 step 177500 loss=3.0769 lr=1.94e-05 tok=23.357B dt=1017 wait=352 +12:41:13 step 177520 loss=3.1217 lr=1.94e-05 tok=23.360B dt=1081 wait=352 +12:42:17 step 177540 loss=3.0876 lr=1.93e-05 tok=23.362B dt=1146 wait=352 +12:43:22 step 177560 loss=3.1663 lr=1.93e-05 tok=23.365B dt=1211 wait=352 +12:44:27 step 177580 loss=3.0338 lr=1.93e-05 tok=23.368B dt=1275 wait=353 +12:45:31 step 177600 loss=3.0977 lr=1.93e-05 tok=23.370B dt=1340 wait=353 +12:45:44 pushed ckpt_w167.pt,log_w167.txt +12:46:44 waited 60s for global r=732 +12:46:53 global r=732 load 4.279 -> 4.280 +12:47:58 step 177620 loss=3.0122 lr=1.93e-05 tok=23.373B dt=1487 wait=353 +12:49:21 step 177640 loss=2.9988 lr=1.93e-05 tok=23.375B dt=1570 wait=353 +12:50:27 step 177660 loss=3.1263 lr=1.93e-05 tok=23.378B dt=1636 wait=353 +12:51:32 step 177680 loss=3.0901 lr=1.93e-05 tok=23.381B dt=1700 wait=353 +12:52:36 step 177700 loss=3.1693 lr=1.93e-05 tok=23.383B dt=1764 wait=353 +12:53:39 step 177720 loss=3.1662 lr=1.93e-05 tok=23.386B dt=1828 wait=354 +12:54:44 step 177740 loss=2.8682 lr=1.93e-05 tok=23.388B dt=1892 wait=354 +12:55:30 pushed ckpt_w167.pt,log_w167.txt +12:56:16 waited 45s for global r=733 +12:56:22 global r=733 load 4.283 -> 4.282 +12:56:56 step 177760 loss=3.0362 lr=1.93e-05 tok=23.391B dt=2024 wait=354 +12:58:07 step 177780 loss=3.0152 lr=1.93e-05 tok=23.394B dt=2095 wait=354 +12:59:25 step 177800 loss=2.8370 lr=1.92e-05 tok=23.396B dt=2174 wait=354 +13:00:30 step 177820 loss=3.0312 lr=1.92e-05 tok=23.399B dt=2239 wait=354 +13:01:35 step 177840 loss=2.9018 lr=1.92e-05 tok=23.402B dt=2303 wait=355 +13:02:39 step 177860 loss=3.0358 lr=1.92e-05 tok=23.404B dt=2367 wait=355 +13:03:43 step 177880 loss=2.9948 lr=1.92e-05 tok=23.407B dt=2432 wait=355 +13:04:47 step 177900 loss=3.0821 lr=1.92e-05 tok=23.409B dt=2496 wait=355 +13:05:01 pushed ckpt_w167.pt,log_w167.txt +13:06:02 waited 60s for global r=734 +13:06:10 global r=734 load 4.282 -> 4.283 +13:07:15 step 177920 loss=2.9932 lr=1.92e-05 tok=23.412B dt=2644 wait=355 +13:08:37 step 177940 loss=2.9837 lr=1.92e-05 tok=23.415B dt=2725 wait=355 +13:09:44 step 177960 loss=2.9117 lr=1.92e-05 tok=23.417B dt=2793 wait=355 +13:10:48 step 177980 loss=2.7138 lr=1.92e-05 tok=23.420B dt=2856 wait=356 +13:11:51 step 178000 loss=2.8628 lr=1.92e-05 tok=23.423B dt=2919 wait=356 +13:12:54 step 178020 loss=2.9961 lr=1.92e-05 tok=23.425B dt=2983 wait=356 +13:13:58 step 178040 loss=2.8362 lr=1.92e-05 tok=23.428B dt=3047 wait=356 +13:14:44 pushed ckpt_w167.pt,log_w167.txt +13:15:45 waited 60s for global r=735 +13:15:52 global r=735 load 4.282 -> 4.283 +13:16:25 step 178060 loss=3.0036 lr=1.92e-05 tok=23.430B dt=3194 wait=356 +13:17:37 step 178080 loss=2.8920 lr=1.91e-05 tok=23.433B dt=3265 wait=356 +13:18:52 step 178100 loss=2.8338 lr=1.91e-05 tok=23.436B dt=3341 wait=356 +13:19:57 step 178120 loss=2.8778 lr=1.91e-05 tok=23.438B dt=3405 wait=357 +13:21:01 step 178140 loss=2.8377 lr=1.91e-05 tok=23.441B dt=3469 wait=357 +13:22:04 step 178160 loss=2.8802 lr=1.91e-05 tok=23.444B dt=3532 wait=357 +13:23:07 step 178180 loss=2.9026 lr=1.91e-05 tok=23.446B dt=3595 wait=357 +13:24:11 step 178200 loss=2.8845 lr=1.91e-05 tok=23.449B dt=3659 wait=357 +13:24:23 pushed ckpt_w167.pt,log_w167.txt +13:25:24 waited 61s for global r=736 +13:25:30 global r=736 load 4.284 -> 4.283 +13:26:35 step 178220 loss=2.8957 lr=1.91e-05 tok=23.451B dt=3804 wait=357 +13:27:58 step 178240 loss=2.9280 lr=1.91e-05 tok=23.454B dt=3886 wait=357 +13:29:04 step 178260 loss=2.8838 lr=1.91e-05 tok=23.457B dt=3952 wait=358 +13:30:08 step 178280 loss=2.9352 lr=1.91e-05 tok=23.459B dt=4016 wait=358 +13:31:12 step 178300 loss=2.9439 lr=1.91e-05 tok=23.462B dt=4081 wait=358 +13:32:16 step 178320 loss=3.0669 lr=1.91e-05 tok=23.465B dt=4144 wait=358 +13:33:20 step 178340 loss=2.9343 lr=1.90e-05 tok=23.467B dt=4208 wait=358 +13:34:06 pushed ckpt_w167.pt,log_w167.txt +13:35:06 waited 60s for global r=737 +13:35:14 global r=737 load 4.286 -> 4.282 +13:35:47 step 178360 loss=3.0008 lr=1.90e-05 tok=23.470B dt=4356 wait=358 +13:36:59 step 178380 loss=2.8557 lr=1.90e-05 tok=23.472B dt=4428 wait=358 +13:38:15 step 178400 loss=2.8868 lr=1.90e-05 tok=23.475B dt=4503 wait=359 +13:39:18 step 178420 loss=2.9771 lr=1.90e-05 tok=23.478B dt=4567 wait=359 +13:40:22 step 178440 loss=2.8162 lr=1.90e-05 tok=23.480B dt=4630 wait=359 +13:41:26 step 178460 loss=2.8575 lr=1.90e-05 tok=23.483B dt=4694 wait=359 +13:42:29 step 178480 loss=2.8254 lr=1.90e-05 tok=23.485B dt=4758 wait=359 +13:43:33 step 178500 loss=2.8935 lr=1.90e-05 tok=23.488B dt=4821 wait=359 +13:43:45 pushed ckpt_w167.pt,log_w167.txt +13:44:46 waited 61s for global r=738 +13:44:51 global r=738 load 4.287 -> 4.284 +13:45:56 step 178520 loss=2.8286 lr=1.90e-05 tok=23.491B dt=4965 wait=360 +13:47:19 step 178540 loss=2.8998 lr=1.90e-05 tok=23.493B dt=5047 wait=360 +13:48:23 step 178560 loss=2.8628 lr=1.90e-05 tok=23.496B dt=5112 wait=360 +13:49:27 step 178580 loss=2.7057 lr=1.90e-05 tok=23.499B dt=5176 wait=360 +13:50:32 step 178600 loss=2.7047 lr=1.90e-05 tok=23.501B dt=5240 wait=360 +13:51:35 step 178620 loss=2.9514 lr=1.89e-05 tok=23.504B dt=5304 wait=360 +13:52:39 step 178640 loss=2.7758 lr=1.89e-05 tok=23.506B dt=5368 wait=360 +13:53:23 pushed ckpt_w167.pt,log_w167.txt +13:54:24 waited 61s for global r=739 +13:54:33 global r=739 load 4.284 -> 4.282 +13:55:08 step 178660 loss=2.8529 lr=1.89e-05 tok=23.509B dt=5516 wait=361 +13:56:20 step 178680 loss=2.8596 lr=1.89e-05 tok=23.512B dt=5589 wait=361 +13:57:37 step 178700 loss=2.8788 lr=1.89e-05 tok=23.514B dt=5665 wait=361 +13:58:41 step 178720 loss=2.9360 lr=1.89e-05 tok=23.517B dt=5729 wait=361 +13:59:45 step 178740 loss=2.9165 lr=1.89e-05 tok=23.520B dt=5793 wait=361 +14:00:48 step 178760 loss=2.9329 lr=1.89e-05 tok=23.522B dt=5857 wait=361 +14:01:53 step 178780 loss=2.8656 lr=1.89e-05 tok=23.525B dt=5921 wait=362 +14:02:57 step 178800 loss=2.9083 lr=1.89e-05 tok=23.527B dt=5986 wait=362 +14:03:09 pushed ckpt_w167.pt,log_w167.txt +14:04:10 waited 60s for global r=740 +14:04:16 global r=740 load 4.282 -> 4.279 +14:05:28 step 178820 loss=2.9241 lr=1.89e-05 tok=23.530B dt=6136 wait=362 +14:06:47 step 178840 loss=3.0316 lr=1.89e-05 tok=23.533B dt=6215 wait=362 +14:07:51 step 178860 loss=3.1350 lr=1.89e-05 tok=23.535B dt=6280 wait=362 +14:08:55 step 178880 loss=3.1123 lr=1.89e-05 tok=23.538B dt=6344 wait=362 +14:10:00 step 178900 loss=2.9439 lr=1.88e-05 tok=23.541B dt=6408 wait=362 +14:11:04 step 178920 loss=3.0530 lr=1.88e-05 tok=23.543B dt=6472 wait=363 +14:12:08 step 178940 loss=2.9403 lr=1.88e-05 tok=23.546B dt=6536 wait=363 +14:12:58 pushed ckpt_w167.pt,log_w167.txt +14:13:58 waited 60s for global r=741 +14:14:06 global r=741 load 4.280 -> 4.280 +14:14:40 step 178960 loss=3.0431 lr=1.88e-05 tok=23.548B dt=6688 wait=363 +14:15:57 step 178980 loss=2.9677 lr=1.88e-05 tok=23.551B dt=6766 wait=363 +14:17:07 step 179000 loss=2.8806 lr=1.88e-05 tok=23.554B dt=6836 wait=363 +14:18:11 step 179020 loss=2.9874 lr=1.88e-05 tok=23.556B dt=6899 wait=363 +14:19:15 step 179040 loss=2.8763 lr=1.88e-05 tok=23.559B dt=6963 wait=363 +14:20:19 step 179060 loss=2.9973 lr=1.88e-05 tok=23.562B dt=7027 wait=364 +14:21:23 step 179080 loss=2.8815 lr=1.88e-05 tok=23.564B dt=7091 wait=364 +14:22:27 step 179100 loss=2.9444 lr=1.88e-05 tok=23.567B dt=7155 wait=364 +14:22:39 pushed ckpt_w167.pt,log_w167.txt +14:23:40 waited 61s for global r=742 +14:23:49 global r=742 load 4.279 -> 4.282 +14:25:09 step 179120 loss=2.7362 lr=1.88e-05 tok=23.569B dt=7317 wait=364 +14:26:27 step 179140 loss=2.8118 lr=1.88e-05 tok=23.572B dt=7395 wait=364 +14:27:31 step 179160 loss=2.8738 lr=1.88e-05 tok=23.575B dt=7460 wait=364 +14:28:35 step 179180 loss=2.8080 lr=1.88e-05 tok=23.577B dt=7523 wait=364 +14:29:39 step 179200 loss=2.9327 lr=1.87e-05 tok=23.580B dt=7587 wait=365 +14:30:43 step 179220 loss=2.9855 lr=1.87e-05 tok=23.582B dt=7651 wait=365 +14:31:46 step 179240 loss=2.8340 lr=1.87e-05 tok=23.585B dt=7715 wait=365 +14:32:31 pushed ckpt_w167.pt,log_w167.txt +14:33:16 waited 45s for global r=743 +14:33:24 global r=743 load 4.284 -> 4.281 +14:33:57 step 179260 loss=2.9073 lr=1.87e-05 tok=23.588B dt=7846 wait=365 +14:35:14 step 179280 loss=2.8129 lr=1.87e-05 tok=23.590B dt=7923 wait=365 +14:36:25 step 179300 loss=2.8113 lr=1.87e-05 tok=23.593B dt=7994 wait=365 +14:37:29 step 179320 loss=2.9021 lr=1.87e-05 tok=23.596B dt=8057 wait=365 +14:38:33 step 179340 loss=2.7939 lr=1.87e-05 tok=23.598B dt=8121 wait=366 +14:39:37 step 179360 loss=2.8088 lr=1.87e-05 tok=23.601B dt=8185 wait=366 +14:40:40 step 179380 loss=2.8075 lr=1.87e-05 tok=23.603B dt=8249 wait=366 +14:41:44 step 179400 loss=2.8015 lr=1.87e-05 tok=23.606B dt=8312 wait=366 +14:41:55 pushed ckpt_w167.pt,log_w167.txt +14:42:56 waited 61s for global r=744 +14:43:03 global r=744 load 4.282 -> 4.280 +14:44:15 step 179420 loss=2.6950 lr=1.87e-05 tok=23.609B dt=8463 wait=366 +14:45:33 step 179440 loss=2.8067 lr=1.87e-05 tok=23.611B dt=8541 wait=366 +14:46:37 step 179460 loss=2.7785 lr=1.87e-05 tok=23.614B dt=8605 wait=366 +14:47:41 step 179480 loss=2.6518 lr=1.86e-05 tok=23.617B dt=8669 wait=367 +14:48:45 step 179500 loss=2.7497 lr=1.86e-05 tok=23.619B dt=8733 wait=367 +14:49:49 step 179520 loss=2.8911 lr=1.86e-05 tok=23.622B dt=8797 wait=367 +14:50:52 step 179540 loss=2.6901 lr=1.86e-05 tok=23.624B dt=8861 wait=367 +14:51:38 pushed ckpt_w167.pt,log_w167.txt +14:52:38 waited 61s for global r=745 +14:52:45 global r=745 load 4.289 -> 4.281 +14:53:19 step 179560 loss=2.8411 lr=1.86e-05 tok=23.627B dt=9008 wait=367 +14:54:37 step 179580 loss=2.7993 lr=1.86e-05 tok=23.630B dt=9085 wait=367 +14:55:48 step 179600 loss=2.7539 lr=1.86e-05 tok=23.632B dt=9156 wait=367 +14:56:52 step 179620 loss=2.8434 lr=1.86e-05 tok=23.635B dt=9220 wait=368 +14:57:56 step 179640 loss=2.7739 lr=1.86e-05 tok=23.638B dt=9284 wait=368 +14:59:00 step 179660 loss=2.7987 lr=1.86e-05 tok=23.640B dt=9348 wait=368 +15:00:04 step 179680 loss=2.8029 lr=1.86e-05 tok=23.643B dt=9413 wait=368 +15:01:09 step 179700 loss=2.7665 lr=1.86e-05 tok=23.645B dt=9477 wait=368 +15:01:21 pushed ckpt_w167.pt,log_w167.txt +15:05:29 step 179720 loss=2.8068 lr=1.86e-05 tok=23.648B dt=9737 wait=368 +15:06:32 step 179740 loss=2.8195 lr=1.86e-05 tok=23.651B dt=9801 wait=368 +15:07:37 step 179760 loss=2.7376 lr=1.86e-05 tok=23.653B dt=9865 wait=368 +15:08:41 step 179780 loss=2.6882 lr=1.85e-05 tok=23.656B dt=9930 wait=369 +15:09:45 step 179800 loss=2.7179 lr=1.85e-05 tok=23.658B dt=9993 wait=369 +15:10:50 step 179820 loss=2.8298 lr=1.85e-05 tok=23.661B dt=10058 wait=369 +15:11:54 step 179840 loss=2.7367 lr=1.85e-05 tok=23.664B dt=10122 wait=369 +15:12:40 pushed ckpt_w167.pt,log_w167.txt +15:16:16 step 179860 loss=2.8904 lr=1.85e-05 tok=23.666B dt=10384 wait=369 +15:17:21 step 179880 loss=2.9017 lr=1.85e-05 tok=23.669B dt=10449 wait=369 +15:18:25 step 179900 loss=2.8410 lr=1.85e-05 tok=23.672B dt=10513 wait=369 +15:19:29 step 179920 loss=2.8291 lr=1.85e-05 tok=23.674B dt=10577 wait=370 +15:19:37 global r=746 load 4.287 -> 4.282 +15:20:09 step 179860 loss=2.8311 lr=1.85e-05 tok=23.676B dt=10617 wait=370 +15:21:14 step 179880 loss=2.7429 lr=1.85e-05 tok=23.678B dt=10682 wait=370 +15:22:32 step 179900 loss=2.7758 lr=1.85e-05 tok=23.681B dt=10761 wait=370 +15:23:43 step 179920 loss=2.7837 lr=1.85e-05 tok=23.683B dt=10831 wait=370 +15:24:47 step 179940 loss=2.7726 lr=1.85e-05 tok=23.686B dt=10896 wait=370 +15:25:53 step 179960 loss=2.8414 lr=1.85e-05 tok=23.689B dt=10961 wait=370 +15:26:57 step 179980 loss=2.8254 lr=1.85e-05 tok=23.691B dt=11026 wait=370 +15:28:03 step 180000 loss=2.8334 lr=1.85e-05 tok=23.694B dt=11091 wait=371 +15:28:15 pushed ckpt_w167.pt,log_w167.txt +15:31:01 waited 166s for global r=747 +15:31:09 global r=747 load 4.287 -> 4.283 +15:32:35 step 180020 loss=2.8264 lr=1.85e-05 tok=23.697B dt=11364 wait=371 +15:33:42 step 180040 loss=2.8790 lr=1.85e-05 tok=23.699B dt=11430 wait=371 +15:34:47 step 180060 loss=2.9258 lr=1.84e-05 tok=23.702B dt=11495 wait=371 +15:35:52 step 180080 loss=2.8012 lr=1.84e-05 tok=23.704B dt=11561 wait=371 +15:36:58 step 180100 loss=2.8791 lr=1.84e-05 tok=23.707B dt=11626 wait=371 +15:38:03 step 180120 loss=3.0222 lr=1.84e-05 tok=23.710B dt=11691 wait=372 +15:39:07 step 180140 loss=2.8228 lr=1.84e-05 tok=23.712B dt=11756 wait=372 +15:39:54 pushed ckpt_w167.pt,log_w167.txt +15:42:40 waited 166s for global r=748 +15:42:49 global r=748 load 4.284 -> 4.280 +15:43:32 step 180160 loss=2.8465 lr=1.84e-05 tok=23.715B dt=12021 wait=372 +15:44:50 step 180180 loss=2.8400 lr=1.84e-05 tok=23.717B dt=12098 wait=372 +15:45:55 step 180200 loss=2.7540 lr=1.84e-05 tok=23.720B dt=12164 wait=372 +15:46:59 step 180220 loss=3.0713 lr=1.84e-05 tok=23.723B dt=12228 wait=372 +15:48:04 step 180240 loss=2.8251 lr=1.84e-05 tok=23.725B dt=12293 wait=372 +15:49:09 step 180260 loss=3.0307 lr=1.84e-05 tok=23.728B dt=12358 wait=373 +15:50:14 step 180280 loss=3.0029 lr=1.84e-05 tok=23.731B dt=12422 wait=373 +15:51:18 step 180300 loss=3.0446 lr=1.84e-05 tok=23.733B dt=12487 wait=373 +15:51:34 pushed ckpt_w167.pt,log_w167.txt +15:54:06 waited 151s for global r=749 +15:54:13 global r=749 load 4.284 -> 4.282 +15:55:27 step 180320 loss=2.9594 lr=1.84e-05 tok=23.736B dt=12736 wait=373 +15:56:44 step 180340 loss=2.9923 lr=1.84e-05 tok=23.738B dt=12812 wait=373 +15:57:48 step 180360 loss=2.8542 lr=1.84e-05 tok=23.741B dt=12876 wait=373 +15:58:53 step 180380 loss=2.9526 lr=1.83e-05 tok=23.744B dt=12941 wait=374 +15:59:57 step 180400 loss=2.9466 lr=1.83e-05 tok=23.746B dt=13005 wait=374 +16:01:01 step 180420 loss=2.9646 lr=1.83e-05 tok=23.749B dt=13069 wait=374 +16:02:05 step 180440 loss=2.8591 lr=1.83e-05 tok=23.752B dt=13134 wait=374 +16:02:49 pushed ckpt_w167.pt,log_w167.txt +16:05:50 waited 181s for global r=750 +16:05:57 global r=750 load 4.285 -> 4.283 +16:06:33 step 180460 loss=3.0081 lr=1.83e-05 tok=23.754B dt=13401 wait=374 +16:07:55 step 180480 loss=2.9262 lr=1.83e-05 tok=23.757B dt=13484 wait=374 +16:08:59 step 180500 loss=2.9835 lr=1.83e-05 tok=23.759B dt=13547 wait=374 +16:10:03 step 180520 loss=2.9836 lr=1.83e-05 tok=23.762B dt=13611 wait=375 +16:11:07 step 180540 loss=2.8657 lr=1.83e-05 tok=23.765B dt=13675 wait=375 +16:12:11 step 180560 loss=2.9471 lr=1.83e-05 tok=23.767B dt=13739 wait=375 +16:13:15 step 180580 loss=2.9314 lr=1.83e-05 tok=23.770B dt=13803 wait=375 +16:14:19 step 180600 loss=2.8554 lr=1.83e-05 tok=23.773B dt=13867 wait=375 +16:14:31 pushed ckpt_w167.pt,log_w167.txt +16:15:47 waited 76s for global r=751 +16:15:52 global r=751 load 4.284 -> 4.282 +16:17:12 step 180620 loss=2.9487 lr=1.83e-05 tok=23.775B dt=14040 wait=375 +16:18:22 step 180640 loss=2.8974 lr=1.83e-05 tok=23.778B dt=14110 wait=375 +16:19:27 step 180660 loss=2.5685 lr=1.83e-05 tok=23.780B dt=14175 wait=376 +16:20:31 step 180680 loss=2.8191 lr=1.82e-05 tok=23.783B dt=14239 wait=376 +16:21:35 step 180700 loss=2.9635 lr=1.82e-05 tok=23.786B dt=14303 wait=376 +16:22:38 step 180720 loss=2.9820 lr=1.82e-05 tok=23.788B dt=14367 wait=376 +16:23:42 step 180740 loss=2.9135 lr=1.82e-05 tok=23.791B dt=14431 wait=376 +16:24:29 pushed ckpt_w167.pt,log_w167.txt +16:27:00 waited 151s for global r=752 +16:27:09 global r=752 load 4.274 -> 4.281 +16:27:42 step 180760 loss=2.9876 lr=1.82e-05 tok=23.794B dt=14671 wait=376 +16:29:05 step 180780 loss=2.8863 lr=1.82e-05 tok=23.796B dt=14753 wait=376 +16:30:12 step 180800 loss=2.8850 lr=1.82e-05 tok=23.799B dt=14820 wait=377 +16:31:16 step 180820 loss=3.0622 lr=1.82e-05 tok=23.801B dt=14884 wait=377 +16:32:19 step 180840 loss=2.8972 lr=1.82e-05 tok=23.804B dt=14948 wait=377 +16:33:23 step 180860 loss=2.9215 lr=1.82e-05 tok=23.807B dt=15011 wait=377 +16:34:26 step 180880 loss=2.9172 lr=1.82e-05 tok=23.809B dt=15074 wait=377 +16:35:30 step 180900 loss=2.8824 lr=1.82e-05 tok=23.812B dt=15138 wait=377 +16:35:44 pushed ckpt_w167.pt,log_w167.txt +16:39:31 global r=753 load 4.290 -> 4.284 +16:40:56 step 180920 loss=2.8565 lr=1.82e-05 tok=23.816B dt=15465 wait=378 +launch +16:41:29 name=w167 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 114/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:41:33 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:41:33 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:41:43 params 1.08B +16:41:45 resumed step=180900 seen=23.84B round=753 +16:41:45 stream salt=180900 +16:43:41 blocks compiled +16:43:41 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:49:00 step 180901 loss=2.8877 lr=1.76e-05 tok=23.838B dt=319 wait=308 +16:49:07 step 180902 loss=2.8949 lr=1.76e-05 tok=23.838B dt=327 wait=308 +16:50:05 step 180920 loss=3.1662 lr=1.76e-05 tok=23.841B dt=384 wait=308 +16:50:51 global r=754 load 4.286 -> 4.281 +16:51:30 step 181060 loss=3.0804 lr=1.75e-05 tok=23.843B dt=470 wait=308 +16:52:49 step 181080 loss=3.0415 lr=1.75e-05 tok=23.846B dt=548 wait=308 +16:53:54 step 181100 loss=2.8458 lr=1.75e-05 tok=23.849B dt=613 wait=308 +16:54:58 step 181120 loss=3.2577 lr=1.74e-05 tok=23.851B dt=678 wait=308 +16:56:04 step 181140 loss=3.1548 lr=1.74e-05 tok=23.854B dt=743 wait=309 +16:57:09 step 181160 loss=3.0823 lr=1.74e-05 tok=23.856B dt=808 wait=309 +16:58:13 step 181180 loss=3.0878 lr=1.74e-05 tok=23.859B dt=872 wait=309 +16:59:19 step 181200 loss=2.9552 lr=1.74e-05 tok=23.862B dt=938 wait=309 +16:59:34 pushed ckpt_w167.pt,log_w167.txt +17:01:05 waited 91s for global r=755 +17:01:13 global r=755 load 4.279 -> 4.280 +17:02:20 step 181220 loss=3.0046 lr=1.73e-05 tok=23.864B dt=1119 wait=309 +17:03:33 step 181240 loss=3.1586 lr=1.73e-05 tok=23.867B dt=1192 wait=309 +17:04:50 step 181260 loss=3.0898 lr=1.73e-05 tok=23.870B dt=1269 wait=310 +17:05:54 step 181280 loss=2.9197 lr=1.73e-05 tok=23.872B dt=1334 wait=310 +17:06:59 step 181300 loss=2.9569 lr=1.73e-05 tok=23.875B dt=1398 wait=310 +17:08:03 step 181320 loss=3.1664 lr=1.73e-05 tok=23.877B dt=1462 wait=310 +17:09:07 step 181340 loss=2.9270 lr=1.72e-05 tok=23.880B dt=1526 wait=310 diff --git a/log_w168.txt b/log_w168.txt new file mode 100644 index 0000000000000000000000000000000000000000..9111f440b60ac2b2a8430a8e55a1e43ff48bb5f3 --- /dev/null +++ b/log_w168.txt @@ -0,0 +1,193 @@ +launch +05:30:07 name=w168 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 115/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +05:30:13 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +05:30:13 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +05:30:23 params 1.08B +05:30:25 resumed step=173250 seen=22.82B round=711 +05:30:25 stream salt=173250 +05:32:08 blocks compiled +05:32:08 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:34:14 mix source 4 paused 278s ((Request ID: Root=1-6abc9f56-4aa5142f1c4060f42216def9;ac81ca3f-129c-46ca-954e-00) +05:34:15 mix source 6 paused 379s ((Request ID: Root=1-6abc9f57-31f0fbc2119b33e63f59743e;cbdf97c2-ab2d-40d7-963d-e5) +05:34:15 mix source 8 paused 372s ((Request ID: Root=1-6abc9f57-084844bb453e70ed22cc60a7;2f799c0c-9b64-4a60-bbf0-98) +05:34:15 mix source 7 paused 373s ((Request ID: Root=1-6abc9f57-6f37f90f13a0e22a108bed49;619faaea-2c73-4d0b-87be-96) +05:34:15 mix source 5 paused 305s ((Request ID: Root=1-6abc9f57-0e684c091773621458ceddb2;395b5ee0-d500-41e8-a017-f7) +05:34:15 mix source 3 paused 339s ((Request ID: Root=1-6abc9f57-7f7906241b63058e69879001;fdf6dd0b-e793-46a9-bdfa-57) +05:34:48 step 173251 loss=3.1109 lr=2.11e-05 tok=22.821B dt=160 wait=127 +05:35:04 step 173252 loss=3.1026 lr=2.11e-05 tok=22.821B dt=176 wait=127 +05:35:29 step 173260 loss=3.6458 lr=2.11e-05 tok=22.822B dt=201 wait=127 +05:36:32 step 173280 loss=3.3649 lr=2.11e-05 tok=22.825B dt=264 wait=127 +05:37:36 step 173300 loss=3.4041 lr=2.11e-05 tok=22.827B dt=328 wait=128 +05:38:39 step 173320 loss=3.4049 lr=2.11e-05 tok=22.830B dt=390 wait=128 +05:39:43 step 173340 loss=3.4148 lr=2.11e-05 tok=22.833B dt=455 wait=129 +05:41:15 step 173360 loss=3.2542 lr=2.10e-05 tok=22.835B dt=547 wait=158 +05:42:19 step 173380 loss=2.9571 lr=2.10e-05 tok=22.838B dt=611 wait=158 +05:43:23 step 173400 loss=3.0376 lr=2.10e-05 tok=22.841B dt=675 wait=158 +05:43:41 pushed ckpt_w168.pt,log_w168.txt +05:46:27 waited 166s for global r=712 +05:46:39 global r=712 load 4.289 -> 4.280 +05:47:46 step 173420 loss=2.9549 lr=2.10e-05 tok=22.843B dt=938 wait=158 +05:48:50 step 173440 loss=3.0122 lr=2.10e-05 tok=22.846B dt=1002 wait=158 +05:49:53 step 173460 loss=3.0785 lr=2.10e-05 tok=22.848B dt=1065 wait=158 +05:50:58 step 173480 loss=2.9993 lr=2.10e-05 tok=22.851B dt=1129 wait=159 +05:52:02 step 173500 loss=2.9835 lr=2.10e-05 tok=22.854B dt=1193 wait=159 +05:53:06 step 173520 loss=3.0894 lr=2.10e-05 tok=22.856B dt=1258 wait=159 +05:54:10 step 173540 loss=2.9387 lr=2.10e-05 tok=22.859B dt=1322 wait=159 +05:54:55 pushed ckpt_w168.pt,log_w168.txt +05:56:11 waited 76s for global r=713 +05:56:19 global r=713 load 4.277 -> 4.280 +05:56:53 step 173560 loss=3.0685 lr=2.10e-05 tok=22.862B dt=1485 wait=159 +05:57:57 step 173580 loss=2.9848 lr=2.09e-05 tok=22.864B dt=1549 wait=159 +05:59:02 step 173600 loss=3.0081 lr=2.09e-05 tok=22.867B dt=1614 wait=160 +06:00:06 step 173620 loss=3.0425 lr=2.09e-05 tok=22.869B dt=1678 wait=160 +06:01:10 step 173640 loss=2.9615 lr=2.09e-05 tok=22.872B dt=1742 wait=160 +06:02:28 step 173660 loss=2.9939 lr=2.09e-05 tok=22.875B dt=1820 wait=174 +06:03:32 step 173680 loss=2.9595 lr=2.09e-05 tok=22.877B dt=1884 wait=174 +06:04:36 step 173700 loss=2.9605 lr=2.09e-05 tok=22.880B dt=1948 wait=174 +06:04:50 pushed ckpt_w168.pt,log_w168.txt +06:05:50 waited 61s for global r=714 +06:05:57 global r=714 load 4.286 -> 4.281 +06:07:01 step 173720 loss=2.9417 lr=2.09e-05 tok=22.883B dt=2093 wait=174 +06:08:06 step 173740 loss=2.9738 lr=2.09e-05 tok=22.885B dt=2158 wait=174 +06:09:11 step 173760 loss=2.9803 lr=2.09e-05 tok=22.888B dt=2223 wait=175 +06:10:16 step 173780 loss=2.9883 lr=2.09e-05 tok=22.890B dt=2288 wait=175 +06:11:21 step 173800 loss=3.0042 lr=2.09e-05 tok=22.893B dt=2353 wait=175 +06:12:26 step 173820 loss=3.1468 lr=2.08e-05 tok=22.896B dt=2418 wait=175 +06:13:31 step 173840 loss=3.0968 lr=2.08e-05 tok=22.898B dt=2482 wait=175 +06:14:15 pushed ckpt_w168.pt,log_w168.txt +06:15:31 waited 76s for global r=715 +06:15:36 global r=715 load 4.277 -> 4.281 +06:16:09 step 173860 loss=3.0913 lr=2.08e-05 tok=22.901B dt=2641 wait=175 +06:17:14 step 173880 loss=2.9852 lr=2.08e-05 tok=22.904B dt=2705 wait=176 +06:18:18 step 173900 loss=2.9799 lr=2.08e-05 tok=22.906B dt=2770 wait=176 +06:19:23 step 173920 loss=3.0270 lr=2.08e-05 tok=22.909B dt=2835 wait=176 +06:20:27 step 173940 loss=2.9726 lr=2.08e-05 tok=22.911B dt=2899 wait=176 +06:21:32 step 173960 loss=3.1105 lr=2.08e-05 tok=22.914B dt=2964 wait=176 +06:22:37 step 173980 loss=2.9665 lr=2.08e-05 tok=22.917B dt=3029 wait=176 +06:23:41 step 174000 loss=2.9946 lr=2.08e-05 tok=22.919B dt=3093 wait=176 +06:23:53 pushed ckpt_w168.pt,log_w168.txt +06:25:09 waited 76s for global r=716 +06:25:15 global r=716 load 4.280 -> 4.282 +06:26:20 step 174020 loss=2.9880 lr=2.08e-05 tok=22.922B dt=3252 wait=177 +06:27:25 step 174040 loss=2.9620 lr=2.07e-05 tok=22.924B dt=3316 wait=177 +06:28:31 step 174060 loss=2.4651 lr=2.07e-05 tok=22.927B dt=3382 wait=177 +06:29:35 step 174080 loss=2.9466 lr=2.07e-05 tok=22.930B dt=3447 wait=177 +06:30:39 step 174100 loss=3.0069 lr=2.07e-05 tok=22.932B dt=3511 wait=177 +06:31:42 step 174120 loss=3.0066 lr=2.07e-05 tok=22.935B dt=3574 wait=177 +06:32:46 step 174140 loss=2.8537 lr=2.07e-05 tok=22.938B dt=3638 wait=178 +06:33:32 pushed ckpt_w168.pt,log_w168.txt +06:34:48 waited 76s for global r=717 +06:34:55 global r=717 load 4.280 -> 4.279 +06:35:28 step 174160 loss=3.0277 lr=2.07e-05 tok=22.940B dt=3800 wait=178 +06:36:31 step 174180 loss=2.9116 lr=2.07e-05 tok=22.943B dt=3863 wait=178 +06:37:35 step 174200 loss=2.9421 lr=2.07e-05 tok=22.945B dt=3927 wait=178 +06:38:39 step 174220 loss=2.9959 lr=2.07e-05 tok=22.948B dt=3991 wait=178 +06:39:43 step 174240 loss=2.8930 lr=2.07e-05 tok=22.951B dt=4055 wait=178 +06:40:46 step 174260 loss=2.9897 lr=2.07e-05 tok=22.953B dt=4118 wait=178 +06:41:50 step 174280 loss=2.9549 lr=2.06e-05 tok=22.956B dt=4182 wait=179 +06:42:54 step 174300 loss=2.9033 lr=2.06e-05 tok=22.959B dt=4246 wait=179 +06:43:08 pushed ckpt_w168.pt,log_w168.txt +06:44:23 waited 76s for global r=718 +06:44:29 global r=718 load 4.289 -> 4.281 +06:45:34 step 174320 loss=2.8840 lr=2.06e-05 tok=22.961B dt=4406 wait=179 +06:46:38 step 174340 loss=2.9470 lr=2.06e-05 tok=22.964B dt=4470 wait=179 +06:47:42 step 174360 loss=2.9291 lr=2.06e-05 tok=22.966B dt=4534 wait=179 +06:48:46 step 174380 loss=2.9290 lr=2.06e-05 tok=22.969B dt=4598 wait=179 +06:49:51 step 174400 loss=2.9567 lr=2.06e-05 tok=22.972B dt=4663 wait=180 +06:50:55 step 174420 loss=3.0536 lr=2.06e-05 tok=22.974B dt=4727 wait=180 +06:52:00 step 174440 loss=2.8885 lr=2.06e-05 tok=22.977B dt=4792 wait=180 +06:52:44 pushed ckpt_w168.pt,log_w168.txt +06:54:00 waited 76s for global r=719 +06:54:05 global r=719 load 4.282 -> 4.278 +06:54:39 step 174460 loss=2.9968 lr=2.06e-05 tok=22.980B dt=4950 wait=180 +06:55:42 step 174480 loss=2.9779 lr=2.06e-05 tok=22.982B dt=5014 wait=180 +06:56:47 step 174500 loss=2.9284 lr=2.06e-05 tok=22.985B dt=5078 wait=180 +06:57:51 step 174520 loss=3.0219 lr=2.05e-05 tok=22.987B dt=5143 wait=180 +06:58:56 step 174540 loss=2.9187 lr=2.05e-05 tok=22.990B dt=5208 wait=181 +07:00:00 step 174560 loss=3.1039 lr=2.05e-05 tok=22.993B dt=5272 wait=181 +07:01:04 step 174580 loss=2.8967 lr=2.05e-05 tok=22.995B dt=5336 wait=181 +07:02:08 step 174600 loss=2.9523 lr=2.05e-05 tok=22.998B dt=5400 wait=181 +07:02:20 pushed ckpt_w168.pt,log_w168.txt +07:03:36 waited 76s for global r=720 +07:03:41 global r=720 load 4.290 -> 4.282 +07:04:46 step 174620 loss=2.9699 lr=2.05e-05 tok=23.001B dt=5558 wait=181 +07:05:50 step 174640 loss=2.9391 lr=2.05e-05 tok=23.003B dt=5622 wait=181 +07:06:55 step 174660 loss=3.0146 lr=2.05e-05 tok=23.006B dt=5687 wait=181 +07:07:59 step 174680 loss=2.9604 lr=2.05e-05 tok=23.008B dt=5751 wait=181 +07:09:03 step 174700 loss=2.9550 lr=2.05e-05 tok=23.011B dt=5815 wait=182 +07:10:08 step 174720 loss=3.0345 lr=2.05e-05 tok=23.014B dt=5880 wait=182 +07:11:12 step 174740 loss=2.8338 lr=2.05e-05 tok=23.016B dt=5943 wait=182 +07:12:00 pushed ckpt_w168.pt,log_w168.txt +07:13:31 waited 91s for global r=721 +07:13:37 global r=721 load 4.287 -> 4.282 +07:14:10 step 174760 loss=3.0596 lr=2.04e-05 tok=23.019B dt=6122 wait=182 +07:15:14 step 174780 loss=2.9373 lr=2.04e-05 tok=23.021B dt=6186 wait=182 +07:16:18 step 174800 loss=2.9918 lr=2.04e-05 tok=23.024B dt=6250 wait=182 +07:17:22 step 174820 loss=3.0323 lr=2.04e-05 tok=23.027B dt=6313 wait=183 +07:18:26 step 174840 loss=2.9260 lr=2.04e-05 tok=23.029B dt=6378 wait=183 +07:19:30 step 174860 loss=2.9997 lr=2.04e-05 tok=23.032B dt=6442 wait=183 +07:20:34 step 174880 loss=2.8891 lr=2.04e-05 tok=23.035B dt=6506 wait=183 +07:21:39 step 174900 loss=2.9189 lr=2.04e-05 tok=23.037B dt=6571 wait=183 +07:21:51 pushed ckpt_w168.pt,log_w168.txt +07:23:07 waited 76s for global r=722 +07:23:14 global r=722 load 4.283 -> 4.284 +07:24:19 step 174920 loss=2.9479 lr=2.04e-05 tok=23.040B dt=6731 wait=183 +07:25:23 step 174940 loss=2.9690 lr=2.04e-05 tok=23.042B dt=6795 wait=183 +07:26:27 step 174960 loss=2.8790 lr=2.04e-05 tok=23.045B dt=6859 wait=184 +07:27:31 step 174980 loss=2.8259 lr=2.04e-05 tok=23.048B dt=6923 wait=184 +07:28:35 step 175000 loss=2.8695 lr=2.03e-05 tok=23.050B dt=6987 wait=184 +07:29:39 step 175020 loss=2.9330 lr=2.03e-05 tok=23.053B dt=7051 wait=184 +07:30:44 step 175040 loss=2.7951 lr=2.03e-05 tok=23.056B dt=7115 wait=184 +07:31:30 pushed ckpt_w168.pt,log_w168.txt +07:32:46 waited 76s for global r=723 +07:32:53 global r=723 load 4.290 -> 4.282 +07:33:26 step 175060 loss=2.8727 lr=2.03e-05 tok=23.058B dt=7278 wait=184 +07:34:30 step 175080 loss=2.7851 lr=2.03e-05 tok=23.061B dt=7342 wait=184 +07:35:35 step 175100 loss=2.5913 lr=2.03e-05 tok=23.063B dt=7407 wait=185 +07:36:39 step 175120 loss=2.9116 lr=2.03e-05 tok=23.066B dt=7471 wait=185 +07:37:43 step 175140 loss=2.8429 lr=2.03e-05 tok=23.069B dt=7535 wait=185 +07:38:48 step 175160 loss=2.8522 lr=2.03e-05 tok=23.071B dt=7599 wait=185 +07:39:52 step 175180 loss=2.8644 lr=2.03e-05 tok=23.074B dt=7664 wait=185 +07:40:56 step 175200 loss=2.8325 lr=2.03e-05 tok=23.077B dt=7728 wait=185 +07:41:09 pushed ckpt_w168.pt,log_w168.txt +07:42:25 waited 76s for global r=724 +07:42:30 global r=724 load 4.285 -> 4.283 +07:43:35 step 175220 loss=2.8844 lr=2.03e-05 tok=23.079B dt=7887 wait=185 +07:44:38 step 175240 loss=2.9131 lr=2.02e-05 tok=23.082B dt=7950 wait=186 +07:45:42 step 175260 loss=2.8019 lr=2.02e-05 tok=23.084B dt=8014 wait=186 +07:46:46 step 175280 loss=2.7979 lr=2.02e-05 tok=23.087B dt=8078 wait=186 +07:47:49 step 175300 loss=2.8758 lr=2.02e-05 tok=23.090B dt=8141 wait=186 +07:48:53 step 175320 loss=2.9031 lr=2.02e-05 tok=23.092B dt=8205 wait=186 +07:49:57 step 175340 loss=2.8044 lr=2.02e-05 tok=23.095B dt=8269 wait=186 +07:50:41 pushed ckpt_w168.pt,log_w168.txt +07:51:57 waited 76s for global r=725 +07:52:02 global r=725 load 4.285 -> 4.286 +07:52:35 step 175360 loss=2.9060 lr=2.02e-05 tok=23.098B dt=8427 wait=187 +07:53:38 step 175380 loss=2.8452 lr=2.02e-05 tok=23.100B dt=8490 wait=187 +07:54:42 step 175400 loss=2.8997 lr=2.02e-05 tok=23.103B dt=8554 wait=187 +07:55:47 step 175420 loss=2.7222 lr=2.02e-05 tok=23.105B dt=8618 wait=187 +07:56:51 step 175440 loss=2.8964 lr=2.02e-05 tok=23.108B dt=8683 wait=187 +07:57:54 step 175460 loss=2.9545 lr=2.02e-05 tok=23.111B dt=8746 wait=187 +07:58:59 step 175480 loss=2.8143 lr=2.01e-05 tok=23.113B dt=8810 wait=187 +08:00:03 step 175500 loss=2.9112 lr=2.01e-05 tok=23.116B dt=8874 wait=188 +08:00:15 pushed ckpt_w168.pt,log_w168.txt +08:01:30 waited 76s for global r=726 +08:01:35 global r=726 load 4.287 -> 4.283 +08:02:39 step 175520 loss=2.9408 lr=2.01e-05 tok=23.118B dt=9031 wait=188 +08:03:44 step 175540 loss=2.8936 lr=2.01e-05 tok=23.121B dt=9096 wait=188 +08:04:48 step 175560 loss=2.8488 lr=2.01e-05 tok=23.124B dt=9160 wait=188 +08:05:52 step 175580 loss=2.7861 lr=2.01e-05 tok=23.126B dt=9224 wait=188 +08:06:57 step 175600 loss=2.9126 lr=2.01e-05 tok=23.129B dt=9289 wait=188 +08:08:01 step 175620 loss=2.8670 lr=2.01e-05 tok=23.132B dt=9353 wait=188 +08:09:05 step 175640 loss=2.7529 lr=2.01e-05 tok=23.134B dt=9417 wait=189 +08:09:52 pushed ckpt_w168.pt,log_w168.txt +08:11:23 waited 91s for global r=727 +08:11:27 global r=727 load 4.284 -> 4.284 +08:12:01 step 175660 loss=2.8753 lr=2.01e-05 tok=23.137B dt=9592 wait=189 +08:13:04 step 175680 loss=2.8273 lr=2.01e-05 tok=23.139B dt=9656 wait=189 +08:14:08 step 175700 loss=2.8242 lr=2.01e-05 tok=23.142B dt=9720 wait=189 +08:15:12 step 175720 loss=2.9402 lr=2.00e-05 tok=23.145B dt=9784 wait=189 +08:16:16 step 175740 loss=2.7761 lr=2.00e-05 tok=23.147B dt=9848 wait=189 +08:17:20 step 175760 loss=2.8969 lr=2.00e-05 tok=23.150B dt=9912 wait=189 +08:18:24 step 175780 loss=2.8286 lr=2.00e-05 tok=23.153B dt=9976 wait=190 +08:19:28 step 175800 loss=2.7993 lr=2.00e-05 tok=23.155B dt=10040 wait=190 diff --git a/log_w169.txt b/log_w169.txt new file mode 100644 index 0000000000000000000000000000000000000000..91ab7d954347c6931212cd47cd5ca838c3cf1789 --- /dev/null +++ b/log_w169.txt @@ -0,0 +1,197 @@ +launch +05:28:53 name=w169 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 116/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +05:28:59 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +05:28:59 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +05:29:09 params 1.08B +05:29:11 resumed step=173250 seen=22.82B round=711 +05:29:11 stream salt=173250 +05:30:58 blocks compiled +05:30:58 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:34:11 mix source 6 paused 300s (hf://datasets/OpenCoder-LLM/opc-annealing-corpus@078e4bc1d39725cccc18514d2e3278a) +05:34:45 step 173251 loss=1.9041 lr=2.11e-05 tok=22.821B dt=226 wait=191 +05:35:01 step 173252 loss=2.2822 lr=2.11e-05 tok=22.821B dt=243 wait=191 +05:35:12 dolma file skipped data/ingredient1-cranemath/shard_00000183-withid.jsonl.zst: (Request ID: Root=1-6abc9f90-7521306c56423983474e2bb6;6aaef298-1f21-433f-8bdd-ccb8cd09ba94) + +429 Too Many Requests: you +05:35:13 dolma file skipped data/ingredient1-megamatt/shard_00000880.jsonl.zst: (Request ID: Root=1-6abc9f91-5a618cb97de80f8755dc41ba;517f2d62-4a04-4aed-bc0d-8a5f22368728) + +429 Too Many Requests: you +05:35:13 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-science_tech-2e12/science_tech_p020_shard_00001474.jsonl.zst: (Request ID: Root=1-6abc9f91-0b1c8d016be2664744ad6398;be31c75d-fe76-4491-8c11-39dcd1f5fba4) + +429 Too Many Requests: you +05:35:27 step 173260 loss=3.1413 lr=2.11e-05 tok=22.822B dt=269 wait=191 +05:36:32 step 173280 loss=2.9756 lr=2.11e-05 tok=22.825B dt=333 wait=191 +05:37:37 step 173300 loss=2.9143 lr=2.11e-05 tok=22.827B dt=399 wait=191 +05:38:41 step 173320 loss=3.1289 lr=2.11e-05 tok=22.830B dt=463 wait=192 +05:39:45 step 173340 loss=3.2067 lr=2.11e-05 tok=22.833B dt=527 wait=192 +05:40:50 step 173360 loss=3.2850 lr=2.10e-05 tok=22.835B dt=592 wait=192 +05:41:55 step 173380 loss=3.0534 lr=2.10e-05 tok=22.838B dt=656 wait=192 +05:42:59 step 173400 loss=3.2210 lr=2.10e-05 tok=22.841B dt=721 wait=192 +05:43:18 pushed ckpt_w169.pt,log_w169.txt +05:46:19 waited 181s for global r=712 +05:46:33 global r=712 load 4.273 -> 4.280 +05:47:41 step 173420 loss=3.1068 lr=2.10e-05 tok=22.843B dt=1003 wait=192 +05:48:45 step 173440 loss=3.2424 lr=2.10e-05 tok=22.846B dt=1067 wait=193 +05:49:50 step 173460 loss=3.2521 lr=2.10e-05 tok=22.848B dt=1131 wait=193 +05:50:54 step 173480 loss=3.2010 lr=2.10e-05 tok=22.851B dt=1196 wait=193 +05:51:58 step 173500 loss=3.1904 lr=2.10e-05 tok=22.854B dt=1260 wait=193 +05:53:03 step 173520 loss=3.2750 lr=2.10e-05 tok=22.856B dt=1325 wait=193 +05:54:06 step 173540 loss=3.0406 lr=2.10e-05 tok=22.859B dt=1388 wait=193 +05:54:51 pushed ckpt_w169.pt,log_w169.txt +05:56:06 waited 76s for global r=713 +05:56:15 global r=713 load 4.275 -> 4.280 +05:56:48 step 173560 loss=3.2029 lr=2.10e-05 tok=22.862B dt=1550 wait=194 +05:57:52 step 173580 loss=3.0457 lr=2.09e-05 tok=22.864B dt=1614 wait=194 +05:58:55 step 173600 loss=3.1312 lr=2.09e-05 tok=22.867B dt=1677 wait=194 +05:59:59 step 173620 loss=2.9632 lr=2.09e-05 tok=22.869B dt=1741 wait=194 +06:01:03 step 173640 loss=2.7688 lr=2.09e-05 tok=22.872B dt=1805 wait=194 +06:02:07 step 173660 loss=3.1725 lr=2.09e-05 tok=22.875B dt=1868 wait=194 +06:03:11 step 173680 loss=3.0582 lr=2.09e-05 tok=22.877B dt=1932 wait=194 +06:04:15 step 173700 loss=3.1763 lr=2.09e-05 tok=22.880B dt=1997 wait=195 +06:04:30 pushed ckpt_w169.pt,log_w169.txt +06:06:01 waited 91s for global r=714 +06:06:08 global r=714 load 4.283 -> 4.281 +06:07:13 step 173720 loss=3.1115 lr=2.09e-05 tok=22.883B dt=2174 wait=195 +06:08:17 step 173740 loss=3.1327 lr=2.09e-05 tok=22.885B dt=2239 wait=195 +06:09:21 step 173760 loss=3.2188 lr=2.09e-05 tok=22.888B dt=2303 wait=195 +06:10:25 step 173780 loss=3.1869 lr=2.09e-05 tok=22.890B dt=2367 wait=195 +06:11:30 step 173800 loss=3.2312 lr=2.09e-05 tok=22.893B dt=2432 wait=195 +06:12:34 step 173820 loss=3.2158 lr=2.08e-05 tok=22.896B dt=2496 wait=196 +06:13:39 step 173840 loss=2.8828 lr=2.08e-05 tok=22.898B dt=2561 wait=196 +06:14:23 pushed ckpt_w169.pt,log_w169.txt +06:15:39 waited 76s for global r=715 +06:15:45 global r=715 load 4.283 -> 4.281 +06:16:18 step 173860 loss=3.3021 lr=2.08e-05 tok=22.901B dt=2720 wait=196 +06:17:22 step 173880 loss=3.2743 lr=2.08e-05 tok=22.904B dt=2784 wait=196 +06:18:27 step 173900 loss=3.2033 lr=2.08e-05 tok=22.906B dt=2849 wait=196 +06:19:31 step 173920 loss=3.2862 lr=2.08e-05 tok=22.909B dt=2913 wait=196 +06:20:35 step 173940 loss=3.1088 lr=2.08e-05 tok=22.911B dt=2977 wait=196 +06:21:39 step 173960 loss=3.2677 lr=2.08e-05 tok=22.914B dt=3041 wait=197 +06:22:44 step 173980 loss=3.0990 lr=2.08e-05 tok=22.917B dt=3105 wait=197 +06:23:48 step 174000 loss=3.1421 lr=2.08e-05 tok=22.919B dt=3169 wait=197 +06:23:59 pushed ckpt_w169.pt,log_w169.txt +06:25:15 waited 76s for global r=716 +06:25:21 global r=716 load 4.283 -> 4.282 +06:26:25 step 174020 loss=3.0029 lr=2.08e-05 tok=22.922B dt=3327 wait=197 +06:27:29 step 174040 loss=3.0033 lr=2.07e-05 tok=22.924B dt=3391 wait=197 +06:28:33 step 174060 loss=3.0469 lr=2.07e-05 tok=22.927B dt=3455 wait=197 +06:29:38 step 174080 loss=3.0575 lr=2.07e-05 tok=22.930B dt=3520 wait=198 +06:30:42 step 174100 loss=3.0752 lr=2.07e-05 tok=22.932B dt=3583 wait=198 +06:31:46 step 174120 loss=3.1379 lr=2.07e-05 tok=22.935B dt=3648 wait=198 +06:32:50 step 174140 loss=3.0559 lr=2.07e-05 tok=22.938B dt=3712 wait=198 +06:33:35 pushed ckpt_w169.pt,log_w169.txt +06:34:35 waited 61s for global r=717 +06:34:43 global r=717 load 4.277 -> 4.279 +06:35:19 step 174160 loss=3.1106 lr=2.07e-05 tok=22.940B dt=3861 wait=198 +06:36:23 step 174180 loss=2.9745 lr=2.07e-05 tok=22.943B dt=3925 wait=198 +06:37:27 step 174200 loss=3.0194 lr=2.07e-05 tok=22.945B dt=3989 wait=199 +06:38:32 step 174220 loss=3.1181 lr=2.07e-05 tok=22.948B dt=4054 wait=199 +06:39:36 step 174240 loss=3.0021 lr=2.07e-05 tok=22.951B dt=4118 wait=199 +06:40:40 step 174260 loss=3.1543 lr=2.07e-05 tok=22.953B dt=4182 wait=199 +06:41:44 step 174280 loss=2.9816 lr=2.06e-05 tok=22.956B dt=4246 wait=199 +06:42:49 step 174300 loss=3.0115 lr=2.06e-05 tok=22.959B dt=4311 wait=199 +06:43:03 pushed ckpt_w169.pt,log_w169.txt +06:44:19 waited 76s for global r=718 +06:44:26 global r=718 load 4.283 -> 4.281 +06:45:31 step 174320 loss=3.0222 lr=2.06e-05 tok=22.961B dt=4472 wait=199 +06:46:34 step 174340 loss=3.0812 lr=2.06e-05 tok=22.964B dt=4536 wait=200 +06:47:38 step 174360 loss=3.0405 lr=2.06e-05 tok=22.966B dt=4600 wait=200 +06:48:42 step 174380 loss=2.9752 lr=2.06e-05 tok=22.969B dt=4664 wait=200 +06:49:47 step 174400 loss=3.0513 lr=2.06e-05 tok=22.972B dt=4729 wait=200 +06:50:51 step 174420 loss=3.1867 lr=2.06e-05 tok=22.974B dt=4793 wait=200 +06:51:55 step 174440 loss=3.0360 lr=2.06e-05 tok=22.977B dt=4857 wait=200 +06:52:40 pushed ckpt_w169.pt,log_w169.txt +06:53:56 waited 76s for global r=719 +06:54:03 global r=719 load 4.279 -> 4.278 +06:54:37 step 174460 loss=3.1335 lr=2.06e-05 tok=22.980B dt=5018 wait=200 +06:55:40 step 174480 loss=3.1043 lr=2.06e-05 tok=22.982B dt=5082 wait=201 +06:56:44 step 174500 loss=3.0730 lr=2.06e-05 tok=22.985B dt=5146 wait=201 +06:57:49 step 174520 loss=3.0739 lr=2.05e-05 tok=22.987B dt=5211 wait=201 +06:58:54 step 174540 loss=2.9229 lr=2.05e-05 tok=22.990B dt=5276 wait=201 +06:59:58 step 174560 loss=3.1350 lr=2.05e-05 tok=22.993B dt=5340 wait=201 +07:01:02 step 174580 loss=3.0360 lr=2.05e-05 tok=22.995B dt=5404 wait=201 +07:02:07 step 174600 loss=3.0712 lr=2.05e-05 tok=22.998B dt=5469 wait=201 +07:02:21 pushed ckpt_w169.pt,log_w169.txt +07:03:36 waited 76s for global r=720 +07:03:42 global r=720 load 4.280 -> 4.282 +07:04:47 step 174620 loss=3.0692 lr=2.05e-05 tok=23.001B dt=5629 wait=202 +07:05:52 step 174640 loss=3.0036 lr=2.05e-05 tok=23.003B dt=5693 wait=202 +07:06:57 step 174660 loss=3.0211 lr=2.05e-05 tok=23.006B dt=5758 wait=202 +07:08:01 step 174680 loss=2.9275 lr=2.05e-05 tok=23.008B dt=5823 wait=202 +07:09:06 step 174700 loss=2.9856 lr=2.05e-05 tok=23.011B dt=5887 wait=202 +07:10:10 step 174720 loss=3.0735 lr=2.05e-05 tok=23.014B dt=5952 wait=202 +07:11:14 step 174740 loss=2.9564 lr=2.05e-05 tok=23.016B dt=6016 wait=202 +07:12:01 pushed ckpt_w169.pt,log_w169.txt +07:13:32 waited 91s for global r=721 +07:13:40 global r=721 load 4.283 -> 4.282 +07:14:13 step 174760 loss=3.0692 lr=2.04e-05 tok=23.019B dt=6195 wait=203 +07:15:17 step 174780 loss=2.9679 lr=2.04e-05 tok=23.021B dt=6259 wait=203 +07:16:22 step 174800 loss=2.9751 lr=2.04e-05 tok=23.024B dt=6324 wait=203 +07:17:26 step 174820 loss=2.9627 lr=2.04e-05 tok=23.027B dt=6388 wait=203 +07:18:30 step 174840 loss=2.8792 lr=2.04e-05 tok=23.029B dt=6452 wait=203 +07:19:34 step 174860 loss=3.0246 lr=2.04e-05 tok=23.032B dt=6516 wait=203 +07:20:38 step 174880 loss=2.9414 lr=2.04e-05 tok=23.035B dt=6580 wait=203 +07:21:42 step 174900 loss=3.0023 lr=2.04e-05 tok=23.037B dt=6644 wait=204 +07:21:56 pushed ckpt_w169.pt,log_w169.txt +07:22:56 waited 61s for global r=722 +07:23:03 global r=722 load 4.285 -> 4.284 +07:24:08 step 174920 loss=2.9907 lr=2.04e-05 tok=23.040B dt=6789 wait=204 +07:25:11 step 174940 loss=2.9113 lr=2.04e-05 tok=23.042B dt=6853 wait=204 +07:26:16 step 174960 loss=2.9723 lr=2.04e-05 tok=23.045B dt=6918 wait=204 +07:27:21 step 174980 loss=2.9557 lr=2.04e-05 tok=23.048B dt=6982 wait=204 +07:28:25 step 175000 loss=2.9592 lr=2.03e-05 tok=23.050B dt=7047 wait=204 +07:29:29 step 175020 loss=3.0420 lr=2.03e-05 tok=23.053B dt=7111 wait=204 +07:30:33 step 175040 loss=2.9147 lr=2.03e-05 tok=23.056B dt=7175 wait=205 +07:31:18 pushed ckpt_w169.pt,log_w169.txt +07:32:34 waited 76s for global r=723 +07:32:43 global r=723 load 4.280 -> 4.282 +07:33:16 step 175060 loss=2.9687 lr=2.03e-05 tok=23.058B dt=7338 wait=205 +07:34:20 step 175080 loss=2.9503 lr=2.03e-05 tok=23.061B dt=7402 wait=205 +07:35:24 step 175100 loss=3.0325 lr=2.03e-05 tok=23.063B dt=7466 wait=205 +07:36:28 step 175120 loss=3.1200 lr=2.03e-05 tok=23.066B dt=7530 wait=205 +07:37:31 step 175140 loss=3.0072 lr=2.03e-05 tok=23.069B dt=7593 wait=205 +07:38:36 step 175160 loss=2.7470 lr=2.03e-05 tok=23.071B dt=7658 wait=205 +07:39:40 step 175180 loss=2.8751 lr=2.03e-05 tok=23.074B dt=7722 wait=206 +07:40:43 step 175200 loss=2.8909 lr=2.03e-05 tok=23.077B dt=7785 wait=206 +07:40:55 pushed ckpt_w169.pt,log_w169.txt +07:42:26 waited 91s for global r=724 +07:42:32 global r=724 load 4.283 -> 4.283 +07:43:36 step 175220 loss=2.9046 lr=2.03e-05 tok=23.079B dt=7958 wait=206 +07:44:40 step 175240 loss=2.9440 lr=2.02e-05 tok=23.082B dt=8022 wait=206 +07:45:44 step 175260 loss=2.9708 lr=2.02e-05 tok=23.084B dt=8086 wait=206 +07:46:48 step 175280 loss=2.8447 lr=2.02e-05 tok=23.087B dt=8150 wait=206 +07:47:51 step 175300 loss=2.9665 lr=2.02e-05 tok=23.090B dt=8213 wait=206 +07:48:56 step 175320 loss=2.9302 lr=2.02e-05 tok=23.092B dt=8277 wait=207 +07:50:00 step 175340 loss=2.9287 lr=2.02e-05 tok=23.095B dt=8341 wait=207 +07:50:44 pushed ckpt_w169.pt,log_w169.txt +07:52:00 waited 76s for global r=725 +07:52:05 global r=725 load 4.291 -> 4.286 +07:52:39 step 175360 loss=3.0079 lr=2.02e-05 tok=23.098B dt=8501 wait=207 +07:53:43 step 175380 loss=2.9856 lr=2.02e-05 tok=23.100B dt=8564 wait=207 +07:54:47 step 175400 loss=3.0260 lr=2.02e-05 tok=23.103B dt=8629 wait=207 +07:55:52 step 175420 loss=3.0557 lr=2.02e-05 tok=23.105B dt=8693 wait=207 +07:56:57 step 175440 loss=2.6394 lr=2.02e-05 tok=23.108B dt=8759 wait=207 +07:58:01 step 175460 loss=3.0710 lr=2.02e-05 tok=23.111B dt=8823 wait=208 +07:59:05 step 175480 loss=2.9456 lr=2.01e-05 tok=23.113B dt=8887 wait=208 +08:00:10 step 175500 loss=3.0724 lr=2.01e-05 tok=23.116B dt=8952 wait=208 +08:00:24 pushed ckpt_w169.pt,log_w169.txt +08:01:39 waited 76s for global r=726 +08:01:44 global r=726 load 4.290 -> 4.283 +08:02:49 step 175520 loss=3.0146 lr=2.01e-05 tok=23.118B dt=9111 wait=208 +08:03:53 step 175540 loss=2.9144 lr=2.01e-05 tok=23.121B dt=9175 wait=208 +08:04:57 step 175560 loss=2.8789 lr=2.01e-05 tok=23.124B dt=9239 wait=208 +08:06:01 step 175580 loss=2.8270 lr=2.01e-05 tok=23.126B dt=9303 wait=208 +08:07:05 step 175600 loss=2.8843 lr=2.01e-05 tok=23.129B dt=9367 wait=209 +08:08:11 step 175620 loss=2.5119 lr=2.01e-05 tok=23.132B dt=9433 wait=209 +08:09:15 step 175640 loss=2.7920 lr=2.01e-05 tok=23.134B dt=9497 wait=209 +08:10:01 pushed ckpt_w169.pt,log_w169.txt +08:11:17 waited 76s for global r=727 +08:11:25 global r=727 load 4.294 -> 4.284 +08:11:58 step 175660 loss=3.0385 lr=2.01e-05 tok=23.137B dt=9660 wait=209 +08:13:01 step 175680 loss=2.9486 lr=2.01e-05 tok=23.139B dt=9723 wait=209 +08:14:06 step 175700 loss=2.8178 lr=2.01e-05 tok=23.142B dt=9788 wait=209 +08:15:10 step 175720 loss=2.9663 lr=2.00e-05 tok=23.145B dt=9852 wait=209 +08:16:15 step 175740 loss=2.9582 lr=2.00e-05 tok=23.147B dt=9917 wait=210 +08:17:20 step 175760 loss=2.9917 lr=2.00e-05 tok=23.150B dt=9982 wait=210 +08:18:25 step 175780 loss=2.9820 lr=2.00e-05 tok=23.153B dt=10047 wait=210 +08:19:29 step 175800 loss=2.9762 lr=2.00e-05 tok=23.155B dt=10111 wait=210 diff --git a/log_w170.txt b/log_w170.txt new file mode 100644 index 0000000000000000000000000000000000000000..73966f49247abdd562c2a23fc28059e5fa75fb54 --- /dev/null +++ b/log_w170.txt @@ -0,0 +1,192 @@ +launch +05:28:58 name=w170 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 117/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +05:29:05 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +05:29:05 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +05:29:14 params 1.08B +05:29:16 resumed step=173250 seen=22.82B round=711 +05:29:16 stream salt=173250 +05:31:03 blocks compiled +05:31:03 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:34:11 mix source 6 paused 173s ((Request ID: Root=1-6abc9f53-6c3dd2762f5286cb2fcdb20d;04506311-be46-4792-b9a5-a2) +05:34:11 mix source 5 paused 377s ((Request ID: Root=1-6abc9f53-466616d9060fae0c4569d3e0;fd72851c-5fbc-45f7-8f0d-6f) +05:34:12 mix source 7 paused 210s ((Request ID: Root=1-6abc9f54-723ef24776248f9333f970ef;d0c98d1b-af5e-4ea4-9cb4-1f) +05:34:37 step 173251 loss=3.1780 lr=2.11e-05 tok=22.821B dt=214 wait=188 +05:34:53 step 173252 loss=3.2457 lr=2.11e-05 tok=22.821B dt=230 wait=188 +05:35:14 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-science_tech-2e13/science_tech_p050_shard_00000291.jsonl.zst: (Request ID: Root=1-6abc9f92-57ea54c72d5763b424bc4367;77127777-6fbd-4fbd-bb0f-929a682488e3) + +429 Too Many Requests: you +05:35:18 step 173260 loss=3.4378 lr=2.11e-05 tok=22.822B dt=255 wait=188 +05:36:24 step 173280 loss=3.2151 lr=2.11e-05 tok=22.825B dt=321 wait=188 +05:37:29 step 173300 loss=3.2938 lr=2.11e-05 tok=22.827B dt=386 wait=189 +05:38:34 step 173320 loss=3.3715 lr=2.11e-05 tok=22.830B dt=451 wait=189 +05:39:39 step 173340 loss=3.3761 lr=2.11e-05 tok=22.833B dt=516 wait=189 +05:40:43 step 173360 loss=3.4047 lr=2.10e-05 tok=22.835B dt=580 wait=189 +05:41:48 step 173380 loss=3.2010 lr=2.10e-05 tok=22.838B dt=645 wait=189 +05:42:52 step 173400 loss=3.3518 lr=2.10e-05 tok=22.841B dt=709 wait=189 +05:43:06 pushed ckpt_w170.pt,log_w170.txt +05:46:59 global r=712 load 4.281 -> 4.280 +05:48:03 step 173420 loss=3.0853 lr=2.10e-05 tok=22.845B dt=1020 wait=190 +05:49:07 step 173440 loss=3.1694 lr=2.10e-05 tok=22.847B dt=1084 wait=190 +05:50:11 step 173460 loss=3.0391 lr=2.10e-05 tok=22.850B dt=1148 wait=190 +05:51:15 step 173480 loss=3.1353 lr=2.10e-05 tok=22.852B dt=1212 wait=190 +05:52:19 step 173500 loss=3.0924 lr=2.10e-05 tok=22.855B dt=1276 wait=190 +05:53:24 step 173520 loss=3.0267 lr=2.10e-05 tok=22.858B dt=1341 wait=190 +05:54:27 step 173540 loss=2.9125 lr=2.10e-05 tok=22.860B dt=1404 wait=190 +05:55:14 pushed ckpt_w170.pt,log_w170.txt +05:56:14 waited 61s for global r=713 +05:56:21 global r=713 load 4.282 -> 4.280 +05:56:55 step 173560 loss=3.0619 lr=2.10e-05 tok=22.863B dt=1552 wait=191 +05:57:58 step 173580 loss=2.9720 lr=2.09e-05 tok=22.866B dt=1615 wait=191 +05:59:02 step 173600 loss=2.9768 lr=2.09e-05 tok=22.868B dt=1679 wait=191 +06:00:05 step 173620 loss=3.1156 lr=2.09e-05 tok=22.871B dt=1742 wait=191 +06:01:08 step 173640 loss=2.9291 lr=2.09e-05 tok=22.873B dt=1805 wait=191 +06:02:42 step 173660 loss=3.0065 lr=2.09e-05 tok=22.876B dt=1899 wait=222 +06:03:47 step 173680 loss=3.0483 lr=2.09e-05 tok=22.879B dt=1964 wait=224 +06:04:50 step 173700 loss=3.1409 lr=2.09e-05 tok=22.881B dt=2027 wait=224 +06:05:06 pushed ckpt_w170.pt,log_w170.txt +06:05:51 waited 45s for global r=714 +06:05:59 global r=714 load 4.287 -> 4.281 +06:07:03 step 173720 loss=3.0471 lr=2.09e-05 tok=22.884B dt=2160 wait=224 +06:08:08 step 173740 loss=3.0459 lr=2.09e-05 tok=22.886B dt=2225 wait=224 +06:09:11 step 173760 loss=3.1188 lr=2.09e-05 tok=22.889B dt=2288 wait=224 +06:10:16 step 173780 loss=3.1002 lr=2.09e-05 tok=22.892B dt=2353 wait=224 +06:11:19 step 173800 loss=3.0347 lr=2.09e-05 tok=22.894B dt=2417 wait=225 +06:12:24 step 173820 loss=2.8288 lr=2.08e-05 tok=22.897B dt=2481 wait=225 +06:13:27 step 173840 loss=2.9874 lr=2.08e-05 tok=22.900B dt=2544 wait=225 +06:14:11 pushed ckpt_w170.pt,log_w170.txt +06:15:27 waited 76s for global r=715 +06:15:33 global r=715 load 4.278 -> 4.281 +06:16:06 step 173860 loss=3.1310 lr=2.08e-05 tok=22.902B dt=2703 wait=225 +06:17:11 step 173880 loss=3.0469 lr=2.08e-05 tok=22.905B dt=2768 wait=225 +06:18:15 step 173900 loss=3.0754 lr=2.08e-05 tok=22.907B dt=2832 wait=225 +06:19:20 step 173920 loss=3.2542 lr=2.08e-05 tok=22.910B dt=2897 wait=225 +06:20:23 step 173940 loss=2.8986 lr=2.08e-05 tok=22.913B dt=2960 wait=226 +06:21:28 step 173960 loss=3.1445 lr=2.08e-05 tok=22.915B dt=3025 wait=226 +06:22:33 step 173980 loss=3.0189 lr=2.08e-05 tok=22.918B dt=3090 wait=226 +06:23:37 step 174000 loss=2.9946 lr=2.08e-05 tok=22.921B dt=3154 wait=226 +06:23:51 pushed ckpt_w170.pt,log_w170.txt +06:25:07 waited 76s for global r=716 +06:25:12 global r=716 load 4.281 -> 4.282 +06:26:17 step 174020 loss=2.9036 lr=2.08e-05 tok=22.923B dt=3314 wait=226 +06:27:21 step 174040 loss=3.0034 lr=2.07e-05 tok=22.926B dt=3378 wait=226 +06:28:25 step 174060 loss=3.0026 lr=2.07e-05 tok=22.928B dt=3442 wait=226 +06:29:29 step 174080 loss=3.0422 lr=2.07e-05 tok=22.931B dt=3506 wait=226 +06:30:33 step 174100 loss=3.1066 lr=2.07e-05 tok=22.934B dt=3570 wait=227 +06:31:36 step 174120 loss=3.1034 lr=2.07e-05 tok=22.936B dt=3634 wait=227 +06:32:40 step 174140 loss=3.1376 lr=2.07e-05 tok=22.939B dt=3697 wait=227 +06:33:24 pushed ckpt_w170.pt,log_w170.txt +06:34:39 waited 76s for global r=717 +06:34:45 global r=717 load 4.282 -> 4.279 +06:35:18 step 174160 loss=3.0368 lr=2.07e-05 tok=22.942B dt=3855 wait=227 +06:36:21 step 174180 loss=2.9953 lr=2.07e-05 tok=22.944B dt=3918 wait=227 +06:37:25 step 174200 loss=3.0390 lr=2.07e-05 tok=22.947B dt=3982 wait=227 +06:38:29 step 174220 loss=3.1064 lr=2.07e-05 tok=22.949B dt=4046 wait=227 +06:39:32 step 174240 loss=2.9071 lr=2.07e-05 tok=22.952B dt=4109 wait=228 +06:40:36 step 174260 loss=3.1081 lr=2.07e-05 tok=22.955B dt=4173 wait=228 +06:41:39 step 174280 loss=2.9489 lr=2.06e-05 tok=22.957B dt=4236 wait=228 +06:42:43 step 174300 loss=3.0369 lr=2.06e-05 tok=22.960B dt=4300 wait=228 +06:42:55 pushed ckpt_w170.pt,log_w170.txt +06:44:11 waited 76s for global r=718 +06:44:18 global r=718 load 4.287 -> 4.281 +06:45:23 step 174320 loss=2.9915 lr=2.06e-05 tok=22.963B dt=4460 wait=228 +06:46:26 step 174340 loss=3.1138 lr=2.06e-05 tok=22.965B dt=4523 wait=228 +06:47:30 step 174360 loss=3.0320 lr=2.06e-05 tok=22.968B dt=4587 wait=228 +06:48:34 step 174380 loss=3.0743 lr=2.06e-05 tok=22.970B dt=4651 wait=229 +06:49:37 step 174400 loss=2.9865 lr=2.06e-05 tok=22.973B dt=4715 wait=229 +06:50:41 step 174420 loss=2.9173 lr=2.06e-05 tok=22.976B dt=4778 wait=229 +06:51:44 step 174440 loss=2.9729 lr=2.06e-05 tok=22.978B dt=4842 wait=229 +06:52:28 pushed ckpt_w170.pt,log_w170.txt +06:53:59 waited 91s for global r=719 +06:54:04 global r=719 load 4.277 -> 4.278 +06:54:37 step 174460 loss=3.0261 lr=2.06e-05 tok=22.981B dt=5014 wait=229 +06:55:40 step 174480 loss=2.9540 lr=2.06e-05 tok=22.983B dt=5077 wait=229 +06:56:43 step 174500 loss=2.8674 lr=2.06e-05 tok=22.986B dt=5140 wait=229 +06:57:47 step 174520 loss=3.0146 lr=2.05e-05 tok=22.989B dt=5204 wait=229 +06:58:50 step 174540 loss=2.9290 lr=2.05e-05 tok=22.991B dt=5267 wait=230 +06:59:54 step 174560 loss=3.0247 lr=2.05e-05 tok=22.994B dt=5331 wait=230 +07:00:57 step 174580 loss=2.8953 lr=2.05e-05 tok=22.997B dt=5394 wait=230 +07:02:01 step 174600 loss=2.9428 lr=2.05e-05 tok=22.999B dt=5458 wait=230 +07:02:16 pushed ckpt_w170.pt,log_w170.txt +07:03:47 waited 91s for global r=720 +07:03:53 global r=720 load 4.285 -> 4.282 +07:04:58 step 174620 loss=2.9702 lr=2.05e-05 tok=23.002B dt=5635 wait=230 +07:06:02 step 174640 loss=2.9822 lr=2.05e-05 tok=23.004B dt=5699 wait=230 +07:07:07 step 174660 loss=3.0271 lr=2.05e-05 tok=23.007B dt=5764 wait=230 +07:08:11 step 174680 loss=3.0243 lr=2.05e-05 tok=23.010B dt=5828 wait=230 +07:09:15 step 174700 loss=3.0783 lr=2.05e-05 tok=23.012B dt=5893 wait=231 +07:10:20 step 174720 loss=3.0828 lr=2.05e-05 tok=23.015B dt=5957 wait=231 +07:11:24 step 174740 loss=2.9295 lr=2.05e-05 tok=23.018B dt=6021 wait=231 +07:12:08 pushed ckpt_w170.pt,log_w170.txt +07:13:24 waited 76s for global r=721 +07:13:30 global r=721 load 4.281 -> 4.282 +07:14:04 step 174760 loss=2.7878 lr=2.04e-05 tok=23.020B dt=6181 wait=231 +07:15:08 step 174780 loss=2.8841 lr=2.04e-05 tok=23.023B dt=6245 wait=231 +07:16:11 step 174800 loss=2.9691 lr=2.04e-05 tok=23.025B dt=6308 wait=231 +07:17:14 step 174820 loss=3.0001 lr=2.04e-05 tok=23.028B dt=6371 wait=231 +07:18:17 step 174840 loss=2.8462 lr=2.04e-05 tok=23.031B dt=6434 wait=232 +07:19:20 step 174860 loss=2.9578 lr=2.04e-05 tok=23.033B dt=6497 wait=232 +07:20:24 step 174880 loss=2.8669 lr=2.04e-05 tok=23.036B dt=6561 wait=232 +07:21:27 step 174900 loss=2.9028 lr=2.04e-05 tok=23.039B dt=6624 wait=232 +07:21:41 pushed ckpt_w170.pt,log_w170.txt +07:22:56 waited 76s for global r=722 +07:23:03 global r=722 load 4.280 -> 4.284 +07:24:07 step 174920 loss=2.9140 lr=2.04e-05 tok=23.041B dt=6784 wait=232 +07:25:10 step 174940 loss=2.9241 lr=2.04e-05 tok=23.044B dt=6847 wait=232 +07:26:14 step 174960 loss=2.9224 lr=2.04e-05 tok=23.046B dt=6911 wait=232 +07:27:18 step 174980 loss=2.8246 lr=2.04e-05 tok=23.049B dt=6975 wait=233 +07:28:21 step 175000 loss=2.9409 lr=2.03e-05 tok=23.052B dt=7038 wait=233 +07:29:24 step 175020 loss=2.9799 lr=2.03e-05 tok=23.054B dt=7101 wait=233 +07:30:28 step 175040 loss=2.8929 lr=2.03e-05 tok=23.057B dt=7165 wait=233 +07:31:11 pushed ckpt_w170.pt,log_w170.txt +07:32:42 waited 91s for global r=723 +07:32:47 global r=723 load 4.277 -> 4.282 +07:33:20 step 175060 loss=2.9535 lr=2.03e-05 tok=23.059B dt=7337 wait=233 +07:34:25 step 175080 loss=2.5498 lr=2.03e-05 tok=23.062B dt=7402 wait=233 +07:35:28 step 175100 loss=2.9812 lr=2.03e-05 tok=23.065B dt=7465 wait=233 +07:36:32 step 175120 loss=3.0460 lr=2.03e-05 tok=23.067B dt=7529 wait=234 +07:37:35 step 175140 loss=2.8382 lr=2.03e-05 tok=23.070B dt=7592 wait=234 +07:38:38 step 175160 loss=2.9830 lr=2.03e-05 tok=23.073B dt=7655 wait=234 +07:39:41 step 175180 loss=2.9391 lr=2.03e-05 tok=23.075B dt=7718 wait=234 +07:40:45 step 175200 loss=2.7984 lr=2.03e-05 tok=23.078B dt=7782 wait=234 +07:40:58 pushed ckpt_w170.pt,log_w170.txt +07:42:28 waited 91s for global r=724 +07:42:36 global r=724 load 4.289 -> 4.283 +07:43:40 step 175220 loss=2.8542 lr=2.03e-05 tok=23.080B dt=7957 wait=234 +07:44:43 step 175240 loss=2.9039 lr=2.02e-05 tok=23.083B dt=8020 wait=234 +07:45:47 step 175260 loss=2.9442 lr=2.02e-05 tok=23.086B dt=8084 wait=235 +07:46:50 step 175280 loss=2.9326 lr=2.02e-05 tok=23.088B dt=8148 wait=235 +07:47:54 step 175300 loss=2.9327 lr=2.02e-05 tok=23.091B dt=8211 wait=235 +07:48:58 step 175320 loss=2.9322 lr=2.02e-05 tok=23.094B dt=8275 wait=235 +07:50:01 step 175340 loss=2.8304 lr=2.02e-05 tok=23.096B dt=8338 wait=235 +07:50:47 pushed ckpt_w170.pt,log_w170.txt +07:52:03 waited 76s for global r=725 +07:52:09 global r=725 load 4.284 -> 4.286 +07:52:42 step 175360 loss=3.0123 lr=2.02e-05 tok=23.099B dt=8499 wait=235 +07:53:46 step 175380 loss=2.9057 lr=2.02e-05 tok=23.101B dt=8563 wait=235 +07:54:50 step 175400 loss=2.9016 lr=2.02e-05 tok=23.104B dt=8627 wait=236 +07:55:54 step 175420 loss=3.0337 lr=2.02e-05 tok=23.107B dt=8691 wait=236 +07:56:59 step 175440 loss=2.6683 lr=2.02e-05 tok=23.109B dt=8756 wait=236 +07:58:02 step 175460 loss=3.0590 lr=2.02e-05 tok=23.112B dt=8819 wait=236 +07:59:06 step 175480 loss=2.9261 lr=2.01e-05 tok=23.115B dt=8884 wait=236 +08:00:10 step 175500 loss=2.9360 lr=2.01e-05 tok=23.117B dt=8947 wait=236 +08:00:22 pushed ckpt_w170.pt,log_w170.txt +08:01:37 waited 76s for global r=726 +08:01:44 global r=726 load 4.273 -> 4.283 +08:02:48 step 175520 loss=2.8719 lr=2.01e-05 tok=23.120B dt=9105 wait=237 +08:03:52 step 175540 loss=2.9307 lr=2.01e-05 tok=23.122B dt=9169 wait=237 +08:04:56 step 175560 loss=2.9079 lr=2.01e-05 tok=23.125B dt=9233 wait=237 +08:05:59 step 175580 loss=2.8768 lr=2.01e-05 tok=23.128B dt=9296 wait=237 +08:07:02 step 175600 loss=3.0199 lr=2.01e-05 tok=23.130B dt=9359 wait=237 +08:08:06 step 175620 loss=2.9541 lr=2.01e-05 tok=23.133B dt=9423 wait=237 +08:09:11 step 175640 loss=2.8393 lr=2.01e-05 tok=23.136B dt=9488 wait=237 +08:09:54 pushed ckpt_w170.pt,log_w170.txt +08:11:25 waited 91s for global r=727 +08:11:31 global r=727 load 4.282 -> 4.284 +08:12:04 step 175660 loss=3.0236 lr=2.01e-05 tok=23.138B dt=9661 wait=237 +08:13:07 step 175680 loss=2.9837 lr=2.01e-05 tok=23.141B dt=9724 wait=238 +08:14:11 step 175700 loss=2.9240 lr=2.01e-05 tok=23.143B dt=9788 wait=238 +08:15:16 step 175720 loss=3.0484 lr=2.00e-05 tok=23.146B dt=9853 wait=238 +08:16:20 step 175740 loss=2.8136 lr=2.00e-05 tok=23.149B dt=9917 wait=238 +08:17:25 step 175760 loss=2.9918 lr=2.00e-05 tok=23.151B dt=9982 wait=238 +08:18:29 step 175780 loss=3.0174 lr=2.00e-05 tok=23.154B dt=10046 wait=238 +08:19:33 step 175800 loss=2.9498 lr=2.00e-05 tok=23.156B dt=10110 wait=238 diff --git a/log_w171.txt b/log_w171.txt new file mode 100644 index 0000000000000000000000000000000000000000..8003b816b7eb2087f826b7d1d82eb923614cd5ed --- /dev/null +++ b/log_w171.txt @@ -0,0 +1,231 @@ +launch +05:29:23 name=w171 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 118/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +05:29:30 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +05:29:30 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +05:29:39 params 1.08B +05:29:41 resumed step=173250 seen=22.82B round=711 +05:29:41 stream salt=173250 +05:31:26 blocks compiled +05:31:26 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:37:36 step 173251 loss=3.0212 lr=2.11e-05 tok=22.821B dt=370 wait=343 +05:37:51 step 173252 loss=3.1200 lr=2.11e-05 tok=22.821B dt=385 wait=343 +05:38:17 step 173260 loss=3.3151 lr=2.11e-05 tok=22.822B dt=411 wait=343 +05:39:21 step 173280 loss=3.0693 lr=2.11e-05 tok=22.825B dt=475 wait=344 +05:40:26 step 173300 loss=3.1025 lr=2.11e-05 tok=22.827B dt=540 wait=344 +05:41:30 step 173320 loss=3.1466 lr=2.11e-05 tok=22.830B dt=604 wait=344 +05:42:34 step 173340 loss=3.0402 lr=2.11e-05 tok=22.833B dt=668 wait=344 +05:43:39 step 173360 loss=3.4228 lr=2.10e-05 tok=22.835B dt=733 wait=344 +05:44:44 step 173380 loss=3.2746 lr=2.10e-05 tok=22.838B dt=798 wait=344 +05:45:48 step 173400 loss=3.3477 lr=2.10e-05 tok=22.841B dt=863 wait=344 +05:46:03 pushed ckpt_w171.pt,log_w171.txt +05:46:18 waited 15s for global r=712 +05:46:33 global r=712 load 4.282 -> 4.280 +05:47:40 step 173420 loss=3.2241 lr=2.10e-05 tok=22.843B dt=974 wait=345 +05:48:45 step 173440 loss=3.3383 lr=2.10e-05 tok=22.846B dt=1039 wait=345 +05:49:50 step 173460 loss=3.2942 lr=2.10e-05 tok=22.848B dt=1104 wait=345 +05:50:55 step 173480 loss=3.1559 lr=2.10e-05 tok=22.851B dt=1169 wait=345 +05:51:59 step 173500 loss=3.1799 lr=2.10e-05 tok=22.854B dt=1233 wait=345 +05:53:04 step 173520 loss=3.2947 lr=2.10e-05 tok=22.856B dt=1298 wait=345 +05:54:08 step 173540 loss=3.1674 lr=2.10e-05 tok=22.859B dt=1362 wait=345 +05:54:52 pushed ckpt_w171.pt,log_w171.txt +05:56:08 waited 76s for global r=713 +05:56:18 global r=713 load 4.279 -> 4.280 +05:56:52 step 173560 loss=3.2966 lr=2.10e-05 tok=22.862B dt=1526 wait=346 +05:57:56 step 173580 loss=3.0775 lr=2.09e-05 tok=22.864B dt=1591 wait=346 +05:59:00 step 173600 loss=3.2623 lr=2.09e-05 tok=22.867B dt=1654 wait=346 +06:00:05 step 173620 loss=3.2425 lr=2.09e-05 tok=22.869B dt=1719 wait=346 +06:01:09 step 173640 loss=3.1185 lr=2.09e-05 tok=22.872B dt=1783 wait=346 +06:02:14 step 173660 loss=2.8617 lr=2.09e-05 tok=22.875B dt=1848 wait=346 +06:03:26 step 173680 loss=3.0713 lr=2.09e-05 tok=22.877B dt=1921 wait=354 +06:04:31 step 173700 loss=3.1896 lr=2.09e-05 tok=22.880B dt=1985 wait=354 +06:04:43 pushed ckpt_w171.pt,log_w171.txt +06:05:59 waited 76s for global r=714 +06:06:05 global r=714 load 4.278 -> 4.281 +06:07:10 step 173720 loss=3.1495 lr=2.09e-05 tok=22.883B dt=2144 wait=355 +06:08:15 step 173740 loss=3.1784 lr=2.09e-05 tok=22.885B dt=2209 wait=355 +06:09:19 step 173760 loss=3.1296 lr=2.09e-05 tok=22.888B dt=2273 wait=355 +06:10:24 step 173780 loss=3.0838 lr=2.09e-05 tok=22.890B dt=2338 wait=355 +06:11:29 step 173800 loss=2.9098 lr=2.09e-05 tok=22.893B dt=2403 wait=355 +06:12:33 step 173820 loss=3.1132 lr=2.08e-05 tok=22.896B dt=2467 wait=355 +06:13:37 step 173840 loss=3.0141 lr=2.08e-05 tok=22.898B dt=2531 wait=355 +06:14:22 pushed ckpt_w171.pt,log_w171.txt +06:15:38 waited 76s for global r=715 +06:15:47 global r=715 load 4.284 -> 4.281 +06:16:20 step 173860 loss=3.1070 lr=2.08e-05 tok=22.901B dt=2694 wait=356 +06:17:24 step 173880 loss=3.0092 lr=2.08e-05 tok=22.904B dt=2758 wait=356 +06:18:28 step 173900 loss=3.0067 lr=2.08e-05 tok=22.906B dt=2822 wait=356 +06:19:31 step 173920 loss=3.0075 lr=2.08e-05 tok=22.909B dt=2885 wait=356 +06:20:34 step 173940 loss=2.8781 lr=2.08e-05 tok=22.911B dt=2948 wait=356 +06:21:37 step 173960 loss=2.9964 lr=2.08e-05 tok=22.914B dt=3012 wait=356 +06:22:41 step 173980 loss=2.8869 lr=2.08e-05 tok=22.917B dt=3075 wait=356 +06:23:44 step 174000 loss=2.9358 lr=2.08e-05 tok=22.919B dt=3138 wait=356 +06:23:57 pushed ckpt_w171.pt,log_w171.txt +06:25:12 waited 76s for global r=716 +06:25:19 global r=716 load 4.287 -> 4.282 +06:26:23 step 174020 loss=2.9377 lr=2.08e-05 tok=22.922B dt=3297 wait=357 +06:27:26 step 174040 loss=2.8649 lr=2.07e-05 tok=22.924B dt=3360 wait=357 +06:28:30 step 174060 loss=2.8471 lr=2.07e-05 tok=22.927B dt=3424 wait=357 +06:29:34 step 174080 loss=2.7659 lr=2.07e-05 tok=22.930B dt=3488 wait=357 +06:30:38 step 174100 loss=2.8814 lr=2.07e-05 tok=22.932B dt=3552 wait=357 +06:31:41 step 174120 loss=3.0378 lr=2.07e-05 tok=22.935B dt=3615 wait=357 +06:32:45 step 174140 loss=2.9666 lr=2.07e-05 tok=22.938B dt=3679 wait=358 +06:33:29 pushed ckpt_w171.pt,log_w171.txt +06:34:44 waited 76s for global r=717 +06:34:52 global r=717 load 4.278 -> 4.279 +06:35:25 step 174160 loss=3.1491 lr=2.07e-05 tok=22.940B dt=3839 wait=358 +06:36:29 step 174180 loss=3.0153 lr=2.07e-05 tok=22.943B dt=3903 wait=358 +06:37:33 step 174200 loss=3.1103 lr=2.07e-05 tok=22.945B dt=3967 wait=358 +06:38:37 step 174220 loss=3.2020 lr=2.07e-05 tok=22.948B dt=4031 wait=358 +06:39:41 step 174240 loss=3.1514 lr=2.07e-05 tok=22.951B dt=4096 wait=358 +06:40:45 step 174260 loss=3.1833 lr=2.07e-05 tok=22.953B dt=4160 wait=358 +06:41:50 step 174280 loss=3.1122 lr=2.06e-05 tok=22.956B dt=4224 wait=359 +06:42:54 step 174300 loss=3.1330 lr=2.06e-05 tok=22.959B dt=4288 wait=359 +06:43:08 pushed ckpt_w171.pt,log_w171.txt +06:44:24 waited 76s for global r=718 +06:44:31 global r=718 load 4.282 -> 4.281 +06:45:35 step 174320 loss=2.9857 lr=2.06e-05 tok=22.961B dt=4449 wait=359 +06:46:39 step 174340 loss=3.0300 lr=2.06e-05 tok=22.964B dt=4513 wait=359 +06:47:43 step 174360 loss=3.0647 lr=2.06e-05 tok=22.966B dt=4577 wait=359 +06:48:47 step 174380 loss=3.0592 lr=2.06e-05 tok=22.969B dt=4641 wait=359 +06:49:51 step 174400 loss=3.0661 lr=2.06e-05 tok=22.972B dt=4705 wait=359 +06:50:55 step 174420 loss=3.1368 lr=2.06e-05 tok=22.974B dt=4769 wait=360 +06:51:58 step 174440 loss=2.9677 lr=2.06e-05 tok=22.977B dt=4832 wait=360 +06:52:48 pushed ckpt_w171.pt,log_w171.txt +06:53:49 waited 61s for global r=719 +06:53:59 global r=719 load 4.277 -> 4.278 +06:54:33 step 174460 loss=3.0570 lr=2.06e-05 tok=22.980B dt=4987 wait=360 +06:55:36 step 174480 loss=3.0840 lr=2.06e-05 tok=22.982B dt=5050 wait=360 +06:56:40 step 174500 loss=2.9685 lr=2.06e-05 tok=22.985B dt=5114 wait=360 +06:57:44 step 174520 loss=3.0396 lr=2.05e-05 tok=22.987B dt=5178 wait=360 +06:58:48 step 174540 loss=2.9346 lr=2.05e-05 tok=22.990B dt=5242 wait=361 +06:59:52 step 174560 loss=3.0664 lr=2.05e-05 tok=22.993B dt=5306 wait=361 +07:00:55 step 174580 loss=2.9627 lr=2.05e-05 tok=22.995B dt=5370 wait=361 +07:01:59 step 174600 loss=2.9953 lr=2.05e-05 tok=22.998B dt=5433 wait=361 +07:02:14 pushed ckpt_w171.pt,log_w171.txt +07:03:45 waited 91s for global r=720 +07:03:52 global r=720 load 4.280 -> 4.282 +07:04:57 step 174620 loss=2.9355 lr=2.05e-05 tok=23.001B dt=5611 wait=361 +07:06:01 step 174640 loss=2.9873 lr=2.05e-05 tok=23.003B dt=5675 wait=361 +07:07:04 step 174660 loss=3.0562 lr=2.05e-05 tok=23.006B dt=5738 wait=361 +07:08:08 step 174680 loss=2.9724 lr=2.05e-05 tok=23.008B dt=5802 wait=362 +07:09:12 step 174700 loss=3.0096 lr=2.05e-05 tok=23.011B dt=5866 wait=362 +07:10:16 step 174720 loss=3.0959 lr=2.05e-05 tok=23.014B dt=5930 wait=362 +07:11:21 step 174740 loss=2.7078 lr=2.05e-05 tok=23.016B dt=5995 wait=362 +07:12:08 pushed ckpt_w171.pt,log_w171.txt +07:13:24 waited 76s for global r=721 +07:13:32 global r=721 load 4.282 -> 4.282 +07:14:05 step 174760 loss=3.0855 lr=2.04e-05 tok=23.019B dt=6160 wait=362 +07:15:10 step 174780 loss=3.0100 lr=2.04e-05 tok=23.021B dt=6224 wait=362 +07:16:14 step 174800 loss=3.0298 lr=2.04e-05 tok=23.024B dt=6288 wait=362 +07:17:18 step 174820 loss=2.8882 lr=2.04e-05 tok=23.027B dt=6352 wait=362 +07:18:23 step 174840 loss=2.9186 lr=2.04e-05 tok=23.029B dt=6417 wait=363 +07:19:27 step 174860 loss=3.0471 lr=2.04e-05 tok=23.032B dt=6481 wait=363 +07:20:32 step 174880 loss=2.9277 lr=2.04e-05 tok=23.035B dt=6546 wait=363 +07:21:36 step 174900 loss=3.0579 lr=2.04e-05 tok=23.037B dt=6610 wait=363 +07:21:49 pushed ckpt_w171.pt,log_w171.txt +07:23:05 waited 76s for global r=722 +07:23:12 global r=722 load 4.287 -> 4.284 +07:24:17 step 174920 loss=2.8717 lr=2.04e-05 tok=23.040B dt=6771 wait=363 +07:25:21 step 174940 loss=3.0591 lr=2.04e-05 tok=23.042B dt=6835 wait=363 +07:26:25 step 174960 loss=2.9730 lr=2.04e-05 tok=23.045B dt=6899 wait=364 +07:27:30 step 174980 loss=2.9145 lr=2.04e-05 tok=23.048B dt=6964 wait=364 +07:28:34 step 175000 loss=3.0029 lr=2.03e-05 tok=23.050B dt=7028 wait=364 +07:29:39 step 175020 loss=3.2337 lr=2.03e-05 tok=23.053B dt=7093 wait=364 +07:30:43 step 175040 loss=2.9183 lr=2.03e-05 tok=23.056B dt=7157 wait=364 +07:31:33 pushed ckpt_w171.pt,log_w171.txt +07:32:49 waited 76s for global r=723 +07:32:54 global r=723 load 4.283 -> 4.282 +07:33:28 step 175060 loss=3.0425 lr=2.03e-05 tok=23.058B dt=7322 wait=364 +07:34:32 step 175080 loss=2.8849 lr=2.03e-05 tok=23.061B dt=7386 wait=364 +07:35:37 step 175100 loss=2.9927 lr=2.03e-05 tok=23.063B dt=7451 wait=365 +07:36:42 step 175120 loss=3.0133 lr=2.03e-05 tok=23.066B dt=7516 wait=365 +07:37:47 step 175140 loss=2.7965 lr=2.03e-05 tok=23.069B dt=7581 wait=365 +07:38:51 step 175160 loss=3.0226 lr=2.03e-05 tok=23.071B dt=7645 wait=365 +07:39:56 step 175180 loss=2.3475 lr=2.03e-05 tok=23.074B dt=7710 wait=365 +07:41:01 step 175200 loss=2.7058 lr=2.03e-05 tok=23.077B dt=7776 wait=365 +07:41:14 pushed ckpt_w171.pt,log_w171.txt +07:42:29 waited 76s for global r=724 +07:42:39 global r=724 load 4.287 -> 4.283 +07:43:44 step 175220 loss=2.9822 lr=2.03e-05 tok=23.079B dt=7938 wait=366 +07:44:48 step 175240 loss=3.0105 lr=2.02e-05 tok=23.082B dt=8002 wait=366 +07:45:52 step 175260 loss=2.9266 lr=2.02e-05 tok=23.084B dt=8066 wait=366 +07:46:57 step 175280 loss=2.8443 lr=2.02e-05 tok=23.087B dt=8131 wait=366 +07:48:00 step 175300 loss=3.0029 lr=2.02e-05 tok=23.090B dt=8195 wait=366 +07:49:04 step 175320 loss=2.9699 lr=2.02e-05 tok=23.092B dt=8258 wait=366 +07:50:08 step 175340 loss=2.8618 lr=2.02e-05 tok=23.095B dt=8322 wait=366 +07:50:55 pushed ckpt_w171.pt,log_w171.txt +07:51:55 waited 61s for global r=725 +07:52:01 global r=725 load 4.292 -> 4.286 +07:52:34 step 175360 loss=2.8706 lr=2.02e-05 tok=23.098B dt=8468 wait=367 +07:53:37 step 175380 loss=2.8252 lr=2.02e-05 tok=23.100B dt=8531 wait=367 +07:54:41 step 175400 loss=2.8126 lr=2.02e-05 tok=23.103B dt=8595 wait=367 +07:55:45 step 175420 loss=2.8747 lr=2.02e-05 tok=23.105B dt=8660 wait=367 +07:56:50 step 175440 loss=2.7732 lr=2.02e-05 tok=23.108B dt=8724 wait=367 +07:57:53 step 175460 loss=2.8997 lr=2.02e-05 tok=23.111B dt=8787 wait=367 +07:58:57 step 175480 loss=2.8251 lr=2.01e-05 tok=23.113B dt=8851 wait=367 +08:00:01 step 175500 loss=2.8027 lr=2.01e-05 tok=23.116B dt=8915 wait=368 +08:00:14 pushed ckpt_w171.pt,log_w171.txt +08:01:29 waited 76s for global r=726 +08:01:35 global r=726 load 4.283 -> 4.283 +08:02:40 step 175520 loss=2.9203 lr=2.01e-05 tok=23.118B dt=9074 wait=368 +08:03:44 step 175540 loss=2.8421 lr=2.01e-05 tok=23.121B dt=9138 wait=368 +08:04:48 step 175560 loss=2.8052 lr=2.01e-05 tok=23.124B dt=9202 wait=368 +08:05:52 step 175580 loss=2.7520 lr=2.01e-05 tok=23.126B dt=9267 wait=368 +08:06:57 step 175600 loss=2.8187 lr=2.01e-05 tok=23.129B dt=9331 wait=368 +08:08:01 step 175620 loss=2.7816 lr=2.01e-05 tok=23.132B dt=9395 wait=368 +08:09:05 step 175640 loss=2.9002 lr=2.01e-05 tok=23.134B dt=9459 wait=369 +08:09:49 pushed ckpt_w171.pt,log_w171.txt +08:11:20 waited 91s for global r=727 +08:11:26 global r=727 load 4.281 -> 4.284 +08:12:00 step 175660 loss=3.0153 lr=2.01e-05 tok=23.137B dt=9634 wait=369 +08:13:03 step 175680 loss=2.9420 lr=2.01e-05 tok=23.139B dt=9697 wait=369 +08:14:07 step 175700 loss=3.0479 lr=2.01e-05 tok=23.142B dt=9761 wait=369 +08:15:12 step 175720 loss=3.0420 lr=2.00e-05 tok=23.145B dt=9826 wait=369 +08:16:15 step 175740 loss=2.9926 lr=2.00e-05 tok=23.147B dt=9890 wait=369 +08:17:20 step 175760 loss=2.9040 lr=2.00e-05 tok=23.150B dt=9954 wait=370 +08:18:25 step 175780 loss=2.9158 lr=2.00e-05 tok=23.153B dt=10019 wait=370 +08:19:29 step 175800 loss=2.9912 lr=2.00e-05 tok=23.155B dt=10083 wait=370 +08:19:41 pushed ckpt_w171.pt,log_w171.txt +08:20:57 waited 76s for global r=728 +08:21:04 global r=728 load 4.293 -> 4.285 +08:22:10 step 175820 loss=2.9302 lr=2.00e-05 tok=23.158B dt=10244 wait=370 +08:23:14 step 175840 loss=3.0278 lr=2.00e-05 tok=23.160B dt=10308 wait=370 +08:24:17 step 175860 loss=2.9776 lr=2.00e-05 tok=23.163B dt=10371 wait=370 +08:25:21 step 175880 loss=2.9377 lr=2.00e-05 tok=23.166B dt=10435 wait=370 +08:26:25 step 175900 loss=2.8831 lr=2.00e-05 tok=23.168B dt=10499 wait=371 +08:27:28 step 175920 loss=2.9549 lr=2.00e-05 tok=23.171B dt=10563 wait=371 +08:28:33 step 175940 loss=2.7945 lr=2.00e-05 tok=23.174B dt=10627 wait=371 +08:29:19 pushed ckpt_w171.pt,log_w171.txt +08:32:54 step 175960 loss=2.9533 lr=2.00e-05 tok=23.176B dt=10888 wait=371 +08:33:58 step 175980 loss=2.8665 lr=1.99e-05 tok=23.179B dt=10952 wait=371 +08:35:03 step 176000 loss=2.8449 lr=1.99e-05 tok=23.181B dt=11017 wait=371 +08:36:07 step 176020 loss=2.8286 lr=1.99e-05 tok=23.184B dt=11081 wait=371 +08:37:10 step 176040 loss=2.9320 lr=1.99e-05 tok=23.187B dt=11144 wait=372 +08:38:13 step 176060 loss=2.8851 lr=1.99e-05 tok=23.189B dt=11208 wait=372 +08:39:17 step 176080 loss=2.9248 lr=1.99e-05 tok=23.192B dt=11271 wait=372 +08:40:21 step 176100 loss=2.9724 lr=1.99e-05 tok=23.195B dt=11335 wait=372 +08:40:33 pushed ckpt_w171.pt,log_w171.txt +08:44:40 step 176120 loss=2.7998 lr=1.99e-05 tok=23.197B dt=11594 wait=372 +08:45:43 step 176140 loss=3.0976 lr=1.99e-05 tok=23.200B dt=11657 wait=372 +08:46:47 step 176160 loss=3.0450 lr=1.99e-05 tok=23.202B dt=11721 wait=372 +08:47:52 step 176180 loss=3.0045 lr=1.99e-05 tok=23.205B dt=11786 wait=373 +08:48:57 step 176200 loss=3.0027 lr=1.99e-05 tok=23.208B dt=11851 wait=373 +08:50:01 step 176220 loss=3.0567 lr=1.99e-05 tok=23.210B dt=11915 wait=373 +08:51:05 step 176240 loss=2.7175 lr=1.98e-05 tok=23.213B dt=11979 wait=373 +08:51:50 pushed ckpt_w171.pt,log_w171.txt +08:55:25 step 176260 loss=2.9297 lr=1.98e-05 tok=23.215B dt=12239 wait=373 +08:56:29 step 176280 loss=2.9604 lr=1.98e-05 tok=23.218B dt=12303 wait=373 +08:57:33 step 176300 loss=3.0240 lr=1.98e-05 tok=23.221B dt=12367 wait=374 +08:58:37 step 176320 loss=2.9418 lr=1.98e-05 tok=23.223B dt=12431 wait=374 +08:59:41 step 176340 loss=2.9663 lr=1.98e-05 tok=23.226B dt=12496 wait=374 +09:00:45 step 176360 loss=3.0742 lr=1.98e-05 tok=23.229B dt=12559 wait=374 +09:01:49 step 176380 loss=3.1032 lr=1.98e-05 tok=23.231B dt=12623 wait=374 +09:02:53 step 176400 loss=2.9460 lr=1.98e-05 tok=23.234B dt=12687 wait=374 +09:03:05 pushed ckpt_w171.pt,log_w171.txt +09:07:13 step 176420 loss=2.8450 lr=1.98e-05 tok=23.236B dt=12947 wait=375 +09:08:17 step 176440 loss=2.9753 lr=1.98e-05 tok=23.239B dt=13011 wait=375 +09:09:21 step 176460 loss=3.0475 lr=1.98e-05 tok=23.242B dt=13075 wait=375 +09:10:25 step 176480 loss=2.9345 lr=1.97e-05 tok=23.244B dt=13139 wait=375 +09:11:29 step 176500 loss=3.0511 lr=1.97e-05 tok=23.247B dt=13203 wait=375 +09:12:33 step 176520 loss=3.0505 lr=1.97e-05 tok=23.250B dt=13267 wait=375 +09:13:37 step 176540 loss=3.0089 lr=1.97e-05 tok=23.252B dt=13331 wait=375 diff --git a/log_w172.txt b/log_w172.txt new file mode 100644 index 0000000000000000000000000000000000000000..254608acec9b0f9a33c981efde27a585694ae5f5 --- /dev/null +++ b/log_w172.txt @@ -0,0 +1,175 @@ +launch +12:06:58 name=w172 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 119/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:07:04 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:07:04 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:07:14 params 1.08B +12:07:15 resumed step=177150 seen=23.33B round=729 +12:07:15 stream salt=177150 +12:09:28 blocks compiled +12:09:28 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:13:13 step 177151 loss=3.0305 lr=1.95e-05 tok=23.332B dt=226 wait=188 +12:13:36 step 177152 loss=3.0251 lr=1.95e-05 tok=23.332B dt=249 wait=188 +12:14:03 step 177160 loss=3.3263 lr=1.95e-05 tok=23.333B dt=275 wait=188 +12:15:11 step 177180 loss=3.0923 lr=1.95e-05 tok=23.336B dt=343 wait=189 +12:16:15 step 177200 loss=3.0752 lr=1.95e-05 tok=23.339B dt=408 wait=189 +12:17:20 step 177220 loss=3.2081 lr=1.95e-05 tok=23.341B dt=472 wait=189 +12:18:23 step 177240 loss=3.1144 lr=1.95e-05 tok=23.344B dt=536 wait=189 +12:19:28 step 177260 loss=3.1602 lr=1.94e-05 tok=23.347B dt=600 wait=190 +12:20:32 step 177280 loss=3.0989 lr=1.94e-05 tok=23.349B dt=664 wait=190 +12:21:36 step 177300 loss=3.1788 lr=1.94e-05 tok=23.352B dt=729 wait=190 +12:21:51 pushed ckpt_w172.pt,log_w172.txt +12:25:50 global r=730 load 4.283 -> 4.280 +12:26:55 step 177320 loss=3.0508 lr=1.94e-05 tok=23.356B dt=1047 wait=190 +12:27:59 step 177340 loss=3.2004 lr=1.94e-05 tok=23.358B dt=1112 wait=190 +12:29:04 step 177360 loss=3.2103 lr=1.94e-05 tok=23.361B dt=1176 wait=190 +12:30:09 step 177380 loss=3.1691 lr=1.94e-05 tok=23.364B dt=1241 wait=190 +12:31:13 step 177400 loss=3.1477 lr=1.94e-05 tok=23.366B dt=1306 wait=191 +12:32:18 step 177420 loss=3.2627 lr=1.94e-05 tok=23.369B dt=1371 wait=191 +12:33:22 step 177440 loss=3.2458 lr=1.94e-05 tok=23.371B dt=1435 wait=191 +12:34:15 pushed ckpt_w172.pt,log_w172.txt +12:37:02 waited 167s for global r=731 +12:37:10 global r=731 load 4.285 -> 4.278 +12:37:45 step 177460 loss=3.2884 lr=1.94e-05 tok=23.374B dt=1697 wait=191 +12:38:49 step 177480 loss=3.0752 lr=1.94e-05 tok=23.377B dt=1761 wait=191 +12:39:54 step 177500 loss=3.1144 lr=1.94e-05 tok=23.379B dt=1827 wait=191 +12:40:59 step 177520 loss=3.2425 lr=1.94e-05 tok=23.382B dt=1891 wait=192 +12:42:03 step 177540 loss=3.1364 lr=1.93e-05 tok=23.385B dt=1955 wait=192 +12:43:08 step 177560 loss=3.2600 lr=1.93e-05 tok=23.387B dt=2021 wait=192 +12:44:13 step 177580 loss=3.2638 lr=1.93e-05 tok=23.390B dt=2085 wait=192 +12:45:17 step 177600 loss=3.1747 lr=1.93e-05 tok=23.392B dt=2150 wait=192 +12:45:30 pushed ckpt_w172.pt,log_w172.txt +12:46:46 waited 77s for global r=732 +12:46:55 global r=732 load 4.287 -> 4.280 +12:48:01 step 177620 loss=3.1250 lr=1.93e-05 tok=23.395B dt=2314 wait=192 +12:49:06 step 177640 loss=3.1756 lr=1.93e-05 tok=23.398B dt=2379 wait=192 +12:50:11 step 177660 loss=3.1765 lr=1.93e-05 tok=23.400B dt=2443 wait=193 +12:51:16 step 177680 loss=3.0527 lr=1.93e-05 tok=23.403B dt=2508 wait=193 +12:52:20 step 177700 loss=3.1564 lr=1.93e-05 tok=23.406B dt=2573 wait=193 +12:53:25 step 177720 loss=3.2226 lr=1.93e-05 tok=23.408B dt=2637 wait=193 +12:54:29 step 177740 loss=3.1814 lr=1.93e-05 tok=23.411B dt=2701 wait=193 +12:55:16 pushed ckpt_w172.pt,log_w172.txt +12:56:17 waited 61s for global r=733 +12:56:23 global r=733 load 4.285 -> 4.282 +12:56:57 step 177760 loss=3.1374 lr=1.93e-05 tok=23.413B dt=2849 wait=193 +12:58:01 step 177780 loss=3.1097 lr=1.93e-05 tok=23.416B dt=2913 wait=193 +12:59:05 step 177800 loss=3.0490 lr=1.92e-05 tok=23.419B dt=2978 wait=194 +13:00:10 step 177820 loss=3.1891 lr=1.92e-05 tok=23.421B dt=3042 wait=194 +13:01:15 step 177840 loss=3.1628 lr=1.92e-05 tok=23.424B dt=3107 wait=194 +13:02:20 step 177860 loss=3.2009 lr=1.92e-05 tok=23.426B dt=3172 wait=194 +13:03:24 step 177880 loss=3.1724 lr=1.92e-05 tok=23.429B dt=3237 wait=194 +13:04:29 step 177900 loss=3.1622 lr=1.92e-05 tok=23.432B dt=3301 wait=194 +13:04:47 pushed ckpt_w172.pt,log_w172.txt +13:05:56 waited 69s for global r=734 +13:06:03 global r=734 load 4.275 -> 4.283 +13:07:09 step 177920 loss=3.2419 lr=1.92e-05 tok=23.434B dt=3461 wait=194 +13:08:13 step 177940 loss=3.2011 lr=1.92e-05 tok=23.437B dt=3526 wait=194 +13:09:18 step 177960 loss=3.1427 lr=1.92e-05 tok=23.440B dt=3590 wait=195 +13:10:22 step 177980 loss=3.1753 lr=1.92e-05 tok=23.442B dt=3654 wait=195 +13:11:26 step 178000 loss=3.1830 lr=1.92e-05 tok=23.445B dt=3718 wait=195 +13:12:30 step 178020 loss=3.2362 lr=1.92e-05 tok=23.447B dt=3783 wait=195 +13:13:34 step 178040 loss=3.1205 lr=1.92e-05 tok=23.450B dt=3847 wait=195 +13:14:19 pushed ckpt_w172.pt,log_w172.txt +13:15:42 waited 82s for global r=735 +13:15:48 global r=735 load 4.287 -> 4.283 +13:16:21 step 178060 loss=3.1819 lr=1.92e-05 tok=23.453B dt=4013 wait=195 +13:17:25 step 178080 loss=3.0891 lr=1.91e-05 tok=23.455B dt=4077 wait=196 +13:18:29 step 178100 loss=2.9835 lr=1.91e-05 tok=23.458B dt=4141 wait=196 +13:19:33 step 178120 loss=3.1022 lr=1.91e-05 tok=23.461B dt=4205 wait=196 +13:20:38 step 178140 loss=2.9731 lr=1.91e-05 tok=23.463B dt=4270 wait=196 +13:21:42 step 178160 loss=3.1247 lr=1.91e-05 tok=23.466B dt=4334 wait=196 +13:22:46 step 178180 loss=3.0552 lr=1.91e-05 tok=23.468B dt=4399 wait=196 +13:23:52 step 178200 loss=2.9774 lr=1.91e-05 tok=23.471B dt=4465 wait=196 +13:24:08 pushed ckpt_w172.pt,log_w172.txt +13:25:30 waited 82s for global r=736 +13:25:41 global r=736 load 4.280 -> 4.283 +13:26:47 step 178220 loss=3.0400 lr=1.91e-05 tok=23.474B dt=4639 wait=197 +13:27:51 step 178240 loss=3.0698 lr=1.91e-05 tok=23.476B dt=4704 wait=197 +13:28:57 step 178260 loss=3.1310 lr=1.91e-05 tok=23.479B dt=4769 wait=197 +13:30:02 step 178280 loss=3.1218 lr=1.91e-05 tok=23.482B dt=4834 wait=197 +13:31:07 step 178300 loss=3.1256 lr=1.91e-05 tok=23.484B dt=4899 wait=197 +13:32:12 step 178320 loss=3.1423 lr=1.91e-05 tok=23.487B dt=4964 wait=197 +13:33:16 step 178340 loss=3.1729 lr=1.90e-05 tok=23.489B dt=5029 wait=197 +13:34:07 pushed ckpt_w172.pt,log_w172.txt +13:35:08 waited 61s for global r=737 +13:35:21 global r=737 load 4.283 -> 4.282 +13:35:55 step 178360 loss=3.2612 lr=1.90e-05 tok=23.492B dt=5187 wait=198 +13:36:59 step 178380 loss=3.1043 lr=1.90e-05 tok=23.495B dt=5251 wait=198 +13:38:04 step 178400 loss=3.0078 lr=1.90e-05 tok=23.497B dt=5316 wait=198 +13:39:10 step 178420 loss=3.0178 lr=1.90e-05 tok=23.500B dt=5382 wait=198 +13:40:14 step 178440 loss=3.0803 lr=1.90e-05 tok=23.503B dt=5447 wait=198 +13:41:19 step 178460 loss=3.0522 lr=1.90e-05 tok=23.505B dt=5511 wait=198 +13:42:23 step 178480 loss=2.9993 lr=1.90e-05 tok=23.508B dt=5575 wait=198 +13:43:27 step 178500 loss=3.0830 lr=1.90e-05 tok=23.510B dt=5639 wait=199 +13:43:43 pushed ckpt_w172.pt,log_w172.txt +13:44:45 waited 62s for global r=738 +13:44:54 global r=738 load 4.282 -> 4.284 +13:45:59 step 178520 loss=2.9541 lr=1.90e-05 tok=23.513B dt=5792 wait=199 +13:47:03 step 178540 loss=3.0768 lr=1.90e-05 tok=23.516B dt=5855 wait=199 +13:48:07 step 178560 loss=3.0060 lr=1.90e-05 tok=23.518B dt=5920 wait=199 +13:49:11 step 178580 loss=3.0335 lr=1.90e-05 tok=23.521B dt=5984 wait=199 +13:50:15 step 178600 loss=2.9589 lr=1.90e-05 tok=23.523B dt=6048 wait=199 +13:51:20 step 178620 loss=3.1570 lr=1.89e-05 tok=23.526B dt=6112 wait=199 +13:52:25 step 178640 loss=3.0733 lr=1.89e-05 tok=23.529B dt=6177 wait=200 +13:53:12 pushed ckpt_w172.pt,log_w172.txt +13:54:28 waited 76s for global r=739 +13:54:36 global r=739 load 4.283 -> 4.282 +13:55:10 step 178660 loss=3.1883 lr=1.89e-05 tok=23.531B dt=6342 wait=200 +13:56:14 step 178680 loss=3.1259 lr=1.89e-05 tok=23.534B dt=6407 wait=200 +13:57:18 step 178700 loss=3.1877 lr=1.89e-05 tok=23.537B dt=6471 wait=200 +13:58:23 step 178720 loss=3.3106 lr=1.89e-05 tok=23.539B dt=6535 wait=200 +13:59:28 step 178740 loss=3.1781 lr=1.89e-05 tok=23.542B dt=6600 wait=200 +14:00:32 step 178760 loss=3.2036 lr=1.89e-05 tok=23.544B dt=6664 wait=201 +14:01:37 step 178780 loss=3.0509 lr=1.89e-05 tok=23.547B dt=6729 wait=201 +14:02:40 step 178800 loss=3.0027 lr=1.89e-05 tok=23.550B dt=6793 wait=201 +14:02:54 pushed ckpt_w172.pt,log_w172.txt +14:04:10 waited 76s for global r=740 +14:04:17 global r=740 load 4.277 -> 4.279 +14:05:23 step 178820 loss=3.0512 lr=1.89e-05 tok=23.552B dt=6955 wait=201 +14:06:27 step 178840 loss=3.1304 lr=1.89e-05 tok=23.555B dt=7019 wait=201 +14:07:32 step 178860 loss=2.9584 lr=1.89e-05 tok=23.558B dt=7084 wait=201 +14:08:36 step 178880 loss=3.0310 lr=1.89e-05 tok=23.560B dt=7149 wait=202 +14:09:40 step 178900 loss=3.0812 lr=1.88e-05 tok=23.563B dt=7212 wait=202 +14:10:44 step 178920 loss=3.0507 lr=1.88e-05 tok=23.565B dt=7276 wait=202 +14:11:49 step 178940 loss=3.0421 lr=1.88e-05 tok=23.568B dt=7341 wait=202 +14:12:33 pushed ckpt_w172.pt,log_w172.txt +14:13:48 waited 76s for global r=741 +14:14:00 global r=741 load 4.281 -> 4.280 +14:14:36 step 178960 loss=3.1412 lr=1.88e-05 tok=23.571B dt=7508 wait=202 +14:15:40 step 178980 loss=3.1716 lr=1.88e-05 tok=23.573B dt=7573 wait=202 +14:16:45 step 179000 loss=3.1753 lr=1.88e-05 tok=23.576B dt=7637 wait=202 +14:17:50 step 179020 loss=3.2096 lr=1.88e-05 tok=23.579B dt=7702 wait=203 +14:18:55 step 179040 loss=3.0705 lr=1.88e-05 tok=23.581B dt=7767 wait=203 +14:19:59 step 179060 loss=3.2521 lr=1.88e-05 tok=23.584B dt=7831 wait=203 +14:21:04 step 179080 loss=3.2283 lr=1.88e-05 tok=23.586B dt=7896 wait=203 +14:22:08 step 179100 loss=3.1204 lr=1.88e-05 tok=23.589B dt=7960 wait=203 +14:22:25 pushed ckpt_w172.pt,log_w172.txt +14:23:26 waited 62s for global r=742 +14:23:35 global r=742 load 4.277 -> 4.282 +14:24:40 step 179120 loss=3.1827 lr=1.88e-05 tok=23.592B dt=8113 wait=203 +14:25:44 step 179140 loss=3.1128 lr=1.88e-05 tok=23.594B dt=8176 wait=204 +14:26:49 step 179160 loss=3.1639 lr=1.88e-05 tok=23.597B dt=8241 wait=204 +14:27:54 step 179180 loss=3.1262 lr=1.88e-05 tok=23.600B dt=8306 wait=204 +14:28:58 step 179200 loss=3.2534 lr=1.87e-05 tok=23.602B dt=8371 wait=204 +14:30:02 step 179220 loss=3.1883 lr=1.87e-05 tok=23.605B dt=8434 wait=204 +14:31:07 step 179240 loss=3.1252 lr=1.87e-05 tok=23.607B dt=8499 wait=204 +14:31:52 pushed ckpt_w172.pt,log_w172.txt +14:33:13 waited 81s for global r=743 +14:33:25 global r=743 load 4.277 -> 4.281 +14:33:58 step 179260 loss=3.1641 lr=1.87e-05 tok=23.610B dt=8671 wait=205 +14:35:02 step 179280 loss=3.1271 lr=1.87e-05 tok=23.613B dt=8735 wait=205 +14:36:07 step 179300 loss=3.1059 lr=1.87e-05 tok=23.615B dt=8799 wait=205 +14:37:11 step 179320 loss=3.0019 lr=1.87e-05 tok=23.618B dt=8863 wait=205 +14:38:15 step 179340 loss=2.9397 lr=1.87e-05 tok=23.620B dt=8928 wait=205 +14:39:19 step 179360 loss=3.0338 lr=1.87e-05 tok=23.623B dt=8991 wait=205 +14:40:22 step 179380 loss=2.8709 lr=1.87e-05 tok=23.626B dt=9055 wait=206 +14:41:26 step 179400 loss=2.9825 lr=1.87e-05 tok=23.628B dt=9118 wait=206 +14:41:38 pushed ckpt_w172.pt,log_w172.txt +14:42:54 waited 76s for global r=744 +14:43:04 global r=744 load 4.280 -> 4.280 +14:44:08 step 179420 loss=2.9193 lr=1.87e-05 tok=23.631B dt=9281 wait=206 +14:45:12 step 179440 loss=2.9928 lr=1.87e-05 tok=23.634B dt=9344 wait=206 +14:46:16 step 179460 loss=2.8086 lr=1.87e-05 tok=23.636B dt=9408 wait=206 +14:47:20 step 179480 loss=2.8194 lr=1.86e-05 tok=23.639B dt=9472 wait=206 +14:48:23 step 179500 loss=2.8944 lr=1.86e-05 tok=23.641B dt=9536 wait=206 +14:49:27 step 179520 loss=2.9915 lr=1.86e-05 tok=23.644B dt=9599 wait=207 +14:50:31 step 179540 loss=2.8229 lr=1.86e-05 tok=23.647B dt=9663 wait=207 diff --git a/log_w173.txt b/log_w173.txt new file mode 100644 index 0000000000000000000000000000000000000000..58d37941912f1cde99aafb00c957c87140042300 --- /dev/null +++ b/log_w173.txt @@ -0,0 +1,176 @@ +launch +12:07:05 name=w173 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 120/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:07:11 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:07:11 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:07:21 params 1.08B +12:07:22 resumed step=177150 seen=23.33B round=729 +12:07:22 stream salt=177150 +12:09:33 blocks compiled +12:09:33 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:15:50 step 177151 loss=2.6771 lr=1.95e-05 tok=23.332B dt=377 wait=350 +12:16:05 step 177152 loss=2.9046 lr=1.95e-05 tok=23.332B dt=392 wait=351 +12:16:30 step 177160 loss=3.2269 lr=1.95e-05 tok=23.333B dt=418 wait=351 +12:17:35 step 177180 loss=3.0282 lr=1.95e-05 tok=23.336B dt=482 wait=351 +12:18:40 step 177200 loss=3.0963 lr=1.95e-05 tok=23.339B dt=547 wait=351 +12:19:44 step 177220 loss=2.9987 lr=1.95e-05 tok=23.341B dt=611 wait=351 +12:20:47 step 177240 loss=3.0543 lr=1.95e-05 tok=23.344B dt=675 wait=351 +12:21:52 step 177260 loss=3.1473 lr=1.94e-05 tok=23.347B dt=739 wait=351 +12:22:56 step 177280 loss=3.0448 lr=1.94e-05 tok=23.349B dt=804 wait=351 +12:24:00 step 177300 loss=3.0774 lr=1.94e-05 tok=23.352B dt=868 wait=351 +12:24:14 pushed ckpt_w173.pt,log_w173.txt +12:24:59 waited 45s for global r=730 +12:25:18 global r=730 load 4.282 -> 4.280 +12:26:26 step 177320 loss=3.0059 lr=1.94e-05 tok=23.354B dt=1013 wait=352 +12:27:29 step 177340 loss=3.0679 lr=1.94e-05 tok=23.357B dt=1077 wait=352 +12:28:33 step 177360 loss=3.0712 lr=1.94e-05 tok=23.360B dt=1140 wait=352 +12:29:37 step 177380 loss=2.9417 lr=1.94e-05 tok=23.362B dt=1204 wait=352 +12:30:41 step 177400 loss=2.9834 lr=1.94e-05 tok=23.365B dt=1268 wait=352 +12:31:46 step 177420 loss=3.0598 lr=1.94e-05 tok=23.368B dt=1333 wait=352 +12:32:50 step 177440 loss=2.9848 lr=1.94e-05 tok=23.370B dt=1397 wait=352 +12:33:36 pushed ckpt_w173.pt,log_w173.txt +12:37:13 step 177460 loss=3.0187 lr=1.94e-05 tok=23.373B dt=1660 wait=353 +12:37:21 global r=731 load 4.279 -> 4.278 +12:37:53 step 177460 loss=3.0297 lr=1.94e-05 tok=23.374B dt=1701 wait=353 +12:38:57 step 177480 loss=2.8773 lr=1.94e-05 tok=23.377B dt=1764 wait=353 +12:40:01 step 177500 loss=3.0293 lr=1.94e-05 tok=23.379B dt=1828 wait=353 +12:41:05 step 177520 loss=3.1118 lr=1.94e-05 tok=23.382B dt=1892 wait=353 +12:42:08 step 177540 loss=2.9780 lr=1.93e-05 tok=23.385B dt=1955 wait=353 +12:43:13 step 177560 loss=3.0375 lr=1.93e-05 tok=23.387B dt=2020 wait=353 +12:44:17 step 177580 loss=2.9312 lr=1.93e-05 tok=23.390B dt=2084 wait=354 +12:45:21 step 177600 loss=2.8669 lr=1.93e-05 tok=23.392B dt=2149 wait=354 +12:45:33 pushed ckpt_w173.pt,log_w173.txt +12:46:49 waited 76s for global r=732 +12:46:54 global r=732 load 4.287 -> 4.280 +12:48:00 step 177620 loss=3.0387 lr=1.93e-05 tok=23.395B dt=2307 wait=354 +12:49:04 step 177640 loss=3.0098 lr=1.93e-05 tok=23.398B dt=2371 wait=354 +12:50:08 step 177660 loss=2.9653 lr=1.93e-05 tok=23.400B dt=2436 wait=354 +12:51:13 step 177680 loss=2.9635 lr=1.93e-05 tok=23.403B dt=2500 wait=354 +12:52:17 step 177700 loss=2.7638 lr=1.93e-05 tok=23.406B dt=2565 wait=354 +12:53:22 step 177720 loss=3.0395 lr=1.93e-05 tok=23.408B dt=2629 wait=355 +12:54:25 step 177740 loss=2.9045 lr=1.93e-05 tok=23.411B dt=2693 wait=355 +12:55:09 pushed ckpt_w173.pt,log_w173.txt +12:56:10 waited 60s for global r=733 +12:56:16 global r=733 load 4.284 -> 4.282 +12:56:49 step 177760 loss=2.9940 lr=1.93e-05 tok=23.413B dt=2837 wait=355 +12:57:54 step 177780 loss=2.8080 lr=1.93e-05 tok=23.416B dt=2901 wait=355 +12:58:57 step 177800 loss=2.8891 lr=1.92e-05 tok=23.419B dt=2965 wait=355 +13:00:02 step 177820 loss=3.0056 lr=1.92e-05 tok=23.421B dt=3029 wait=355 +13:01:05 step 177840 loss=2.9087 lr=1.92e-05 tok=23.424B dt=3093 wait=355 +13:02:10 step 177860 loss=2.9904 lr=1.92e-05 tok=23.426B dt=3157 wait=355 +13:03:13 step 177880 loss=2.9644 lr=1.92e-05 tok=23.429B dt=3221 wait=356 +13:04:17 step 177900 loss=2.9076 lr=1.92e-05 tok=23.432B dt=3285 wait=356 +13:04:29 pushed ckpt_w173.pt,log_w173.txt +13:06:00 waited 91s for global r=734 +13:06:04 global r=734 load 4.283 -> 4.283 +13:07:09 step 177920 loss=2.8848 lr=1.92e-05 tok=23.434B dt=3457 wait=356 +13:08:13 step 177940 loss=2.9095 lr=1.92e-05 tok=23.437B dt=3521 wait=356 +13:09:18 step 177960 loss=2.8794 lr=1.92e-05 tok=23.440B dt=3585 wait=356 +13:10:22 step 177980 loss=2.9259 lr=1.92e-05 tok=23.442B dt=3649 wait=356 +13:11:25 step 178000 loss=2.9159 lr=1.92e-05 tok=23.445B dt=3713 wait=356 +13:12:29 step 178020 loss=2.9699 lr=1.92e-05 tok=23.447B dt=3777 wait=357 +13:13:33 step 178040 loss=2.9118 lr=1.92e-05 tok=23.450B dt=3840 wait=357 +13:14:20 pushed ckpt_w173.pt,log_w173.txt +13:15:36 waited 76s for global r=735 +13:15:43 global r=735 load 4.288 -> 4.283 +13:16:16 step 178060 loss=2.9632 lr=1.92e-05 tok=23.453B dt=4003 wait=357 +13:17:19 step 178080 loss=2.9396 lr=1.91e-05 tok=23.455B dt=4067 wait=357 +13:18:24 step 178100 loss=2.9176 lr=1.91e-05 tok=23.458B dt=4131 wait=357 +13:19:29 step 178120 loss=2.7859 lr=1.91e-05 tok=23.461B dt=4196 wait=357 +13:20:33 step 178140 loss=2.9396 lr=1.91e-05 tok=23.463B dt=4260 wait=358 +13:21:37 step 178160 loss=3.2178 lr=1.91e-05 tok=23.466B dt=4324 wait=358 +13:22:41 step 178180 loss=3.0152 lr=1.91e-05 tok=23.468B dt=4389 wait=358 +13:23:46 step 178200 loss=3.0913 lr=1.91e-05 tok=23.471B dt=4453 wait=358 +13:23:58 pushed ckpt_w173.pt,log_w173.txt +13:25:28 waited 91s for global r=736 +13:25:34 global r=736 load 4.286 -> 4.283 +13:26:39 step 178220 loss=3.0690 lr=1.91e-05 tok=23.474B dt=4627 wait=358 +13:27:43 step 178240 loss=3.0815 lr=1.91e-05 tok=23.476B dt=4690 wait=358 +13:28:47 step 178260 loss=3.0096 lr=1.91e-05 tok=23.479B dt=4754 wait=358 +13:29:52 step 178280 loss=2.9963 lr=1.91e-05 tok=23.482B dt=4819 wait=359 +13:30:56 step 178300 loss=2.9903 lr=1.91e-05 tok=23.484B dt=4883 wait=359 +13:32:00 step 178320 loss=2.9850 lr=1.91e-05 tok=23.487B dt=4947 wait=359 +13:33:04 step 178340 loss=2.9363 lr=1.90e-05 tok=23.489B dt=5011 wait=359 +13:33:51 pushed ckpt_w173.pt,log_w173.txt +13:35:06 waited 76s for global r=737 +13:35:13 global r=737 load 4.275 -> 4.282 +13:35:46 step 178360 loss=3.0608 lr=1.90e-05 tok=23.492B dt=5174 wait=359 +13:36:49 step 178380 loss=2.9027 lr=1.90e-05 tok=23.495B dt=5237 wait=359 +13:37:55 step 178400 loss=2.5138 lr=1.90e-05 tok=23.497B dt=5302 wait=359 +13:38:59 step 178420 loss=3.0445 lr=1.90e-05 tok=23.500B dt=5366 wait=359 +13:40:03 step 178440 loss=3.0521 lr=1.90e-05 tok=23.503B dt=5430 wait=360 +13:41:07 step 178460 loss=3.0760 lr=1.90e-05 tok=23.505B dt=5494 wait=360 +13:42:10 step 178480 loss=3.0067 lr=1.90e-05 tok=23.508B dt=5558 wait=360 +13:43:14 step 178500 loss=3.1067 lr=1.90e-05 tok=23.510B dt=5622 wait=360 +13:43:26 pushed ckpt_w173.pt,log_w173.txt +13:44:41 waited 76s for global r=738 +13:44:47 global r=738 load 4.283 -> 4.284 +13:45:52 step 178520 loss=2.9860 lr=1.90e-05 tok=23.513B dt=5779 wait=360 +13:46:55 step 178540 loss=3.1130 lr=1.90e-05 tok=23.516B dt=5843 wait=360 +13:48:00 step 178560 loss=3.0515 lr=1.90e-05 tok=23.518B dt=5907 wait=360 +13:49:04 step 178580 loss=3.0264 lr=1.90e-05 tok=23.521B dt=5971 wait=361 +13:50:07 step 178600 loss=2.9398 lr=1.90e-05 tok=23.523B dt=6034 wait=361 +13:51:11 step 178620 loss=3.0945 lr=1.89e-05 tok=23.526B dt=6098 wait=361 +13:52:15 step 178640 loss=3.0174 lr=1.89e-05 tok=23.529B dt=6162 wait=361 +13:52:59 pushed ckpt_w173.pt,log_w173.txt +13:54:30 waited 91s for global r=739 +13:54:44 global r=739 load 4.291 -> 4.282 +13:55:17 step 178660 loss=3.1106 lr=1.89e-05 tok=23.531B dt=6344 wait=361 +13:56:20 step 178680 loss=2.9670 lr=1.89e-05 tok=23.534B dt=6408 wait=361 +13:57:24 step 178700 loss=2.8440 lr=1.89e-05 tok=23.537B dt=6472 wait=361 +13:58:28 step 178720 loss=3.0942 lr=1.89e-05 tok=23.539B dt=6535 wait=362 +13:59:32 step 178740 loss=2.9259 lr=1.89e-05 tok=23.542B dt=6599 wait=362 +14:00:35 step 178760 loss=3.1018 lr=1.89e-05 tok=23.544B dt=6663 wait=362 +14:01:40 step 178780 loss=2.9875 lr=1.89e-05 tok=23.547B dt=6727 wait=362 +14:02:44 step 178800 loss=2.9608 lr=1.89e-05 tok=23.550B dt=6791 wait=362 +14:02:56 pushed ckpt_w173.pt,log_w173.txt +14:04:11 waited 76s for global r=740 +14:04:16 global r=740 load 4.285 -> 4.279 +14:05:21 step 178820 loss=2.9949 lr=1.89e-05 tok=23.552B dt=6948 wait=362 +14:06:25 step 178840 loss=3.2786 lr=1.89e-05 tok=23.555B dt=7012 wait=362 +14:07:30 step 178860 loss=3.1260 lr=1.89e-05 tok=23.558B dt=7077 wait=363 +14:08:35 step 178880 loss=3.0567 lr=1.89e-05 tok=23.560B dt=7142 wait=363 +14:09:39 step 178900 loss=3.1134 lr=1.88e-05 tok=23.563B dt=7206 wait=363 +14:10:44 step 178920 loss=3.2228 lr=1.88e-05 tok=23.565B dt=7271 wait=363 +14:11:48 step 178940 loss=2.9312 lr=1.88e-05 tok=23.568B dt=7335 wait=363 +14:12:32 pushed ckpt_w173.pt,log_w173.txt +14:14:03 waited 91s for global r=741 +14:14:09 global r=741 load 4.284 -> 4.280 +14:14:42 step 178960 loss=2.9641 lr=1.88e-05 tok=23.571B dt=7509 wait=363 +14:15:46 step 178980 loss=2.8578 lr=1.88e-05 tok=23.573B dt=7573 wait=363 +14:16:50 step 179000 loss=2.8840 lr=1.88e-05 tok=23.576B dt=7637 wait=364 +14:17:54 step 179020 loss=2.8791 lr=1.88e-05 tok=23.579B dt=7701 wait=364 +14:18:58 step 179040 loss=2.9355 lr=1.88e-05 tok=23.581B dt=7765 wait=364 +14:20:01 step 179060 loss=2.9323 lr=1.88e-05 tok=23.584B dt=7829 wait=364 +14:21:05 step 179080 loss=2.9967 lr=1.88e-05 tok=23.586B dt=7892 wait=364 +14:22:09 step 179100 loss=2.8826 lr=1.88e-05 tok=23.589B dt=7956 wait=364 +14:22:20 pushed ckpt_w173.pt,log_w173.txt +14:23:36 waited 76s for global r=742 +14:23:42 global r=742 load 4.282 -> 4.282 +14:24:47 step 179120 loss=2.8754 lr=1.88e-05 tok=23.592B dt=8114 wait=364 +14:25:50 step 179140 loss=2.9778 lr=1.88e-05 tok=23.594B dt=8178 wait=365 +14:26:54 step 179160 loss=2.8819 lr=1.88e-05 tok=23.597B dt=8242 wait=365 +14:27:58 step 179180 loss=2.9418 lr=1.88e-05 tok=23.600B dt=8305 wait=365 +14:29:02 step 179200 loss=2.9572 lr=1.87e-05 tok=23.602B dt=8369 wait=365 +14:30:05 step 179220 loss=3.1223 lr=1.87e-05 tok=23.605B dt=8433 wait=365 +14:31:09 step 179240 loss=2.9704 lr=1.87e-05 tok=23.607B dt=8496 wait=365 +14:31:54 pushed ckpt_w173.pt,log_w173.txt +14:33:24 waited 91s for global r=743 +14:33:31 global r=743 load 4.285 -> 4.281 +14:34:04 step 179260 loss=3.0391 lr=1.87e-05 tok=23.610B dt=8671 wait=365 +14:35:08 step 179280 loss=3.0559 lr=1.87e-05 tok=23.613B dt=8735 wait=366 +14:36:12 step 179300 loss=2.9004 lr=1.87e-05 tok=23.615B dt=8799 wait=366 +14:37:16 step 179320 loss=3.0162 lr=1.87e-05 tok=23.618B dt=8863 wait=366 +14:38:21 step 179340 loss=3.0754 lr=1.87e-05 tok=23.620B dt=8928 wait=366 +14:39:25 step 179360 loss=3.1937 lr=1.87e-05 tok=23.623B dt=8993 wait=366 +14:40:29 step 179380 loss=3.0777 lr=1.87e-05 tok=23.626B dt=9057 wait=366 +14:41:34 step 179400 loss=3.0268 lr=1.87e-05 tok=23.628B dt=9121 wait=366 +14:41:45 pushed ckpt_w173.pt,log_w173.txt +14:43:01 waited 76s for global r=744 +14:43:07 global r=744 load 4.281 -> 4.280 +14:44:12 step 179420 loss=2.9539 lr=1.87e-05 tok=23.631B dt=9279 wait=367 +14:45:17 step 179440 loss=3.0080 lr=1.87e-05 tok=23.634B dt=9344 wait=367 +14:46:21 step 179460 loss=2.9359 lr=1.87e-05 tok=23.636B dt=9408 wait=367 +14:47:25 step 179480 loss=2.9937 lr=1.86e-05 tok=23.639B dt=9473 wait=367 +14:48:29 step 179500 loss=2.7057 lr=1.86e-05 tok=23.641B dt=9537 wait=367 +14:49:33 step 179520 loss=3.0974 lr=1.86e-05 tok=23.644B dt=9601 wait=367 +14:50:38 step 179540 loss=2.9130 lr=1.86e-05 tok=23.647B dt=9665 wait=367 diff --git a/log_w174.txt b/log_w174.txt new file mode 100644 index 0000000000000000000000000000000000000000..74cb2bda823a36efef791174da5190a013a1bd2b --- /dev/null +++ b/log_w174.txt @@ -0,0 +1,365 @@ +launch +12:09:48 name=w174 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 121/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:09:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:09:54 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:10:03 params 1.08B +12:10:05 resumed step=177150 seen=23.33B round=729 +12:10:05 stream salt=177150 +12:11:57 blocks compiled +12:11:57 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:19:23 step 177151 loss=3.0129 lr=1.95e-05 tok=23.332B dt=446 wait=417 +12:19:39 step 177152 loss=3.0771 lr=1.95e-05 tok=23.332B dt=462 wait=417 +12:20:04 step 177160 loss=3.2674 lr=1.95e-05 tok=23.333B dt=487 wait=417 +12:21:09 step 177180 loss=3.1035 lr=1.95e-05 tok=23.336B dt=552 wait=417 +12:22:14 step 177200 loss=3.1557 lr=1.95e-05 tok=23.339B dt=617 wait=417 +12:23:19 step 177220 loss=3.1480 lr=1.95e-05 tok=23.341B dt=682 wait=417 +12:24:23 step 177240 loss=3.2333 lr=1.95e-05 tok=23.344B dt=746 wait=417 +12:25:28 step 177260 loss=3.2918 lr=1.94e-05 tok=23.347B dt=811 wait=417 +12:25:46 global r=730 load 4.281 -> 4.280 +12:26:50 step 177320 loss=3.1405 lr=1.94e-05 tok=23.349B dt=893 wait=418 +12:27:55 step 177340 loss=3.2852 lr=1.94e-05 tok=23.352B dt=958 wait=418 +12:29:00 step 177360 loss=3.3732 lr=1.94e-05 tok=23.354B dt=1023 wait=418 +12:30:05 step 177380 loss=3.1097 lr=1.94e-05 tok=23.357B dt=1088 wait=418 +12:31:10 step 177400 loss=3.3381 lr=1.94e-05 tok=23.360B dt=1153 wait=418 +12:32:15 step 177420 loss=3.3509 lr=1.94e-05 tok=23.362B dt=1218 wait=418 +12:33:20 step 177440 loss=3.3158 lr=1.94e-05 tok=23.365B dt=1283 wait=418 +12:34:08 pushed ckpt_w174.pt,log_w174.txt +12:37:09 waited 181s for global r=731 +12:37:18 global r=731 load 4.288 -> 4.278 +12:37:53 step 177460 loss=3.3554 lr=1.94e-05 tok=23.368B dt=1556 wait=419 +12:38:57 step 177480 loss=3.1559 lr=1.94e-05 tok=23.370B dt=1620 wait=419 +12:40:03 step 177500 loss=2.6867 lr=1.94e-05 tok=23.373B dt=1686 wait=419 +12:41:07 step 177520 loss=3.4002 lr=1.94e-05 tok=23.375B dt=1750 wait=419 +12:42:12 step 177540 loss=3.2441 lr=1.93e-05 tok=23.378B dt=1815 wait=419 +12:43:17 step 177560 loss=3.3884 lr=1.93e-05 tok=23.381B dt=1880 wait=419 +12:44:21 step 177580 loss=3.3330 lr=1.93e-05 tok=23.383B dt=1944 wait=420 +12:45:25 step 177600 loss=3.3116 lr=1.93e-05 tok=23.386B dt=2008 wait=420 +12:45:39 pushed ckpt_w174.pt,log_w174.txt +12:46:54 waited 76s for global r=732 +12:47:01 global r=732 load 4.282 -> 4.280 +12:48:07 step 177620 loss=3.1794 lr=1.93e-05 tok=23.388B dt=2170 wait=420 +12:49:11 step 177640 loss=3.1641 lr=1.93e-05 tok=23.391B dt=2234 wait=420 +12:50:15 step 177660 loss=3.2424 lr=1.93e-05 tok=23.394B dt=2298 wait=420 +12:51:20 step 177680 loss=3.1129 lr=1.93e-05 tok=23.396B dt=2363 wait=420 +12:52:26 step 177700 loss=2.9745 lr=1.93e-05 tok=23.399B dt=2429 wait=420 +12:53:31 step 177720 loss=3.1818 lr=1.93e-05 tok=23.402B dt=2494 wait=420 +12:54:35 step 177740 loss=2.9897 lr=1.93e-05 tok=23.404B dt=2558 wait=421 +12:55:21 pushed ckpt_w174.pt,log_w174.txt +12:56:22 waited 60s for global r=733 +12:56:29 global r=733 load 4.289 -> 4.282 +12:57:02 step 177760 loss=3.1057 lr=1.93e-05 tok=23.407B dt=2705 wait=421 +12:58:07 step 177780 loss=3.0296 lr=1.93e-05 tok=23.409B dt=2770 wait=421 +12:59:10 step 177800 loss=2.9978 lr=1.92e-05 tok=23.412B dt=2833 wait=421 +13:00:15 step 177820 loss=3.0769 lr=1.92e-05 tok=23.415B dt=2898 wait=421 +13:01:19 step 177840 loss=2.9850 lr=1.92e-05 tok=23.417B dt=2962 wait=421 +13:02:24 step 177860 loss=2.9954 lr=1.92e-05 tok=23.420B dt=3027 wait=421 +13:03:28 step 177880 loss=3.0594 lr=1.92e-05 tok=23.423B dt=3091 wait=422 +13:04:32 step 177900 loss=3.0358 lr=1.92e-05 tok=23.425B dt=3155 wait=422 +13:04:47 pushed ckpt_w174.pt,log_w174.txt +13:06:03 waited 76s for global r=734 +13:06:12 global r=734 load 4.282 -> 4.283 +13:07:18 step 177920 loss=3.0509 lr=1.92e-05 tok=23.428B dt=3321 wait=422 +13:08:22 step 177940 loss=3.0199 lr=1.92e-05 tok=23.430B dt=3385 wait=422 +13:09:26 step 177960 loss=3.1401 lr=1.92e-05 tok=23.433B dt=3449 wait=422 +13:10:29 step 177980 loss=3.1326 lr=1.92e-05 tok=23.436B dt=3512 wait=422 +13:11:34 step 178000 loss=3.0669 lr=1.92e-05 tok=23.438B dt=3577 wait=422 +13:12:38 step 178020 loss=3.1121 lr=1.92e-05 tok=23.441B dt=3641 wait=423 +13:13:42 step 178040 loss=3.1032 lr=1.92e-05 tok=23.444B dt=3705 wait=423 +13:14:27 pushed ckpt_w174.pt,log_w174.txt +13:15:42 waited 76s for global r=735 +13:15:50 global r=735 load 4.289 -> 4.283 +13:16:24 step 178060 loss=3.1857 lr=1.92e-05 tok=23.446B dt=3867 wait=423 +13:17:28 step 178080 loss=3.0470 lr=1.91e-05 tok=23.449B dt=3931 wait=423 +13:18:32 step 178100 loss=2.9547 lr=1.91e-05 tok=23.451B dt=3995 wait=423 +13:19:35 step 178120 loss=3.0519 lr=1.91e-05 tok=23.454B dt=4058 wait=423 +13:20:39 step 178140 loss=2.9012 lr=1.91e-05 tok=23.457B dt=4122 wait=423 +13:21:42 step 178160 loss=3.0449 lr=1.91e-05 tok=23.459B dt=4185 wait=424 +13:22:46 step 178180 loss=2.9989 lr=1.91e-05 tok=23.462B dt=4249 wait=424 +13:23:49 step 178200 loss=2.9912 lr=1.91e-05 tok=23.465B dt=4312 wait=424 +13:24:01 pushed ckpt_w174.pt,log_w174.txt +13:25:32 waited 91s for global r=736 +13:25:39 global r=736 load 4.281 -> 4.283 +13:26:44 step 178220 loss=2.9951 lr=1.91e-05 tok=23.467B dt=4487 wait=424 +13:27:47 step 178240 loss=2.9817 lr=1.91e-05 tok=23.470B dt=4550 wait=424 +13:28:51 step 178260 loss=2.9982 lr=1.91e-05 tok=23.472B dt=4613 wait=424 +13:29:54 step 178280 loss=3.0517 lr=1.91e-05 tok=23.475B dt=4677 wait=424 +13:30:58 step 178300 loss=3.0903 lr=1.91e-05 tok=23.478B dt=4741 wait=425 +13:32:02 step 178320 loss=3.1520 lr=1.91e-05 tok=23.480B dt=4805 wait=425 +13:33:06 step 178340 loss=2.9877 lr=1.90e-05 tok=23.483B dt=4869 wait=425 +13:33:52 pushed ckpt_w174.pt,log_w174.txt +13:35:08 waited 76s for global r=737 +13:35:14 global r=737 load 4.287 -> 4.282 +13:35:48 step 178360 loss=3.3181 lr=1.90e-05 tok=23.485B dt=5031 wait=425 +13:36:53 step 178380 loss=3.1292 lr=1.90e-05 tok=23.488B dt=5096 wait=425 +13:37:57 step 178400 loss=3.2451 lr=1.90e-05 tok=23.491B dt=5160 wait=425 +13:39:01 step 178420 loss=3.1735 lr=1.90e-05 tok=23.493B dt=5224 wait=425 +13:40:05 step 178440 loss=3.0104 lr=1.90e-05 tok=23.496B dt=5288 wait=426 +13:41:09 step 178460 loss=2.9016 lr=1.90e-05 tok=23.499B dt=5352 wait=426 +13:42:14 step 178480 loss=2.7039 lr=1.90e-05 tok=23.501B dt=5417 wait=426 +13:43:17 step 178500 loss=2.9632 lr=1.90e-05 tok=23.504B dt=5480 wait=426 +13:43:29 pushed ckpt_w174.pt,log_w174.txt +13:44:45 waited 76s for global r=738 +13:44:52 global r=738 load 4.290 -> 4.284 +13:45:56 step 178520 loss=2.8473 lr=1.90e-05 tok=23.506B dt=5639 wait=426 +13:46:59 step 178540 loss=2.9392 lr=1.90e-05 tok=23.509B dt=5702 wait=426 +13:48:02 step 178560 loss=2.8945 lr=1.90e-05 tok=23.512B dt=5765 wait=427 +13:49:06 step 178580 loss=2.7155 lr=1.90e-05 tok=23.514B dt=5829 wait=427 +13:50:09 step 178600 loss=2.9008 lr=1.90e-05 tok=23.517B dt=5892 wait=427 +13:51:12 step 178620 loss=2.8321 lr=1.89e-05 tok=23.520B dt=5955 wait=427 +13:52:15 step 178640 loss=2.9172 lr=1.89e-05 tok=23.522B dt=6018 wait=427 +13:53:00 pushed ckpt_w174.pt,log_w174.txt +13:54:16 waited 76s for global r=739 +13:54:23 global r=739 load 4.286 -> 4.282 +13:54:56 step 178660 loss=2.9001 lr=1.89e-05 tok=23.525B dt=6179 wait=427 +13:55:59 step 178680 loss=2.9144 lr=1.89e-05 tok=23.527B dt=6242 wait=427 +13:57:04 step 178700 loss=2.6896 lr=1.89e-05 tok=23.530B dt=6307 wait=428 +13:58:07 step 178720 loss=3.0661 lr=1.89e-05 tok=23.533B dt=6370 wait=428 +13:59:10 step 178740 loss=2.9679 lr=1.89e-05 tok=23.535B dt=6433 wait=428 +14:00:13 step 178760 loss=2.9906 lr=1.89e-05 tok=23.538B dt=6496 wait=428 +14:01:17 step 178780 loss=2.8465 lr=1.89e-05 tok=23.541B dt=6559 wait=428 +14:02:21 step 178800 loss=2.7602 lr=1.89e-05 tok=23.543B dt=6624 wait=428 +14:02:33 pushed ckpt_w174.pt,log_w174.txt +14:04:19 waited 106s for global r=740 +14:04:26 global r=740 load 4.287 -> 4.279 +14:05:30 step 178820 loss=2.8920 lr=1.89e-05 tok=23.546B dt=6813 wait=429 +14:06:33 step 178840 loss=2.9058 lr=1.89e-05 tok=23.548B dt=6876 wait=429 +14:07:36 step 178860 loss=2.9758 lr=1.89e-05 tok=23.551B dt=6939 wait=429 +14:08:40 step 178880 loss=2.9148 lr=1.89e-05 tok=23.554B dt=7002 wait=429 +14:09:44 step 178900 loss=3.0440 lr=1.88e-05 tok=23.556B dt=7067 wait=429 +14:10:48 step 178920 loss=2.9586 lr=1.88e-05 tok=23.559B dt=7131 wait=429 +14:11:52 step 178940 loss=2.8661 lr=1.88e-05 tok=23.562B dt=7195 wait=429 +14:12:39 pushed ckpt_w174.pt,log_w174.txt +14:13:54 waited 76s for global r=741 +14:14:00 global r=741 load 4.278 -> 4.280 +14:14:34 step 178960 loss=3.0143 lr=1.88e-05 tok=23.564B dt=7357 wait=430 +14:15:38 step 178980 loss=3.0009 lr=1.88e-05 tok=23.567B dt=7421 wait=430 +14:16:42 step 179000 loss=2.8735 lr=1.88e-05 tok=23.569B dt=7485 wait=430 +14:17:46 step 179020 loss=3.0797 lr=1.88e-05 tok=23.572B dt=7549 wait=430 +14:18:50 step 179040 loss=3.0458 lr=1.88e-05 tok=23.575B dt=7613 wait=430 +14:19:53 step 179060 loss=3.0368 lr=1.88e-05 tok=23.577B dt=7676 wait=430 +14:20:57 step 179080 loss=3.0242 lr=1.88e-05 tok=23.580B dt=7740 wait=430 +14:22:01 step 179100 loss=3.0530 lr=1.88e-05 tok=23.582B dt=7804 wait=431 +14:22:16 pushed ckpt_w174.pt,log_w174.txt +14:23:32 waited 76s for global r=742 +14:23:37 global r=742 load 4.287 -> 4.282 +14:24:42 step 179120 loss=3.0792 lr=1.88e-05 tok=23.585B dt=7965 wait=431 +14:25:46 step 179140 loss=3.2049 lr=1.88e-05 tok=23.588B dt=8029 wait=431 +14:26:50 step 179160 loss=3.2022 lr=1.88e-05 tok=23.590B dt=8093 wait=431 +14:27:54 step 179180 loss=3.0922 lr=1.88e-05 tok=23.593B dt=8157 wait=431 +14:28:57 step 179200 loss=3.1283 lr=1.87e-05 tok=23.596B dt=8220 wait=431 +14:30:01 step 179220 loss=3.1710 lr=1.87e-05 tok=23.598B dt=8284 wait=431 +14:31:05 step 179240 loss=3.0359 lr=1.87e-05 tok=23.601B dt=8348 wait=432 +14:31:48 pushed ckpt_w174.pt,log_w174.txt +14:33:19 waited 91s for global r=743 +14:33:27 global r=743 load 4.277 -> 4.281 +14:34:00 step 179260 loss=3.0976 lr=1.87e-05 tok=23.603B dt=8523 wait=432 +14:35:03 step 179280 loss=3.0712 lr=1.87e-05 tok=23.606B dt=8586 wait=432 +14:36:07 step 179300 loss=3.0051 lr=1.87e-05 tok=23.609B dt=8650 wait=432 +14:37:10 step 179320 loss=2.9371 lr=1.87e-05 tok=23.611B dt=8713 wait=432 +14:38:13 step 179340 loss=2.9027 lr=1.87e-05 tok=23.614B dt=8776 wait=432 +14:39:16 step 179360 loss=3.0172 lr=1.87e-05 tok=23.617B dt=8839 wait=432 +14:40:20 step 179380 loss=2.9317 lr=1.87e-05 tok=23.619B dt=8903 wait=433 +14:41:23 step 179400 loss=2.9395 lr=1.87e-05 tok=23.622B dt=8966 wait=433 +14:41:35 pushed ckpt_w174.pt,log_w174.txt +14:42:51 waited 76s for global r=744 +14:42:57 global r=744 load 4.282 -> 4.280 +14:44:01 step 179420 loss=2.8640 lr=1.87e-05 tok=23.624B dt=9124 wait=433 +14:45:04 step 179440 loss=2.9226 lr=1.87e-05 tok=23.627B dt=9187 wait=433 +14:46:07 step 179460 loss=2.9563 lr=1.87e-05 tok=23.630B dt=9250 wait=433 +14:47:10 step 179480 loss=2.8939 lr=1.86e-05 tok=23.632B dt=9313 wait=433 +14:48:14 step 179500 loss=2.8595 lr=1.86e-05 tok=23.635B dt=9377 wait=433 +14:49:17 step 179520 loss=2.9643 lr=1.86e-05 tok=23.638B dt=9439 wait=434 +14:50:21 step 179540 loss=2.5740 lr=1.86e-05 tok=23.640B dt=9504 wait=434 +14:51:04 pushed ckpt_w174.pt,log_w174.txt +14:52:34 waited 91s for global r=745 +14:52:40 global r=745 load 4.286 -> 4.281 +14:53:14 step 179560 loss=2.9870 lr=1.86e-05 tok=23.643B dt=9677 wait=434 +14:54:16 step 179580 loss=2.8703 lr=1.86e-05 tok=23.645B dt=9739 wait=434 +14:55:20 step 179600 loss=2.9313 lr=1.86e-05 tok=23.648B dt=9803 wait=434 +14:56:22 step 179620 loss=3.0685 lr=1.86e-05 tok=23.651B dt=9865 wait=434 +14:57:26 step 179640 loss=2.6751 lr=1.86e-05 tok=23.653B dt=9929 wait=434 +14:58:30 step 179660 loss=2.8777 lr=1.86e-05 tok=23.656B dt=9993 wait=435 +14:59:33 step 179680 loss=2.9130 lr=1.86e-05 tok=23.658B dt=10056 wait=435 +15:00:36 step 179700 loss=2.8918 lr=1.86e-05 tok=23.661B dt=10119 wait=435 +15:00:54 pushed ckpt_w174.pt,log_w174.txt +15:05:00 step 179720 loss=2.8988 lr=1.86e-05 tok=23.664B dt=10383 wait=435 +15:06:04 step 179740 loss=2.6715 lr=1.86e-05 tok=23.666B dt=10447 wait=435 +15:07:08 step 179760 loss=2.7764 lr=1.86e-05 tok=23.669B dt=10511 wait=435 +15:08:11 step 179780 loss=2.8236 lr=1.85e-05 tok=23.672B dt=10574 wait=436 +15:09:14 step 179800 loss=2.8551 lr=1.85e-05 tok=23.674B dt=10637 wait=436 +15:10:18 step 179820 loss=2.9611 lr=1.85e-05 tok=23.677B dt=10701 wait=436 +15:11:21 step 179840 loss=2.9444 lr=1.85e-05 tok=23.679B dt=10764 wait=436 +15:12:06 pushed ckpt_w174.pt,log_w174.txt +15:15:41 step 179860 loss=2.5025 lr=1.85e-05 tok=23.682B dt=11024 wait=436 +15:16:46 step 179880 loss=3.0325 lr=1.85e-05 tok=23.685B dt=11089 wait=436 +15:17:50 step 179900 loss=2.9825 lr=1.85e-05 tok=23.687B dt=11153 wait=437 +15:18:54 step 179920 loss=3.0315 lr=1.85e-05 tok=23.690B dt=11217 wait=437 +15:19:33 global r=746 load 4.278 -> 4.282 +15:20:05 step 179860 loss=2.9459 lr=1.85e-05 tok=23.693B dt=11288 wait=437 +15:21:09 step 179880 loss=2.8485 lr=1.85e-05 tok=23.695B dt=11352 wait=437 +15:22:14 step 179900 loss=2.3009 lr=1.85e-05 tok=23.698B dt=11417 wait=437 +15:23:18 step 179920 loss=2.9720 lr=1.85e-05 tok=23.700B dt=11481 wait=437 +15:24:22 step 179940 loss=2.8432 lr=1.85e-05 tok=23.703B dt=11545 wait=438 +15:25:27 step 179960 loss=2.7487 lr=1.85e-05 tok=23.706B dt=11610 wait=438 +15:26:31 step 179980 loss=2.9736 lr=1.85e-05 tok=23.708B dt=11674 wait=438 +15:27:35 step 180000 loss=2.9817 lr=1.85e-05 tok=23.711B dt=11738 wait=438 +15:27:46 pushed ckpt_w174.pt,log_w174.txt +15:31:27 global r=747 load 4.284 -> 4.283 +15:32:31 step 180020 loss=2.9088 lr=1.85e-05 tok=23.715B dt=12034 wait=438 +15:33:35 step 180040 loss=3.0253 lr=1.85e-05 tok=23.717B dt=12098 wait=438 +15:34:41 step 180060 loss=2.9807 lr=1.84e-05 tok=23.720B dt=12164 wait=438 +15:35:44 step 180080 loss=3.0575 lr=1.84e-05 tok=23.723B dt=12227 wait=439 +15:36:49 step 180100 loss=2.9979 lr=1.84e-05 tok=23.725B dt=12292 wait=439 +15:37:54 step 180120 loss=3.0965 lr=1.84e-05 tok=23.728B dt=12357 wait=439 +15:38:59 step 180140 loss=2.9437 lr=1.84e-05 tok=23.731B dt=12422 wait=439 +15:39:44 pushed ckpt_w174.pt,log_w174.txt +15:42:30 waited 166s for global r=748 +15:42:39 global r=748 load 4.281 -> 4.280 +15:43:13 step 180160 loss=3.0825 lr=1.84e-05 tok=23.733B dt=12676 wait=439 +15:44:17 step 180180 loss=2.9384 lr=1.84e-05 tok=23.736B dt=12740 wait=439 +15:45:21 step 180200 loss=3.0355 lr=1.84e-05 tok=23.738B dt=12804 wait=439 +15:46:25 step 180220 loss=2.9833 lr=1.84e-05 tok=23.741B dt=12868 wait=440 +15:47:29 step 180240 loss=2.9268 lr=1.84e-05 tok=23.744B dt=12932 wait=440 +15:48:33 step 180260 loss=2.9743 lr=1.84e-05 tok=23.746B dt=12996 wait=440 +15:49:37 step 180280 loss=2.9539 lr=1.84e-05 tok=23.749B dt=13060 wait=440 +15:50:42 step 180300 loss=2.9410 lr=1.84e-05 tok=23.752B dt=13125 wait=440 +15:50:54 pushed ckpt_w174.pt,log_w174.txt +15:54:40 global r=749 load 4.285 -> 4.282 +15:55:44 step 180320 loss=2.8884 lr=1.84e-05 tok=23.755B dt=13427 wait=440 +15:56:48 step 180340 loss=2.9250 lr=1.84e-05 tok=23.758B dt=13491 wait=441 +15:57:53 step 180360 loss=2.9238 lr=1.84e-05 tok=23.761B dt=13556 wait=441 +15:58:57 step 180380 loss=2.8528 lr=1.83e-05 tok=23.763B dt=13620 wait=441 +16:00:03 step 180400 loss=2.8536 lr=1.83e-05 tok=23.766B dt=13686 wait=441 +16:01:08 step 180420 loss=2.6523 lr=1.83e-05 tok=23.769B dt=13751 wait=441 +16:02:12 step 180440 loss=2.8773 lr=1.83e-05 tok=23.771B dt=13814 wait=441 +16:03:00 pushed ckpt_w174.pt,log_w174.txt +16:05:46 waited 167s for global r=750 +16:05:52 global r=750 load 4.286 -> 4.283 +16:06:26 step 180460 loss=3.0015 lr=1.83e-05 tok=23.774B dt=14069 wait=442 +16:07:30 step 180480 loss=2.8663 lr=1.83e-05 tok=23.776B dt=14133 wait=442 +16:08:34 step 180500 loss=2.9101 lr=1.83e-05 tok=23.779B dt=14197 wait=442 +16:09:39 step 180520 loss=2.5145 lr=1.83e-05 tok=23.782B dt=14262 wait=442 +16:10:43 step 180540 loss=2.9029 lr=1.83e-05 tok=23.784B dt=14326 wait=442 +16:11:47 step 180560 loss=3.0221 lr=1.83e-05 tok=23.787B dt=14390 wait=442 +16:12:52 step 180580 loss=2.9413 lr=1.83e-05 tok=23.790B dt=14455 wait=443 +16:13:57 step 180600 loss=2.9435 lr=1.83e-05 tok=23.792B dt=14520 wait=443 +16:14:18 pushed ckpt_w174.pt,log_w174.txt +16:15:49 waited 91s for global r=751 +16:15:56 global r=751 load 4.288 -> 4.282 +16:17:02 step 180620 loss=2.9464 lr=1.83e-05 tok=23.795B dt=14705 wait=443 +16:18:06 step 180640 loss=2.9517 lr=1.83e-05 tok=23.797B dt=14769 wait=443 +16:19:10 step 180660 loss=2.8753 lr=1.83e-05 tok=23.800B dt=14833 wait=443 +16:20:14 step 180680 loss=2.8783 lr=1.82e-05 tok=23.803B dt=14897 wait=443 +16:21:17 step 180700 loss=2.9181 lr=1.82e-05 tok=23.805B dt=14960 wait=444 +16:22:27 step 180720 loss=5.1691 lr=1.82e-05 tok=23.808B dt=15030 wait=444 +16:23:30 step 180740 loss=2.9159 lr=1.82e-05 tok=23.811B dt=15093 wait=444 +16:24:23 pushed ckpt_w174.pt,log_w174.txt +16:26:56 waited 153s for global r=752 +16:27:06 global r=752 load 4.286 -> 4.281 +16:27:39 step 180760 loss=2.9490 lr=1.82e-05 tok=23.813B dt=15342 wait=444 +16:28:43 step 180780 loss=2.8776 lr=1.82e-05 tok=23.816B dt=15406 wait=444 +16:29:47 step 180800 loss=2.9697 lr=1.82e-05 tok=23.818B dt=15470 wait=445 +16:30:51 step 180820 loss=2.8451 lr=1.82e-05 tok=23.821B dt=15534 wait=445 +16:31:55 step 180840 loss=2.9330 lr=1.82e-05 tok=23.824B dt=15598 wait=445 +16:32:59 step 180860 loss=3.0450 lr=1.82e-05 tok=23.826B dt=15662 wait=445 +16:34:03 step 180880 loss=2.9291 lr=1.82e-05 tok=23.829B dt=15726 wait=445 +16:35:06 step 180900 loss=2.9663 lr=1.82e-05 tok=23.832B dt=15789 wait=445 +16:35:21 pushed ckpt_w174.pt,log_w174.txt +16:39:07 global r=753 load 4.283 -> 4.284 +16:40:12 step 180920 loss=2.9988 lr=1.82e-05 tok=23.835B dt=16094 wait=446 +16:41:16 step 180940 loss=2.9455 lr=1.82e-05 tok=23.838B dt=16159 wait=446 +16:42:21 step 180960 loss=2.9552 lr=1.82e-05 tok=23.841B dt=16224 wait=446 +16:43:26 step 180980 loss=2.8874 lr=1.81e-05 tok=23.843B dt=16289 wait=446 +launch +16:43:51 name=w174 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 121/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:43:56 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:43:56 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:44:06 params 1.08B +16:44:07 resumed step=180900 seen=23.84B round=753 +16:44:07 stream salt=180900 +16:46:12 blocks compiled +16:46:12 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:51:14 step 180901 loss=2.8306 lr=1.76e-05 tok=23.838B dt=302 wait=291 +16:51:22 step 180902 loss=2.8629 lr=1.76e-05 tok=23.838B dt=311 wait=291 +16:52:02 global r=754 load 4.279 -> 4.281 +16:52:34 step 181060 loss=3.0105 lr=1.75e-05 tok=23.841B dt=383 wait=291 +16:53:39 step 181080 loss=2.8685 lr=1.75e-05 tok=23.843B dt=447 wait=292 +16:54:43 step 181100 loss=2.9419 lr=1.75e-05 tok=23.846B dt=512 wait=292 +16:55:48 step 181120 loss=3.0883 lr=1.74e-05 tok=23.849B dt=576 wait=292 +16:56:52 step 181140 loss=2.9830 lr=1.74e-05 tok=23.851B dt=641 wait=292 +16:57:57 step 181160 loss=3.0620 lr=1.74e-05 tok=23.854B dt=706 wait=292 +16:59:03 step 181180 loss=2.8848 lr=1.74e-05 tok=23.856B dt=771 wait=292 +17:00:07 step 181200 loss=2.9487 lr=1.74e-05 tok=23.859B dt=836 wait=293 +17:00:24 pushed ckpt_w174.pt,log_w174.txt +17:01:10 waited 45s for global r=755 +17:01:36 global r=755 load 4.283 -> 4.280 +17:02:43 step 181220 loss=2.8858 lr=1.73e-05 tok=23.862B dt=992 wait=293 +17:03:48 step 181240 loss=2.9540 lr=1.73e-05 tok=23.864B dt=1057 wait=293 +17:04:54 step 181260 loss=2.8661 lr=1.73e-05 tok=23.867B dt=1122 wait=293 +17:06:00 step 181280 loss=2.9651 lr=1.73e-05 tok=23.870B dt=1188 wait=293 +17:07:06 step 181300 loss=3.0504 lr=1.73e-05 tok=23.872B dt=1254 wait=293 +17:08:12 step 181320 loss=3.0998 lr=1.73e-05 tok=23.875B dt=1320 wait=293 +17:09:16 step 181340 loss=2.9847 lr=1.72e-05 tok=23.877B dt=1385 wait=294 +17:10:05 pushed ckpt_w174.pt,log_w174.txt +17:11:07 waited 61s for global r=756 +17:11:18 global r=756 load 4.287 -> 4.281 +17:11:53 step 181360 loss=3.0690 lr=1.72e-05 tok=23.880B dt=1542 wait=295 +17:12:59 step 181380 loss=2.9330 lr=1.72e-05 tok=23.883B dt=1607 wait=295 +17:14:04 step 181400 loss=3.0445 lr=1.72e-05 tok=23.885B dt=1672 wait=295 +17:15:10 step 181420 loss=3.0229 lr=1.72e-05 tok=23.888B dt=1738 wait=295 +17:16:14 step 181440 loss=3.0008 lr=1.71e-05 tok=23.890B dt=1802 wait=295 +17:17:20 step 181460 loss=2.9726 lr=1.71e-05 tok=23.893B dt=1868 wait=296 +17:18:25 step 181480 loss=3.0887 lr=1.71e-05 tok=23.896B dt=1933 wait=296 +17:19:30 step 181500 loss=3.0599 lr=1.71e-05 tok=23.898B dt=1999 wait=296 +17:19:57 pushed ckpt_w174.pt,log_w174.txt +17:22:28 waited 151s for global r=757 +17:22:41 global r=757 load 4.278 -> 4.282 +17:23:48 step 181520 loss=2.9441 lr=1.71e-05 tok=23.901B dt=2256 wait=296 +17:24:53 step 181540 loss=3.0853 lr=1.70e-05 tok=23.904B dt=2321 wait=296 +17:25:58 step 181560 loss=2.9741 lr=1.70e-05 tok=23.906B dt=2386 wait=296 +17:27:04 step 181580 loss=3.0118 lr=1.70e-05 tok=23.909B dt=2452 wait=297 +17:28:09 step 181600 loss=2.9606 lr=1.70e-05 tok=23.911B dt=2517 wait=297 +17:29:14 step 181620 loss=3.1119 lr=1.70e-05 tok=23.914B dt=2582 wait=297 +17:30:19 step 181640 loss=2.9839 lr=1.70e-05 tok=23.917B dt=2647 wait=297 +17:31:09 pushed ckpt_w174.pt,log_w174.txt +17:32:56 waited 106s for global r=758 +17:33:03 global r=758 load 4.279 -> 4.283 +17:33:37 step 181660 loss=3.0597 lr=1.69e-05 tok=23.919B dt=2846 wait=297 +17:34:43 step 181680 loss=2.9187 lr=1.69e-05 tok=23.922B dt=2911 wait=297 +17:35:48 step 181700 loss=2.9372 lr=1.69e-05 tok=23.925B dt=2976 wait=297 +17:36:53 step 181720 loss=3.0454 lr=1.69e-05 tok=23.927B dt=3042 wait=298 +17:37:58 step 181740 loss=2.8857 lr=1.69e-05 tok=23.930B dt=3106 wait=298 +17:39:05 step 181760 loss=2.9385 lr=1.68e-05 tok=23.932B dt=3173 wait=298 +17:40:10 step 181780 loss=2.9779 lr=1.68e-05 tok=23.935B dt=3238 wait=298 +17:41:16 step 181800 loss=3.0163 lr=1.68e-05 tok=23.938B dt=3304 wait=298 +17:41:33 pushed ckpt_w174.pt,log_w174.txt +17:42:34 waited 61s for global r=759 +17:43:02 global r=759 load 4.286 -> 4.283 +17:44:11 step 181820 loss=2.9882 lr=1.68e-05 tok=23.940B dt=3479 wait=298 +17:45:16 step 181840 loss=3.0158 lr=1.68e-05 tok=23.943B dt=3545 wait=298 +17:46:23 step 181860 loss=2.9497 lr=1.68e-05 tok=23.946B dt=3611 wait=299 +17:47:28 step 181880 loss=2.9078 lr=1.67e-05 tok=23.948B dt=3676 wait=299 +17:48:33 step 181900 loss=2.9150 lr=1.67e-05 tok=23.951B dt=3741 wait=299 +17:49:39 step 181920 loss=2.9813 lr=1.67e-05 tok=23.953B dt=3807 wait=299 +17:50:44 step 181940 loss=2.9757 lr=1.67e-05 tok=23.956B dt=3873 wait=299 +17:51:42 pushed ckpt_w174.pt,log_w174.txt +17:52:43 waited 61s for global r=760 +17:52:58 global r=760 load 4.282 -> 4.281 +17:53:33 step 181960 loss=2.9501 lr=1.67e-05 tok=23.959B dt=4041 wait=299 +17:54:38 step 181980 loss=2.9216 lr=1.66e-05 tok=23.961B dt=4106 wait=300 +17:55:44 step 182000 loss=2.8531 lr=1.66e-05 tok=23.964B dt=4172 wait=300 +17:56:49 step 182020 loss=3.0390 lr=1.66e-05 tok=23.967B dt=4237 wait=300 +17:57:54 step 182040 loss=2.9265 lr=1.66e-05 tok=23.969B dt=4302 wait=300 +17:58:58 step 182060 loss=3.0622 lr=1.66e-05 tok=23.972B dt=4367 wait=300 +18:00:03 step 182080 loss=3.0451 lr=1.65e-05 tok=23.974B dt=4432 wait=300 +18:01:09 step 182100 loss=3.1565 lr=1.65e-05 tok=23.977B dt=4497 wait=300 +18:01:22 pushed ckpt_w174.pt,log_w174.txt +18:02:53 waited 91s for global r=761 +18:03:07 global r=761 load 4.282 -> 4.281 +18:04:12 step 182120 loss=2.9533 lr=1.65e-05 tok=23.980B dt=4680 wait=301 +18:05:17 step 182140 loss=3.0528 lr=1.65e-05 tok=23.982B dt=4745 wait=301 +18:06:23 step 182160 loss=2.8937 lr=1.65e-05 tok=23.985B dt=4812 wait=301 +18:07:29 step 182180 loss=3.0417 lr=1.65e-05 tok=23.987B dt=4878 wait=301 +18:08:34 step 182200 loss=2.9335 lr=1.64e-05 tok=23.990B dt=4942 wait=301 +18:09:39 step 182220 loss=3.0274 lr=1.64e-05 tok=23.993B dt=5007 wait=301 +18:10:44 step 182240 loss=2.9909 lr=1.64e-05 tok=23.995B dt=5072 wait=302 diff --git a/log_w175.txt b/log_w175.txt new file mode 100644 index 0000000000000000000000000000000000000000..d1ec902ad5031876d9ea06dc090b0a914d063ad4 --- /dev/null +++ b/log_w175.txt @@ -0,0 +1,203 @@ +launch +12:06:57 name=w175 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 122/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:07:03 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:07:03 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:07:12 params 1.08B +12:07:14 resumed step=177150 seen=23.33B round=729 +12:07:14 stream salt=177150 +12:09:28 blocks compiled +12:09:28 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:15:13 step 177151 loss=2.8000 lr=1.95e-05 tok=23.332B dt=345 wait=316 +12:15:29 step 177152 loss=2.9330 lr=1.95e-05 tok=23.332B dt=361 wait=316 +12:15:55 step 177160 loss=3.3022 lr=1.95e-05 tok=23.333B dt=387 wait=316 +12:16:10 dolma file skipped data/ingredient1-common_crawl-high-quality_19_industrial/shard_00000194.jsonl.zst: (Request ID: Root=1-6abcfd8a-29df179d62b54c9b7936e0dd;ef882193-6c4c-4548-9755-a5888a33eaa8) + +429 Too Many Requests: you +12:16:11 dolma file skipped data/ingredient1-olmocr_science_pdfs-high_quality-education_jobs-2e13/education_jobs_p040_shard_00002307.jsonl.zst: (Request ID: Root=1-6abcfd8b-4ec3cb687d3a627714b3cb98;21b210af-c2c9-4711-91df-a1b2b3781f8f) + +429 Too Many Requests: you +12:16:12 dolma file skipped data/ingredient1-common_crawl-high-quality_20_science_math_and_technology/shard_00000610.jsonl.zst: (Request ID: Root=1-6abcfd8c-243ffbae249b6e94395c7145;04f4b909-d76c-4560-b59c-f1d62873e515) + +429 Too Many Requests: you +12:16:13 dolma file skipped data/ingredient1-common_crawl-high-quality_19_politics/shard_00000440.jsonl.zst: (Request ID: Root=1-6abcfd8d-170f216f39958a7d623facb4;cc9f1c13-ed3e-4868-989c-2dca699209a2) + +429 Too Many Requests: you +12:16:16 dolma file skipped data/ingredient1-reddit_to_flashcards/merged_qa_prefilter_densesubs_lowthresh-0137_f1.jsonl.zst: (Request ID: Root=1-6abcfd8f-14c8b78314e7252205b391ce;1135eead-b9de-4fab-bb8b-171fa9fd52df) + +429 Too Many Requests: you +12:16:18 dolma file skipped data/ingredient1-common_crawl-high-quality_20_politics/shard_00000204.jsonl.zst: (Request ID: Root=1-6abcfd92-3cf3884f31b5f6de3bde6d6d;2b4ede80-4fac-43ed-bacc-9d51458c23b3) + +429 Too Many Requests: you +12:16:19 dolma file skipped data/ingredient1-common_crawl-high-quality_19_software/shard_00000392.jsonl.zst: (Request ID: Root=1-6abcfd93-046a08280de56ca83e5b7cfe;d88c1eed-5bd8-4307-a19e-39df59d5b91c) + +429 Too Many Requests: you +12:16:21 dolma file skipped data/ingredient1-common_crawl-high-quality_20_literature/shard_00000430.jsonl.zst: (Request ID: Root=1-6abcfd95-047e1a2e29cc520320120dab;49173e3e-1f33-4857-af77-060630f68819) + +429 Too Many Requests: you +12:16:23 dolma file skipped data/ingredient1-common_crawl-high-quality_20_home_and_hobbies/shard_00000942.jsonl.zst: (Request ID: Root=1-6abcfd97-5076d020182783fc6f3004bc;7d6c589d-94d1-48c4-a5d8-cb6c59a77aea) + +429 Too Many Requests: you +12:17:01 step 177180 loss=3.1273 lr=1.95e-05 tok=23.336B dt=453 wait=316 +12:18:07 step 177200 loss=3.1287 lr=1.95e-05 tok=23.339B dt=519 wait=316 +12:19:11 step 177220 loss=3.1888 lr=1.95e-05 tok=23.341B dt=583 wait=316 +12:20:15 step 177240 loss=3.2670 lr=1.95e-05 tok=23.344B dt=647 wait=316 +12:21:21 step 177260 loss=3.3608 lr=1.94e-05 tok=23.347B dt=713 wait=317 +12:22:26 step 177280 loss=3.1568 lr=1.94e-05 tok=23.349B dt=778 wait=317 +12:23:30 step 177300 loss=3.2066 lr=1.94e-05 tok=23.352B dt=842 wait=317 +12:23:45 pushed ckpt_w175.pt,log_w175.txt +12:25:01 waited 76s for global r=730 +12:25:21 global r=730 load 4.271 -> 4.280 +12:26:29 step 177320 loss=3.1948 lr=1.94e-05 tok=23.354B dt=1021 wait=317 +12:27:34 step 177340 loss=3.0755 lr=1.94e-05 tok=23.357B dt=1086 wait=317 +12:28:38 step 177360 loss=3.1725 lr=1.94e-05 tok=23.360B dt=1150 wait=317 +12:29:44 step 177380 loss=3.1767 lr=1.94e-05 tok=23.362B dt=1216 wait=317 +12:30:48 step 177400 loss=3.1583 lr=1.94e-05 tok=23.365B dt=1281 wait=318 +12:31:54 step 177420 loss=3.1550 lr=1.94e-05 tok=23.368B dt=1346 wait=318 +12:32:58 step 177440 loss=3.0728 lr=1.94e-05 tok=23.370B dt=1410 wait=318 +12:33:44 pushed ckpt_w175.pt,log_w175.txt +12:37:20 step 177460 loss=3.2147 lr=1.94e-05 tok=23.373B dt=1672 wait=318 +12:37:26 global r=731 load 4.278 -> 4.278 +12:37:58 step 177460 loss=3.1878 lr=1.94e-05 tok=23.374B dt=1710 wait=318 +12:39:03 step 177480 loss=2.9515 lr=1.94e-05 tok=23.377B dt=1775 wait=318 +12:40:07 step 177500 loss=3.1340 lr=1.94e-05 tok=23.379B dt=1839 wait=318 +12:41:11 step 177520 loss=3.2264 lr=1.94e-05 tok=23.382B dt=1904 wait=319 +12:42:15 step 177540 loss=3.1055 lr=1.93e-05 tok=23.385B dt=1967 wait=319 +12:43:19 step 177560 loss=3.0916 lr=1.93e-05 tok=23.387B dt=2031 wait=319 +12:44:23 step 177580 loss=3.0477 lr=1.93e-05 tok=23.390B dt=2095 wait=319 +12:45:26 step 177600 loss=3.1282 lr=1.93e-05 tok=23.392B dt=2158 wait=319 +12:45:39 pushed ckpt_w175.pt,log_w175.txt +12:46:40 waited 61s for global r=732 +12:46:48 global r=732 load 4.273 -> 4.280 +12:47:53 step 177620 loss=3.0814 lr=1.93e-05 tok=23.395B dt=2305 wait=319 +12:48:58 step 177640 loss=3.0085 lr=1.93e-05 tok=23.398B dt=2370 wait=319 +12:50:02 step 177660 loss=3.0204 lr=1.93e-05 tok=23.400B dt=2434 wait=320 +12:51:06 step 177680 loss=2.9348 lr=1.93e-05 tok=23.403B dt=2498 wait=320 +12:52:09 step 177700 loss=3.0192 lr=1.93e-05 tok=23.406B dt=2561 wait=320 +12:53:13 step 177720 loss=3.0346 lr=1.93e-05 tok=23.408B dt=2625 wait=320 +12:54:18 step 177740 loss=2.6622 lr=1.93e-05 tok=23.411B dt=2690 wait=320 +12:55:01 pushed ckpt_w175.pt,log_w175.txt +12:56:17 waited 76s for global r=733 +12:56:25 global r=733 load 4.283 -> 4.282 +12:56:58 step 177760 loss=3.0454 lr=1.93e-05 tok=23.413B dt=2850 wait=320 +12:58:03 step 177780 loss=2.8556 lr=1.93e-05 tok=23.416B dt=2915 wait=320 +12:59:06 step 177800 loss=2.9768 lr=1.92e-05 tok=23.419B dt=2978 wait=321 +13:00:10 step 177820 loss=3.1450 lr=1.92e-05 tok=23.421B dt=3042 wait=321 +13:01:13 step 177840 loss=2.9590 lr=1.92e-05 tok=23.424B dt=3105 wait=321 +13:02:17 step 177860 loss=3.0384 lr=1.92e-05 tok=23.426B dt=3169 wait=321 +13:03:21 step 177880 loss=3.0795 lr=1.92e-05 tok=23.429B dt=3233 wait=321 +13:04:24 step 177900 loss=2.9588 lr=1.92e-05 tok=23.432B dt=3296 wait=321 +13:04:36 pushed ckpt_w175.pt,log_w175.txt +13:06:08 waited 92s for global r=734 +13:06:13 global r=734 load 4.283 -> 4.283 +13:07:18 step 177920 loss=2.9758 lr=1.92e-05 tok=23.434B dt=3471 wait=321 +13:08:22 step 177940 loss=3.0335 lr=1.92e-05 tok=23.437B dt=3534 wait=322 +13:09:26 step 177960 loss=3.0269 lr=1.92e-05 tok=23.440B dt=3598 wait=322 +13:10:29 step 177980 loss=3.0194 lr=1.92e-05 tok=23.442B dt=3661 wait=322 +13:11:32 step 178000 loss=3.0721 lr=1.92e-05 tok=23.445B dt=3724 wait=322 +13:12:37 step 178020 loss=3.0975 lr=1.92e-05 tok=23.447B dt=3789 wait=322 +13:13:41 step 178040 loss=2.8797 lr=1.92e-05 tok=23.450B dt=3853 wait=322 +13:14:25 pushed ckpt_w175.pt,log_w175.txt +13:15:41 waited 76s for global r=735 +13:15:47 global r=735 load 4.282 -> 4.283 +13:16:20 step 178060 loss=3.0483 lr=1.92e-05 tok=23.453B dt=4012 wait=323 +13:17:24 step 178080 loss=2.9959 lr=1.91e-05 tok=23.455B dt=4076 wait=323 +13:18:30 step 178100 loss=2.9356 lr=1.91e-05 tok=23.458B dt=4142 wait=323 +13:19:34 step 178120 loss=3.0688 lr=1.91e-05 tok=23.461B dt=4206 wait=323 +13:20:39 step 178140 loss=2.9895 lr=1.91e-05 tok=23.463B dt=4271 wait=323 +13:21:43 step 178160 loss=3.0905 lr=1.91e-05 tok=23.466B dt=4335 wait=323 +13:22:48 step 178180 loss=3.0326 lr=1.91e-05 tok=23.468B dt=4400 wait=323 +13:23:53 step 178200 loss=3.0409 lr=1.91e-05 tok=23.471B dt=4465 wait=324 +13:24:06 pushed ckpt_w175.pt,log_w175.txt +13:25:22 waited 76s for global r=736 +13:25:28 global r=736 load 4.285 -> 4.283 +13:26:33 step 178220 loss=3.0659 lr=1.91e-05 tok=23.474B dt=4625 wait=324 +13:27:38 step 178240 loss=2.4707 lr=1.91e-05 tok=23.476B dt=4690 wait=324 +13:28:43 step 178260 loss=2.9891 lr=1.91e-05 tok=23.479B dt=4755 wait=324 +13:29:48 step 178280 loss=2.9515 lr=1.91e-05 tok=23.482B dt=4821 wait=324 +13:30:53 step 178300 loss=2.9736 lr=1.91e-05 tok=23.484B dt=4885 wait=324 +13:31:58 step 178320 loss=2.9971 lr=1.91e-05 tok=23.487B dt=4950 wait=324 +13:33:03 step 178340 loss=3.0609 lr=1.90e-05 tok=23.489B dt=5015 wait=325 +13:33:48 pushed ckpt_w175.pt,log_w175.txt +13:35:04 waited 76s for global r=737 +13:35:13 global r=737 load 4.290 -> 4.282 +13:35:46 step 178360 loss=2.9709 lr=1.90e-05 tok=23.492B dt=5178 wait=325 +13:36:50 step 178380 loss=2.9716 lr=1.90e-05 tok=23.495B dt=5242 wait=325 +13:37:55 step 178400 loss=2.6963 lr=1.90e-05 tok=23.497B dt=5307 wait=325 +13:39:00 step 178420 loss=3.0777 lr=1.90e-05 tok=23.500B dt=5372 wait=325 +13:40:04 step 178440 loss=2.9771 lr=1.90e-05 tok=23.503B dt=5437 wait=325 +13:41:09 step 178460 loss=3.0905 lr=1.90e-05 tok=23.505B dt=5502 wait=326 +13:42:13 step 178480 loss=2.9382 lr=1.90e-05 tok=23.508B dt=5566 wait=326 +13:43:18 step 178500 loss=2.9690 lr=1.90e-05 tok=23.510B dt=5630 wait=326 +13:43:32 pushed ckpt_w175.pt,log_w175.txt +13:44:48 waited 76s for global r=738 +13:44:55 global r=738 load 4.290 -> 4.284 +13:46:01 step 178520 loss=2.8869 lr=1.90e-05 tok=23.513B dt=5793 wait=326 +13:47:06 step 178540 loss=2.9791 lr=1.90e-05 tok=23.516B dt=5858 wait=326 +13:48:10 step 178560 loss=2.9037 lr=1.90e-05 tok=23.518B dt=5922 wait=326 +13:49:15 step 178580 loss=2.9188 lr=1.90e-05 tok=23.521B dt=5987 wait=326 +13:50:19 step 178600 loss=2.8977 lr=1.90e-05 tok=23.523B dt=6051 wait=327 +13:51:23 step 178620 loss=2.9616 lr=1.89e-05 tok=23.526B dt=6115 wait=327 +13:52:28 step 178640 loss=2.8963 lr=1.89e-05 tok=23.529B dt=6180 wait=327 +13:53:12 pushed ckpt_w175.pt,log_w175.txt +13:54:27 waited 76s for global r=739 +13:54:33 global r=739 load 4.286 -> 4.282 +13:55:06 step 178660 loss=2.9799 lr=1.89e-05 tok=23.531B dt=6338 wait=327 +13:56:11 step 178680 loss=3.0015 lr=1.89e-05 tok=23.534B dt=6403 wait=327 +13:57:14 step 178700 loss=2.9648 lr=1.89e-05 tok=23.537B dt=6466 wait=327 +13:58:18 step 178720 loss=3.0279 lr=1.89e-05 tok=23.539B dt=6530 wait=327 +13:59:23 step 178740 loss=2.9580 lr=1.89e-05 tok=23.542B dt=6595 wait=328 +14:00:26 step 178760 loss=3.0498 lr=1.89e-05 tok=23.544B dt=6658 wait=328 +14:01:31 step 178780 loss=2.9709 lr=1.89e-05 tok=23.547B dt=6723 wait=328 +14:02:34 step 178800 loss=3.0475 lr=1.89e-05 tok=23.550B dt=6786 wait=328 +14:02:49 pushed ckpt_w175.pt,log_w175.txt +14:04:19 waited 91s for global r=740 +14:04:25 global r=740 load 4.278 -> 4.279 +14:05:31 step 178820 loss=3.0514 lr=1.89e-05 tok=23.552B dt=6963 wait=328 +14:06:34 step 178840 loss=3.0078 lr=1.89e-05 tok=23.555B dt=7026 wait=328 +14:07:40 step 178860 loss=2.9046 lr=1.89e-05 tok=23.558B dt=7092 wait=328 +14:08:45 step 178880 loss=2.9147 lr=1.89e-05 tok=23.560B dt=7157 wait=329 +14:09:49 step 178900 loss=3.0661 lr=1.88e-05 tok=23.563B dt=7221 wait=329 +14:10:54 step 178920 loss=3.0968 lr=1.88e-05 tok=23.565B dt=7286 wait=329 +14:11:58 step 178940 loss=3.0158 lr=1.88e-05 tok=23.568B dt=7351 wait=329 +14:12:44 pushed ckpt_w175.pt,log_w175.txt +14:13:59 waited 76s for global r=741 +14:14:06 global r=741 load 4.281 -> 4.280 +14:14:40 step 178960 loss=3.1075 lr=1.88e-05 tok=23.571B dt=7512 wait=329 +14:15:45 step 178980 loss=3.1293 lr=1.88e-05 tok=23.573B dt=7577 wait=329 +14:16:48 step 179000 loss=3.0716 lr=1.88e-05 tok=23.576B dt=7641 wait=330 +14:17:54 step 179020 loss=2.7580 lr=1.88e-05 tok=23.579B dt=7706 wait=330 +14:18:59 step 179040 loss=3.0547 lr=1.88e-05 tok=23.581B dt=7771 wait=330 +14:20:04 step 179060 loss=3.1566 lr=1.88e-05 tok=23.584B dt=7836 wait=330 +14:21:09 step 179080 loss=2.6438 lr=1.88e-05 tok=23.586B dt=7901 wait=330 +14:22:14 step 179100 loss=3.0326 lr=1.88e-05 tok=23.589B dt=7966 wait=330 +14:22:28 pushed ckpt_w175.pt,log_w175.txt +14:23:28 waited 61s for global r=742 +14:23:36 global r=742 load 4.294 -> 4.282 +14:24:41 step 179120 loss=3.0053 lr=1.88e-05 tok=23.592B dt=8113 wait=330 +14:25:45 step 179140 loss=3.0527 lr=1.88e-05 tok=23.594B dt=8177 wait=331 +14:26:49 step 179160 loss=3.0177 lr=1.88e-05 tok=23.597B dt=8241 wait=331 +14:27:53 step 179180 loss=2.9091 lr=1.88e-05 tok=23.600B dt=8305 wait=331 +14:28:57 step 179200 loss=2.9835 lr=1.87e-05 tok=23.602B dt=8369 wait=331 +14:30:01 step 179220 loss=3.0480 lr=1.87e-05 tok=23.605B dt=8433 wait=331 +14:31:05 step 179240 loss=2.8461 lr=1.87e-05 tok=23.607B dt=8497 wait=331 +14:31:49 pushed ckpt_w175.pt,log_w175.txt +14:33:20 waited 91s for global r=743 +14:33:28 global r=743 load 4.273 -> 4.281 +14:34:01 step 179260 loss=3.1038 lr=1.87e-05 tok=23.610B dt=8673 wait=331 +14:35:05 step 179280 loss=2.9051 lr=1.87e-05 tok=23.613B dt=8737 wait=332 +14:36:08 step 179300 loss=2.8809 lr=1.87e-05 tok=23.615B dt=8801 wait=332 +14:37:13 step 179320 loss=3.0194 lr=1.87e-05 tok=23.618B dt=8865 wait=332 +14:38:18 step 179340 loss=2.8607 lr=1.87e-05 tok=23.620B dt=8930 wait=332 +14:39:22 step 179360 loss=3.0936 lr=1.87e-05 tok=23.623B dt=8994 wait=332 +14:40:27 step 179380 loss=2.9955 lr=1.87e-05 tok=23.626B dt=9059 wait=332 +14:41:31 step 179400 loss=3.1012 lr=1.87e-05 tok=23.628B dt=9123 wait=332 +14:41:42 pushed ckpt_w175.pt,log_w175.txt +14:42:58 waited 76s for global r=744 +14:43:05 global r=744 load 4.285 -> 4.280 +14:44:11 step 179420 loss=3.1163 lr=1.87e-05 tok=23.631B dt=9283 wait=333 +14:45:16 step 179440 loss=3.3602 lr=1.87e-05 tok=23.634B dt=9348 wait=333 +14:46:20 step 179460 loss=3.1652 lr=1.87e-05 tok=23.636B dt=9412 wait=333 +14:47:25 step 179480 loss=3.3565 lr=1.86e-05 tok=23.639B dt=9477 wait=333 +14:48:29 step 179500 loss=3.2400 lr=1.86e-05 tok=23.641B dt=9541 wait=333 +14:49:33 step 179520 loss=3.1389 lr=1.86e-05 tok=23.644B dt=9605 wait=333 +14:50:37 step 179540 loss=3.0065 lr=1.86e-05 tok=23.647B dt=9669 wait=333 diff --git a/log_w176.txt b/log_w176.txt new file mode 100644 index 0000000000000000000000000000000000000000..7c4874083d51bc54e3c050e4cc205d4184f3d863 --- /dev/null +++ b/log_w176.txt @@ -0,0 +1,376 @@ +launch +12:07:14 name=w176 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 123/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:07:20 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:07:20 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:07:30 params 1.08B +12:07:31 resumed step=177150 seen=23.33B round=729 +12:07:31 stream salt=177150 +12:09:50 blocks compiled +12:09:50 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:14:52 step 177151 loss=3.0937 lr=1.95e-05 tok=23.332B dt=302 wait=276 +12:15:09 step 177152 loss=3.3372 lr=1.95e-05 tok=23.332B dt=319 wait=276 +12:15:35 step 177160 loss=3.5292 lr=1.95e-05 tok=23.333B dt=345 wait=276 +12:16:20 dolma file skipped data/ingredient1-common_crawl-high-quality_20_transportation/shard_00000072.jsonl.zst: (Request ID: Root=1-6abcfd94-7ce6aaa7351d5c4b107053fa;42f80321-62ae-41af-99c8-8b4d926a9cd2) + +429 Too Many Requests: you +12:16:20 dolma file skipped data/ingredient1-reddit_to_flashcards/merged_qa_prefilter_densesubs_lowthresh-0104_f0.jsonl.zst: (Request ID: Root=1-6abcfd94-282e7b9e29bee1220e75167e;99f3ee1d-2200-433f-96be-0999102612c1) + +429 Too Many Requests: you +12:16:40 step 177180 loss=3.2386 lr=1.95e-05 tok=23.336B dt=410 wait=276 +12:17:45 step 177200 loss=3.2145 lr=1.95e-05 tok=23.339B dt=475 wait=276 +12:18:49 step 177220 loss=3.1228 lr=1.95e-05 tok=23.341B dt=539 wait=276 +12:19:53 step 177240 loss=3.2844 lr=1.95e-05 tok=23.344B dt=603 wait=276 +12:20:57 step 177260 loss=3.3385 lr=1.94e-05 tok=23.347B dt=666 wait=276 +12:22:00 step 177280 loss=3.0624 lr=1.94e-05 tok=23.349B dt=730 wait=277 +12:23:04 step 177300 loss=3.1558 lr=1.94e-05 tok=23.352B dt=794 wait=277 +12:23:18 pushed ckpt_w176.pt,log_w176.txt +12:25:04 waited 106s for global r=730 +12:25:24 global r=730 load 4.285 -> 4.280 +12:26:31 step 177320 loss=3.0353 lr=1.94e-05 tok=23.354B dt=1000 wait=277 +12:27:34 step 177340 loss=3.0277 lr=1.94e-05 tok=23.357B dt=1064 wait=277 +12:28:38 step 177360 loss=2.9957 lr=1.94e-05 tok=23.360B dt=1127 wait=277 +12:29:42 step 177380 loss=3.0041 lr=1.94e-05 tok=23.362B dt=1191 wait=277 +12:30:45 step 177400 loss=3.0011 lr=1.94e-05 tok=23.365B dt=1254 wait=277 +12:31:49 step 177420 loss=3.1641 lr=1.94e-05 tok=23.368B dt=1318 wait=278 +12:32:52 step 177440 loss=3.0648 lr=1.94e-05 tok=23.370B dt=1382 wait=278 +12:33:36 pushed ckpt_w176.pt,log_w176.txt +12:37:12 step 177460 loss=3.1273 lr=1.94e-05 tok=23.373B dt=1641 wait=278 +12:37:19 global r=731 load 4.280 -> 4.278 +12:37:51 step 177460 loss=3.1719 lr=1.94e-05 tok=23.374B dt=1680 wait=278 +12:38:54 step 177480 loss=2.9498 lr=1.94e-05 tok=23.377B dt=1743 wait=278 +12:39:58 step 177500 loss=3.0506 lr=1.94e-05 tok=23.379B dt=1807 wait=278 +12:41:02 step 177520 loss=3.2018 lr=1.94e-05 tok=23.382B dt=1871 wait=278 +12:42:04 step 177540 loss=3.1118 lr=1.93e-05 tok=23.385B dt=1934 wait=278 +12:43:09 step 177560 loss=3.0759 lr=1.93e-05 tok=23.387B dt=1998 wait=279 +12:44:12 step 177580 loss=3.0348 lr=1.93e-05 tok=23.390B dt=2062 wait=279 +12:45:16 step 177600 loss=3.0886 lr=1.93e-05 tok=23.392B dt=2126 wait=279 +12:45:28 pushed ckpt_w176.pt,log_w176.txt +12:46:43 waited 76s for global r=732 +12:46:51 global r=732 load 4.280 -> 4.280 +12:47:57 step 177620 loss=3.0619 lr=1.93e-05 tok=23.395B dt=2286 wait=279 +12:49:01 step 177640 loss=3.0602 lr=1.93e-05 tok=23.398B dt=2351 wait=279 +12:50:05 step 177660 loss=2.9037 lr=1.93e-05 tok=23.400B dt=2415 wait=279 +12:51:09 step 177680 loss=2.9834 lr=1.93e-05 tok=23.403B dt=2479 wait=279 +12:52:13 step 177700 loss=2.9646 lr=1.93e-05 tok=23.406B dt=2543 wait=279 +12:53:17 step 177720 loss=3.1087 lr=1.93e-05 tok=23.408B dt=2607 wait=280 +12:54:21 step 177740 loss=2.9094 lr=1.93e-05 tok=23.411B dt=2670 wait=280 +12:55:05 pushed ckpt_w176.pt,log_w176.txt +12:56:21 waited 76s for global r=733 +12:56:29 global r=733 load 4.283 -> 4.282 +12:57:02 step 177760 loss=3.1254 lr=1.93e-05 tok=23.413B dt=2831 wait=280 +12:58:06 step 177780 loss=2.7983 lr=1.93e-05 tok=23.416B dt=2896 wait=280 +12:59:09 step 177800 loss=2.9626 lr=1.92e-05 tok=23.419B dt=2959 wait=280 +13:00:13 step 177820 loss=3.0526 lr=1.92e-05 tok=23.421B dt=3023 wait=280 +13:01:17 step 177840 loss=2.9206 lr=1.92e-05 tok=23.424B dt=3086 wait=280 +13:02:21 step 177860 loss=2.9378 lr=1.92e-05 tok=23.426B dt=3150 wait=281 +13:03:24 step 177880 loss=2.9622 lr=1.92e-05 tok=23.429B dt=3214 wait=281 +13:04:28 step 177900 loss=2.9727 lr=1.92e-05 tok=23.432B dt=3278 wait=281 +13:04:40 pushed ckpt_w176.pt,log_w176.txt +13:05:55 waited 76s for global r=734 +13:06:02 global r=734 load 4.290 -> 4.283 +13:07:07 step 177920 loss=2.9706 lr=1.92e-05 tok=23.434B dt=3436 wait=281 +13:08:11 step 177940 loss=3.0259 lr=1.92e-05 tok=23.437B dt=3501 wait=281 +13:09:16 step 177960 loss=2.9736 lr=1.92e-05 tok=23.440B dt=3565 wait=281 +13:10:20 step 177980 loss=2.9694 lr=1.92e-05 tok=23.442B dt=3630 wait=281 +13:11:25 step 178000 loss=3.0545 lr=1.92e-05 tok=23.445B dt=3694 wait=281 +13:12:29 step 178020 loss=3.0700 lr=1.92e-05 tok=23.447B dt=3759 wait=282 +13:13:34 step 178040 loss=3.0098 lr=1.92e-05 tok=23.450B dt=3824 wait=282 +13:14:22 pushed ckpt_w176.pt,log_w176.txt +13:15:38 waited 76s for global r=735 +13:15:46 global r=735 load 4.284 -> 4.283 +13:16:19 step 178060 loss=2.9806 lr=1.92e-05 tok=23.453B dt=3989 wait=282 +13:17:23 step 178080 loss=3.0077 lr=1.91e-05 tok=23.455B dt=4052 wait=282 +13:18:27 step 178100 loss=2.8616 lr=1.91e-05 tok=23.458B dt=4117 wait=282 +13:19:31 step 178120 loss=2.9605 lr=1.91e-05 tok=23.461B dt=4181 wait=282 +13:20:35 step 178140 loss=2.8953 lr=1.91e-05 tok=23.463B dt=4245 wait=282 +13:21:40 step 178160 loss=2.9476 lr=1.91e-05 tok=23.466B dt=4309 wait=283 +13:22:44 step 178180 loss=2.9395 lr=1.91e-05 tok=23.468B dt=4373 wait=283 +13:23:48 step 178200 loss=2.9603 lr=1.91e-05 tok=23.471B dt=4438 wait=283 +13:24:05 pushed ckpt_w176.pt,log_w176.txt +13:25:21 waited 76s for global r=736 +13:25:30 global r=736 load 4.285 -> 4.283 +13:26:35 step 178220 loss=2.9820 lr=1.91e-05 tok=23.474B dt=4605 wait=283 +13:27:39 step 178240 loss=2.9749 lr=1.91e-05 tok=23.476B dt=4668 wait=283 +13:28:43 step 178260 loss=2.9434 lr=1.91e-05 tok=23.479B dt=4733 wait=283 +13:29:47 step 178280 loss=2.9157 lr=1.91e-05 tok=23.482B dt=4797 wait=283 +13:30:52 step 178300 loss=3.1867 lr=1.91e-05 tok=23.484B dt=4862 wait=284 +13:31:57 step 178320 loss=3.2514 lr=1.91e-05 tok=23.487B dt=4927 wait=284 +13:33:02 step 178340 loss=3.1851 lr=1.90e-05 tok=23.489B dt=4991 wait=284 +13:33:46 pushed ckpt_w176.pt,log_w176.txt +13:35:02 waited 76s for global r=737 +13:35:07 global r=737 load 4.282 -> 4.282 +13:35:41 step 178360 loss=3.3293 lr=1.90e-05 tok=23.492B dt=5151 wait=284 +13:36:45 step 178380 loss=3.1658 lr=1.90e-05 tok=23.495B dt=5215 wait=284 +13:37:50 step 178400 loss=3.0479 lr=1.90e-05 tok=23.497B dt=5280 wait=284 +13:38:56 step 178420 loss=3.2645 lr=1.90e-05 tok=23.500B dt=5345 wait=284 +13:40:00 step 178440 loss=3.1221 lr=1.90e-05 tok=23.503B dt=5410 wait=285 +13:41:05 step 178460 loss=3.0776 lr=1.90e-05 tok=23.505B dt=5474 wait=285 +13:42:08 step 178480 loss=3.0732 lr=1.90e-05 tok=23.508B dt=5538 wait=285 +13:43:13 step 178500 loss=3.0251 lr=1.90e-05 tok=23.510B dt=5603 wait=285 +13:43:27 pushed ckpt_w176.pt,log_w176.txt +13:44:42 waited 76s for global r=738 +13:44:49 global r=738 load 4.283 -> 4.284 +13:45:55 step 178520 loss=2.9816 lr=1.90e-05 tok=23.513B dt=5765 wait=285 +13:47:00 step 178540 loss=3.0090 lr=1.90e-05 tok=23.516B dt=5829 wait=285 +13:48:04 step 178560 loss=3.0248 lr=1.90e-05 tok=23.518B dt=5894 wait=286 +13:49:08 step 178580 loss=2.9718 lr=1.90e-05 tok=23.521B dt=5958 wait=286 +13:50:12 step 178600 loss=2.9517 lr=1.90e-05 tok=23.523B dt=6022 wait=286 +13:51:16 step 178620 loss=3.0384 lr=1.89e-05 tok=23.526B dt=6085 wait=286 +13:52:19 step 178640 loss=2.9016 lr=1.89e-05 tok=23.529B dt=6148 wait=286 +13:53:04 pushed ckpt_w176.pt,log_w176.txt +13:54:20 waited 76s for global r=739 +13:54:27 global r=739 load 4.290 -> 4.282 +13:55:00 step 178660 loss=3.0103 lr=1.89e-05 tok=23.531B dt=6309 wait=286 +13:56:03 step 178680 loss=2.7002 lr=1.89e-05 tok=23.534B dt=6373 wait=286 +13:57:07 step 178700 loss=2.8769 lr=1.89e-05 tok=23.537B dt=6437 wait=286 +13:58:11 step 178720 loss=2.9799 lr=1.89e-05 tok=23.539B dt=6501 wait=287 +13:59:15 step 178740 loss=2.8125 lr=1.89e-05 tok=23.542B dt=6565 wait=287 +14:00:19 step 178760 loss=3.0055 lr=1.89e-05 tok=23.544B dt=6629 wait=287 +14:01:23 step 178780 loss=2.9717 lr=1.89e-05 tok=23.547B dt=6693 wait=287 +14:02:27 step 178800 loss=2.9370 lr=1.89e-05 tok=23.550B dt=6756 wait=287 +14:02:39 pushed ckpt_w176.pt,log_w176.txt +14:04:15 waited 96s for global r=740 +14:04:20 global r=740 load 4.283 -> 4.279 +14:05:25 step 178820 loss=2.8641 lr=1.89e-05 tok=23.552B dt=6935 wait=287 +14:06:29 step 178840 loss=3.0758 lr=1.89e-05 tok=23.555B dt=6999 wait=287 +14:07:34 step 178860 loss=2.8970 lr=1.89e-05 tok=23.558B dt=7063 wait=288 +14:08:38 step 178880 loss=2.8024 lr=1.89e-05 tok=23.560B dt=7128 wait=288 +14:09:42 step 178900 loss=2.6721 lr=1.88e-05 tok=23.563B dt=7192 wait=288 +14:10:47 step 178920 loss=2.9210 lr=1.88e-05 tok=23.565B dt=7256 wait=288 +14:11:51 step 178940 loss=2.8133 lr=1.88e-05 tok=23.568B dt=7321 wait=288 +14:12:35 pushed ckpt_w176.pt,log_w176.txt +14:13:50 waited 76s for global r=741 +14:13:56 global r=741 load 4.280 -> 4.280 +14:14:30 step 178960 loss=2.9609 lr=1.88e-05 tok=23.571B dt=7479 wait=288 +14:15:33 step 178980 loss=2.9235 lr=1.88e-05 tok=23.573B dt=7543 wait=288 +14:16:37 step 179000 loss=2.9167 lr=1.88e-05 tok=23.576B dt=7607 wait=288 +14:17:41 step 179020 loss=2.8331 lr=1.88e-05 tok=23.579B dt=7670 wait=289 +14:18:45 step 179040 loss=2.7520 lr=1.88e-05 tok=23.581B dt=7735 wait=289 +14:19:49 step 179060 loss=2.9401 lr=1.88e-05 tok=23.584B dt=7798 wait=289 +14:20:52 step 179080 loss=2.8509 lr=1.88e-05 tok=23.586B dt=7862 wait=289 +14:21:56 step 179100 loss=2.7222 lr=1.88e-05 tok=23.589B dt=7925 wait=289 +14:22:08 pushed ckpt_w176.pt,log_w176.txt +14:23:38 waited 91s for global r=742 +14:23:43 global r=742 load 4.281 -> 4.282 +14:24:48 step 179120 loss=2.8251 lr=1.88e-05 tok=23.592B dt=8097 wait=289 +14:25:51 step 179140 loss=2.9041 lr=1.88e-05 tok=23.594B dt=8160 wait=289 +14:26:54 step 179160 loss=2.8413 lr=1.88e-05 tok=23.597B dt=8224 wait=290 +14:27:58 step 179180 loss=2.8361 lr=1.88e-05 tok=23.600B dt=8288 wait=290 +14:29:02 step 179200 loss=2.8300 lr=1.87e-05 tok=23.602B dt=8351 wait=290 +14:30:05 step 179220 loss=2.9740 lr=1.87e-05 tok=23.605B dt=8415 wait=290 +14:31:09 step 179240 loss=2.8476 lr=1.87e-05 tok=23.607B dt=8479 wait=290 +14:31:53 pushed ckpt_w176.pt,log_w176.txt +14:33:24 waited 91s for global r=743 +14:33:32 global r=743 load 4.289 -> 4.281 +14:34:05 step 179260 loss=2.9829 lr=1.87e-05 tok=23.610B dt=8654 wait=290 +14:35:09 step 179280 loss=2.8570 lr=1.87e-05 tok=23.613B dt=8718 wait=290 +14:36:13 step 179300 loss=2.8677 lr=1.87e-05 tok=23.615B dt=8783 wait=290 +14:37:17 step 179320 loss=2.9922 lr=1.87e-05 tok=23.618B dt=8847 wait=291 +14:38:21 step 179340 loss=2.8441 lr=1.87e-05 tok=23.620B dt=8910 wait=291 +14:39:24 step 179360 loss=2.9429 lr=1.87e-05 tok=23.623B dt=8974 wait=291 +14:40:28 step 179380 loss=2.8789 lr=1.87e-05 tok=23.626B dt=9038 wait=291 +14:41:33 step 179400 loss=2.8829 lr=1.87e-05 tok=23.628B dt=9102 wait=291 +14:41:44 pushed ckpt_w176.pt,log_w176.txt +14:43:00 waited 76s for global r=744 +14:43:06 global r=744 load 4.279 -> 4.280 +14:44:12 step 179420 loss=2.8264 lr=1.87e-05 tok=23.631B dt=9262 wait=291 +14:45:16 step 179440 loss=2.9405 lr=1.87e-05 tok=23.634B dt=9326 wait=291 +14:46:20 step 179460 loss=2.8190 lr=1.87e-05 tok=23.636B dt=9390 wait=292 +14:47:24 step 179480 loss=2.9182 lr=1.86e-05 tok=23.639B dt=9454 wait=292 +14:48:28 step 179500 loss=2.8654 lr=1.86e-05 tok=23.641B dt=9518 wait=292 +14:49:32 step 179520 loss=2.9773 lr=1.86e-05 tok=23.644B dt=9581 wait=292 +14:50:36 step 179540 loss=2.7608 lr=1.86e-05 tok=23.647B dt=9645 wait=292 +14:51:19 pushed ckpt_w176.pt,log_w176.txt +14:52:35 waited 76s for global r=745 +14:52:44 global r=745 load 4.281 -> 4.281 +14:53:17 step 179560 loss=2.8065 lr=1.86e-05 tok=23.649B dt=9806 wait=292 +14:54:20 step 179580 loss=2.8466 lr=1.86e-05 tok=23.652B dt=9870 wait=292 +14:55:24 step 179600 loss=2.7597 lr=1.86e-05 tok=23.655B dt=9934 wait=293 +14:56:28 step 179620 loss=2.9622 lr=1.86e-05 tok=23.657B dt=9998 wait=293 +14:57:32 step 179640 loss=2.8226 lr=1.86e-05 tok=23.660B dt=10062 wait=293 +14:58:36 step 179660 loss=2.9412 lr=1.86e-05 tok=23.662B dt=10125 wait=293 +14:59:40 step 179680 loss=2.8792 lr=1.86e-05 tok=23.665B dt=10190 wait=293 +15:00:44 step 179700 loss=3.0782 lr=1.86e-05 tok=23.668B dt=10254 wait=293 +15:00:56 pushed ckpt_w176.pt,log_w176.txt +15:05:03 step 179720 loss=2.9510 lr=1.86e-05 tok=23.670B dt=10513 wait=293 +15:06:07 step 179740 loss=2.9027 lr=1.86e-05 tok=23.673B dt=10577 wait=294 +15:07:11 step 179760 loss=2.8560 lr=1.86e-05 tok=23.676B dt=10641 wait=294 +15:08:16 step 179780 loss=2.7572 lr=1.85e-05 tok=23.678B dt=10705 wait=294 +15:09:20 step 179800 loss=2.8737 lr=1.85e-05 tok=23.681B dt=10769 wait=294 +15:10:23 step 179820 loss=3.0379 lr=1.85e-05 tok=23.683B dt=10833 wait=294 +15:11:28 step 179840 loss=2.6952 lr=1.85e-05 tok=23.686B dt=10898 wait=294 +15:12:12 pushed ckpt_w176.pt,log_w176.txt +15:15:48 step 179860 loss=2.9874 lr=1.85e-05 tok=23.689B dt=11158 wait=294 +15:16:53 step 179880 loss=2.9164 lr=1.85e-05 tok=23.691B dt=11222 wait=294 +15:17:57 step 179900 loss=2.8669 lr=1.85e-05 tok=23.694B dt=11287 wait=295 +15:19:01 step 179920 loss=2.9573 lr=1.85e-05 tok=23.697B dt=11351 wait=295 +15:19:39 global r=746 load 4.277 -> 4.282 +15:20:12 step 179860 loss=2.9721 lr=1.85e-05 tok=23.699B dt=11421 wait=295 +15:21:16 step 179880 loss=2.8960 lr=1.85e-05 tok=23.702B dt=11485 wait=295 +15:22:21 step 179900 loss=2.7475 lr=1.85e-05 tok=23.704B dt=11551 wait=295 +15:23:27 step 179920 loss=3.0690 lr=1.85e-05 tok=23.707B dt=11616 wait=295 +15:24:31 step 179940 loss=2.9449 lr=1.85e-05 tok=23.710B dt=11681 wait=296 +15:25:35 step 179960 loss=3.0992 lr=1.85e-05 tok=23.712B dt=11745 wait=296 +15:26:40 step 179980 loss=3.0355 lr=1.85e-05 tok=23.715B dt=11810 wait=296 +15:27:44 step 180000 loss=3.0398 lr=1.85e-05 tok=23.717B dt=11874 wait=296 +15:27:56 pushed ckpt_w176.pt,log_w176.txt +15:30:57 waited 181s for global r=747 +15:31:07 global r=747 load 4.280 -> 4.283 +15:32:13 step 180020 loss=2.8687 lr=1.85e-05 tok=23.720B dt=12143 wait=296 +15:33:16 step 180040 loss=2.9922 lr=1.85e-05 tok=23.723B dt=12206 wait=296 +15:34:20 step 180060 loss=3.0277 lr=1.84e-05 tok=23.725B dt=12270 wait=296 +15:35:24 step 180080 loss=2.8957 lr=1.84e-05 tok=23.728B dt=12334 wait=296 +15:36:28 step 180100 loss=2.9545 lr=1.84e-05 tok=23.731B dt=12397 wait=297 +15:37:31 step 180120 loss=2.9207 lr=1.84e-05 tok=23.733B dt=12461 wait=297 +15:38:34 step 180140 loss=2.8171 lr=1.84e-05 tok=23.736B dt=12524 wait=297 +15:39:19 pushed ckpt_w176.pt,log_w176.txt +15:42:53 step 180160 loss=2.9688 lr=1.84e-05 tok=23.738B dt=12782 wait=297 +15:42:59 global r=748 load 4.279 -> 4.280 +15:43:32 step 180160 loss=3.0046 lr=1.84e-05 tok=23.740B dt=12821 wait=297 +15:44:35 step 180180 loss=2.8555 lr=1.84e-05 tok=23.742B dt=12884 wait=297 +15:45:38 step 180200 loss=2.8808 lr=1.84e-05 tok=23.745B dt=12948 wait=297 +15:46:42 step 180220 loss=2.8935 lr=1.84e-05 tok=23.748B dt=13012 wait=297 +15:47:46 step 180240 loss=2.8437 lr=1.84e-05 tok=23.750B dt=13076 wait=298 +15:48:49 step 180260 loss=2.9281 lr=1.84e-05 tok=23.753B dt=13139 wait=298 +15:49:54 step 180280 loss=2.8809 lr=1.84e-05 tok=23.755B dt=13203 wait=298 +15:50:57 step 180300 loss=2.7752 lr=1.84e-05 tok=23.758B dt=13267 wait=298 +15:51:10 pushed ckpt_w176.pt,log_w176.txt +15:54:11 waited 181s for global r=749 +15:54:19 global r=749 load 4.282 -> 4.282 +15:55:24 step 180320 loss=2.8794 lr=1.84e-05 tok=23.761B dt=13533 wait=298 +15:56:28 step 180340 loss=2.8179 lr=1.84e-05 tok=23.763B dt=13597 wait=298 +15:57:31 step 180360 loss=2.8083 lr=1.84e-05 tok=23.766B dt=13661 wait=298 +15:58:35 step 180380 loss=2.7895 lr=1.83e-05 tok=23.769B dt=13725 wait=299 +15:59:38 step 180400 loss=2.8220 lr=1.83e-05 tok=23.771B dt=13788 wait=299 +16:00:42 step 180420 loss=2.9300 lr=1.83e-05 tok=23.774B dt=13851 wait=299 +16:01:45 step 180440 loss=2.8146 lr=1.83e-05 tok=23.776B dt=13915 wait=299 +16:02:29 pushed ckpt_w176.pt,log_w176.txt +16:06:03 step 180460 loss=2.9254 lr=1.83e-05 tok=23.779B dt=14173 wait=299 +16:06:10 global r=750 load 4.282 -> 4.283 +16:06:41 step 180460 loss=2.9251 lr=1.83e-05 tok=23.780B dt=14211 wait=299 +16:07:45 step 180480 loss=2.8026 lr=1.83e-05 tok=23.783B dt=14274 wait=299 +16:08:48 step 180500 loss=2.8371 lr=1.83e-05 tok=23.786B dt=14337 wait=299 +16:09:51 step 180520 loss=2.8690 lr=1.83e-05 tok=23.788B dt=14400 wait=300 +16:10:54 step 180540 loss=2.8023 lr=1.83e-05 tok=23.791B dt=14463 wait=300 +16:11:57 step 180560 loss=2.9052 lr=1.83e-05 tok=23.794B dt=14527 wait=300 +16:13:01 step 180580 loss=2.7319 lr=1.83e-05 tok=23.796B dt=14590 wait=300 +16:14:03 step 180600 loss=2.8261 lr=1.83e-05 tok=23.799B dt=14653 wait=300 +16:14:17 pushed ckpt_w176.pt,log_w176.txt +16:15:48 waited 91s for global r=751 +16:15:54 global r=751 load 4.281 -> 4.282 +16:16:58 step 180620 loss=2.8324 lr=1.83e-05 tok=23.801B dt=14828 wait=300 +16:18:01 step 180640 loss=2.9412 lr=1.83e-05 tok=23.804B dt=14890 wait=300 +16:19:04 step 180660 loss=2.8486 lr=1.83e-05 tok=23.807B dt=14953 wait=301 +16:20:07 step 180680 loss=2.7691 lr=1.82e-05 tok=23.809B dt=15016 wait=301 +16:21:10 step 180700 loss=2.8108 lr=1.82e-05 tok=23.812B dt=15080 wait=301 +16:22:14 step 180720 loss=2.8547 lr=1.82e-05 tok=23.814B dt=15144 wait=301 +16:23:17 step 180740 loss=2.8152 lr=1.82e-05 tok=23.817B dt=15207 wait=301 +16:24:01 pushed ckpt_w176.pt,log_w176.txt +16:27:02 waited 181s for global r=752 +16:27:10 global r=752 load 4.287 -> 4.281 +16:27:43 step 180760 loss=3.0619 lr=1.82e-05 tok=23.820B dt=15473 wait=301 +16:28:47 step 180780 loss=2.8528 lr=1.82e-05 tok=23.822B dt=15537 wait=301 +16:29:50 step 180800 loss=2.8243 lr=1.82e-05 tok=23.825B dt=15600 wait=302 +16:30:54 step 180820 loss=2.8421 lr=1.82e-05 tok=23.828B dt=15664 wait=302 +16:31:58 step 180840 loss=2.8341 lr=1.82e-05 tok=23.830B dt=15728 wait=302 +16:33:02 step 180860 loss=2.8887 lr=1.82e-05 tok=23.833B dt=15792 wait=302 +16:34:06 step 180880 loss=2.8651 lr=1.82e-05 tok=23.835B dt=15856 wait=302 +16:35:11 step 180900 loss=2.8476 lr=1.82e-05 tok=23.838B dt=15921 wait=302 +16:35:24 pushed ckpt_w176.pt,log_w176.txt +launch +16:39:06 name=w176 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 123/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:39:11 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:39:11 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:39:20 params 1.08B +16:39:22 resumed step=180900 seen=23.84B round=753 +16:39:22 stream salt=180900 +16:41:14 blocks compiled +16:41:14 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:46:31 step 180901 loss=2.8850 lr=1.76e-05 tok=23.838B dt=317 wait=308 +16:46:38 step 180902 loss=2.6837 lr=1.76e-05 tok=23.838B dt=324 wait=308 +16:47:35 step 180920 loss=2.9544 lr=1.76e-05 tok=23.841B dt=380 wait=308 +16:48:38 step 180940 loss=2.9171 lr=1.76e-05 tok=23.843B dt=443 wait=308 +16:49:40 step 180960 loss=3.0314 lr=1.76e-05 tok=23.846B dt=506 wait=308 +16:50:43 step 180980 loss=3.1028 lr=1.76e-05 tok=23.849B dt=569 wait=309 +16:50:56 global r=754 load 4.280 -> 4.281 +16:51:28 step 181060 loss=3.1937 lr=1.75e-05 tok=23.850B dt=613 wait=309 +16:52:31 step 181080 loss=2.9488 lr=1.75e-05 tok=23.852B dt=677 wait=309 +16:53:35 step 181100 loss=2.8956 lr=1.75e-05 tok=23.855B dt=740 wait=309 +16:54:38 step 181120 loss=3.2115 lr=1.74e-05 tok=23.858B dt=804 wait=309 +16:55:41 step 181140 loss=3.0698 lr=1.74e-05 tok=23.860B dt=867 wait=309 +16:56:45 step 181160 loss=3.2051 lr=1.74e-05 tok=23.863B dt=931 wait=309 +16:57:49 step 181180 loss=3.0966 lr=1.74e-05 tok=23.866B dt=995 wait=310 +16:58:53 step 181200 loss=3.2312 lr=1.74e-05 tok=23.868B dt=1058 wait=310 +16:59:06 pushed ckpt_w176.pt,log_w176.txt +17:01:07 waited 121s for global r=755 +17:01:15 global r=755 load 4.286 -> 4.280 +17:02:21 step 181220 loss=3.2346 lr=1.73e-05 tok=23.871B dt=1267 wait=310 +17:03:25 step 181240 loss=3.3001 lr=1.73e-05 tok=23.873B dt=1331 wait=310 +17:04:29 step 181260 loss=3.2441 lr=1.73e-05 tok=23.876B dt=1394 wait=310 +17:05:32 step 181280 loss=3.2721 lr=1.73e-05 tok=23.879B dt=1458 wait=310 +17:06:37 step 181300 loss=3.0257 lr=1.73e-05 tok=23.881B dt=1522 wait=311 +17:07:41 step 181320 loss=3.3075 lr=1.73e-05 tok=23.884B dt=1586 wait=311 +17:08:45 step 181340 loss=3.2141 lr=1.72e-05 tok=23.887B dt=1650 wait=311 +17:09:35 pushed ckpt_w176.pt,log_w176.txt +17:11:05 waited 91s for global r=756 +17:11:14 global r=756 load 4.283 -> 4.281 +17:11:47 step 181360 loss=3.2221 lr=1.72e-05 tok=23.889B dt=1832 wait=311 +17:12:50 step 181380 loss=3.1703 lr=1.72e-05 tok=23.892B dt=1895 wait=311 +17:13:54 step 181400 loss=3.1668 lr=1.72e-05 tok=23.894B dt=1959 wait=311 +17:14:58 step 181420 loss=3.3220 lr=1.72e-05 tok=23.897B dt=2024 wait=312 +17:16:02 step 181440 loss=3.1010 lr=1.71e-05 tok=23.900B dt=2087 wait=312 +17:17:06 step 181460 loss=3.1791 lr=1.71e-05 tok=23.902B dt=2151 wait=312 +17:18:10 step 181480 loss=3.1066 lr=1.71e-05 tok=23.905B dt=2216 wait=312 +17:19:15 step 181500 loss=3.1281 lr=1.71e-05 tok=23.908B dt=2280 wait=312 +17:19:29 pushed ckpt_w176.pt,log_w176.txt +17:22:30 waited 181s for global r=757 +17:22:40 global r=757 load 4.287 -> 4.282 +17:23:46 step 181520 loss=3.1437 lr=1.71e-05 tok=23.910B dt=2551 wait=312 +17:24:49 step 181540 loss=3.1280 lr=1.70e-05 tok=23.913B dt=2615 wait=313 +17:25:53 step 181560 loss=3.0699 lr=1.70e-05 tok=23.915B dt=2679 wait=313 +17:26:57 step 181580 loss=3.0531 lr=1.70e-05 tok=23.918B dt=2742 wait=313 +17:28:01 step 181600 loss=3.1775 lr=1.70e-05 tok=23.921B dt=2807 wait=313 +17:29:06 step 181620 loss=3.3636 lr=1.70e-05 tok=23.923B dt=2871 wait=313 +17:30:10 step 181640 loss=3.0372 lr=1.70e-05 tok=23.926B dt=2936 wait=313 +17:30:56 pushed ckpt_w176.pt,log_w176.txt +17:32:57 waited 121s for global r=758 +17:33:03 global r=758 load 4.284 -> 4.283 +17:33:36 step 181660 loss=3.1021 lr=1.69e-05 tok=23.929B dt=3141 wait=314 +17:34:40 step 181680 loss=3.0108 lr=1.69e-05 tok=23.931B dt=3206 wait=314 +17:35:44 step 181700 loss=3.0030 lr=1.69e-05 tok=23.934B dt=3270 wait=314 +17:36:48 step 181720 loss=3.2368 lr=1.69e-05 tok=23.936B dt=3334 wait=314 +17:37:52 step 181740 loss=3.1131 lr=1.69e-05 tok=23.939B dt=3397 wait=314 +17:38:55 step 181760 loss=3.2496 lr=1.68e-05 tok=23.942B dt=3461 wait=314 +17:40:00 step 181780 loss=3.1183 lr=1.68e-05 tok=23.944B dt=3525 wait=314 +17:41:04 step 181800 loss=3.2704 lr=1.68e-05 tok=23.947B dt=3590 wait=315 +17:41:18 pushed ckpt_w176.pt,log_w176.txt +17:42:34 waited 76s for global r=759 +17:42:40 global r=759 load 4.282 -> 4.283 +17:43:45 step 181820 loss=3.1198 lr=1.68e-05 tok=23.949B dt=3751 wait=315 +17:44:49 step 181840 loss=2.9080 lr=1.68e-05 tok=23.952B dt=3815 wait=315 +17:45:53 step 181860 loss=3.0763 lr=1.68e-05 tok=23.955B dt=3879 wait=315 +17:46:57 step 181880 loss=3.0503 lr=1.67e-05 tok=23.957B dt=3943 wait=315 +17:48:01 step 181900 loss=3.1107 lr=1.67e-05 tok=23.960B dt=4007 wait=315 +17:49:05 step 181920 loss=3.1163 lr=1.67e-05 tok=23.963B dt=4071 wait=316 +17:50:09 step 181940 loss=3.0997 lr=1.67e-05 tok=23.965B dt=4135 wait=316 +17:50:53 pushed ckpt_w176.pt,log_w176.txt +17:52:54 waited 121s for global r=760 +17:53:01 global r=760 load 4.281 -> 4.281 +17:53:34 step 181960 loss=3.1429 lr=1.67e-05 tok=23.968B dt=4340 wait=316 +17:54:37 step 181980 loss=2.9763 lr=1.66e-05 tok=23.970B dt=4403 wait=316 +17:55:42 step 182000 loss=2.9318 lr=1.66e-05 tok=23.973B dt=4467 wait=316 +17:56:46 step 182020 loss=3.1134 lr=1.66e-05 tok=23.976B dt=4532 wait=316 +17:57:51 step 182040 loss=3.1524 lr=1.66e-05 tok=23.978B dt=4596 wait=317 +17:58:55 step 182060 loss=3.2202 lr=1.66e-05 tok=23.981B dt=4661 wait=317 +17:59:59 step 182080 loss=3.1301 lr=1.65e-05 tok=23.984B dt=4725 wait=317 +18:01:04 step 182100 loss=3.1812 lr=1.65e-05 tok=23.986B dt=4790 wait=317 +18:01:18 pushed ckpt_w176.pt,log_w176.txt +18:02:49 waited 91s for global r=761 +18:03:05 global r=761 load 4.281 -> 4.281 +18:04:12 step 182120 loss=3.0253 lr=1.65e-05 tok=23.989B dt=4978 wait=317 +18:05:17 step 182140 loss=3.2031 lr=1.65e-05 tok=23.991B dt=5042 wait=318 +18:06:22 step 182160 loss=3.0985 lr=1.65e-05 tok=23.994B dt=5107 wait=318 +18:07:26 step 182180 loss=3.0392 lr=1.65e-05 tok=23.997B dt=5172 wait=318 +18:08:30 step 182200 loss=3.0926 lr=1.64e-05 tok=23.999B dt=5236 wait=318 +18:09:35 step 182220 loss=3.1949 lr=1.64e-05 tok=24.002B dt=5300 wait=318 +18:10:39 step 182240 loss=3.0587 lr=1.64e-05 tok=24.005B dt=5365 wait=318 diff --git a/log_w177.txt b/log_w177.txt new file mode 100644 index 0000000000000000000000000000000000000000..793312901cf338402aeabbb0790c31361116eb0d --- /dev/null +++ b/log_w177.txt @@ -0,0 +1,176 @@ +launch +12:06:50 name=w177 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 124/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:06:57 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:06:57 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:07:06 params 1.08B +12:07:08 resumed step=177150 seen=23.33B round=729 +12:07:08 stream salt=177150 +12:09:20 blocks compiled +12:09:20 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:14:32 step 177151 loss=2.8116 lr=1.95e-05 tok=23.332B dt=312 wait=286 +12:14:48 step 177152 loss=2.7955 lr=1.95e-05 tok=23.332B dt=328 wait=286 +12:15:13 step 177160 loss=3.1949 lr=1.95e-05 tok=23.333B dt=354 wait=286 +12:16:19 step 177180 loss=2.9829 lr=1.95e-05 tok=23.336B dt=419 wait=287 +12:17:24 step 177200 loss=2.9798 lr=1.95e-05 tok=23.339B dt=485 wait=287 +12:18:28 step 177220 loss=3.0204 lr=1.95e-05 tok=23.341B dt=549 wait=287 +12:19:33 step 177240 loss=3.0952 lr=1.95e-05 tok=23.344B dt=613 wait=287 +12:20:38 step 177260 loss=3.1877 lr=1.94e-05 tok=23.347B dt=678 wait=287 +12:21:43 step 177280 loss=3.1229 lr=1.94e-05 tok=23.349B dt=743 wait=287 +12:22:48 step 177300 loss=3.1234 lr=1.94e-05 tok=23.352B dt=809 wait=287 +12:23:04 pushed ckpt_w177.pt,log_w177.txt +12:25:05 waited 121s for global r=730 +12:25:26 global r=730 load 4.285 -> 4.280 +12:26:34 step 177320 loss=3.1131 lr=1.94e-05 tok=23.354B dt=1034 wait=288 +12:27:39 step 177340 loss=3.1513 lr=1.94e-05 tok=23.357B dt=1099 wait=288 +12:28:44 step 177360 loss=3.0106 lr=1.94e-05 tok=23.360B dt=1164 wait=288 +12:29:48 step 177380 loss=3.0540 lr=1.94e-05 tok=23.362B dt=1229 wait=288 +12:30:54 step 177400 loss=3.0233 lr=1.94e-05 tok=23.365B dt=1294 wait=288 +12:31:58 step 177420 loss=3.1308 lr=1.94e-05 tok=23.368B dt=1359 wait=288 +12:33:03 step 177440 loss=2.9647 lr=1.94e-05 tok=23.370B dt=1423 wait=289 +12:33:48 pushed ckpt_w177.pt,log_w177.txt +12:37:24 step 177460 loss=3.1836 lr=1.94e-05 tok=23.373B dt=1684 wait=289 +12:37:29 global r=731 load 4.275 -> 4.278 +12:38:02 step 177460 loss=3.2221 lr=1.94e-05 tok=23.374B dt=1722 wait=289 +12:39:05 step 177480 loss=3.0848 lr=1.94e-05 tok=23.377B dt=1786 wait=289 +12:40:10 step 177500 loss=3.0854 lr=1.94e-05 tok=23.379B dt=1850 wait=289 +12:41:14 step 177520 loss=3.1214 lr=1.94e-05 tok=23.382B dt=1914 wait=289 +12:42:17 step 177540 loss=2.9512 lr=1.93e-05 tok=23.385B dt=1977 wait=289 +12:43:20 step 177560 loss=3.0978 lr=1.93e-05 tok=23.387B dt=2041 wait=289 +12:44:24 step 177580 loss=3.0370 lr=1.93e-05 tok=23.390B dt=2105 wait=290 +12:45:27 step 177600 loss=3.0989 lr=1.93e-05 tok=23.392B dt=2168 wait=290 +12:45:39 pushed ckpt_w177.pt,log_w177.txt +12:46:54 waited 76s for global r=732 +12:47:02 global r=732 load 4.279 -> 4.280 +12:48:07 step 177620 loss=3.0362 lr=1.93e-05 tok=23.395B dt=2328 wait=290 +12:49:11 step 177640 loss=3.0213 lr=1.93e-05 tok=23.398B dt=2392 wait=290 +12:50:16 step 177660 loss=2.7483 lr=1.93e-05 tok=23.400B dt=2457 wait=290 +12:51:21 step 177680 loss=3.1028 lr=1.93e-05 tok=23.403B dt=2521 wait=290 +12:52:25 step 177700 loss=3.0452 lr=1.93e-05 tok=23.406B dt=2585 wait=290 +12:53:29 step 177720 loss=3.2980 lr=1.93e-05 tok=23.408B dt=2649 wait=291 +12:54:32 step 177740 loss=3.0669 lr=1.93e-05 tok=23.411B dt=2713 wait=291 +12:55:17 pushed ckpt_w177.pt,log_w177.txt +12:56:17 waited 61s for global r=733 +12:56:22 global r=733 load 4.280 -> 4.282 +12:56:56 step 177760 loss=2.8502 lr=1.93e-05 tok=23.413B dt=2856 wait=291 +12:57:59 step 177780 loss=3.0651 lr=1.93e-05 tok=23.416B dt=2920 wait=291 +12:59:03 step 177800 loss=3.1465 lr=1.92e-05 tok=23.419B dt=2983 wait=291 +13:00:07 step 177820 loss=3.1736 lr=1.92e-05 tok=23.421B dt=3047 wait=291 +13:01:10 step 177840 loss=3.0518 lr=1.92e-05 tok=23.424B dt=3110 wait=291 +13:02:14 step 177860 loss=2.9741 lr=1.92e-05 tok=23.426B dt=3174 wait=292 +13:03:18 step 177880 loss=3.0180 lr=1.92e-05 tok=23.429B dt=3238 wait=292 +13:04:22 step 177900 loss=3.1271 lr=1.92e-05 tok=23.432B dt=3302 wait=292 +13:04:34 pushed ckpt_w177.pt,log_w177.txt +13:06:04 waited 91s for global r=734 +13:06:10 global r=734 load 4.293 -> 4.283 +13:07:15 step 177920 loss=3.2044 lr=1.92e-05 tok=23.434B dt=3475 wait=292 +13:08:19 step 177940 loss=3.0948 lr=1.92e-05 tok=23.437B dt=3539 wait=292 +13:09:23 step 177960 loss=3.1535 lr=1.92e-05 tok=23.440B dt=3603 wait=292 +13:10:27 step 177980 loss=3.0488 lr=1.92e-05 tok=23.442B dt=3667 wait=292 +13:11:30 step 178000 loss=3.0724 lr=1.92e-05 tok=23.445B dt=3731 wait=292 +13:12:35 step 178020 loss=3.1377 lr=1.92e-05 tok=23.447B dt=3795 wait=293 +13:13:39 step 178040 loss=3.0424 lr=1.92e-05 tok=23.450B dt=3860 wait=293 +13:14:26 pushed ckpt_w177.pt,log_w177.txt +13:15:41 waited 76s for global r=735 +13:15:47 global r=735 load 4.285 -> 4.283 +13:16:22 step 178060 loss=3.0382 lr=1.92e-05 tok=23.453B dt=4022 wait=293 +13:17:25 step 178080 loss=3.0612 lr=1.91e-05 tok=23.455B dt=4086 wait=293 +13:18:30 step 178100 loss=2.9558 lr=1.91e-05 tok=23.458B dt=4150 wait=293 +13:19:34 step 178120 loss=3.0552 lr=1.91e-05 tok=23.461B dt=4214 wait=293 +13:20:38 step 178140 loss=2.8993 lr=1.91e-05 tok=23.463B dt=4278 wait=294 +13:21:41 step 178160 loss=3.0802 lr=1.91e-05 tok=23.466B dt=4342 wait=294 +13:22:46 step 178180 loss=2.9385 lr=1.91e-05 tok=23.468B dt=4406 wait=294 +13:23:50 step 178200 loss=2.9875 lr=1.91e-05 tok=23.471B dt=4471 wait=294 +13:24:03 pushed ckpt_w177.pt,log_w177.txt +13:25:18 waited 76s for global r=736 +13:25:25 global r=736 load 4.292 -> 4.283 +13:26:30 step 178220 loss=3.0784 lr=1.91e-05 tok=23.474B dt=4630 wait=294 +13:27:34 step 178240 loss=3.0471 lr=1.91e-05 tok=23.476B dt=4694 wait=295 +13:28:38 step 178260 loss=3.0966 lr=1.91e-05 tok=23.479B dt=4758 wait=295 +13:29:42 step 178280 loss=2.9681 lr=1.91e-05 tok=23.482B dt=4822 wait=295 +13:30:46 step 178300 loss=3.0461 lr=1.91e-05 tok=23.484B dt=4886 wait=295 +13:31:50 step 178320 loss=3.1301 lr=1.91e-05 tok=23.487B dt=4950 wait=295 +13:32:53 step 178340 loss=3.1007 lr=1.90e-05 tok=23.489B dt=5014 wait=295 +13:33:39 pushed ckpt_w177.pt,log_w177.txt +13:34:55 waited 76s for global r=737 +13:35:00 global r=737 load 4.289 -> 4.282 +13:35:34 step 178360 loss=3.1824 lr=1.90e-05 tok=23.492B dt=5174 wait=295 +13:36:36 step 178380 loss=3.0323 lr=1.90e-05 tok=23.495B dt=5237 wait=295 +13:37:40 step 178400 loss=3.0348 lr=1.90e-05 tok=23.497B dt=5300 wait=296 +13:38:44 step 178420 loss=3.0270 lr=1.90e-05 tok=23.500B dt=5364 wait=296 +13:39:47 step 178440 loss=2.9804 lr=1.90e-05 tok=23.503B dt=5427 wait=296 +13:40:50 step 178460 loss=3.0637 lr=1.90e-05 tok=23.505B dt=5490 wait=296 +13:41:52 step 178480 loss=2.9671 lr=1.90e-05 tok=23.508B dt=5552 wait=296 +13:42:55 step 178500 loss=2.9723 lr=1.90e-05 tok=23.510B dt=5615 wait=296 +13:43:09 pushed ckpt_w177.pt,log_w177.txt +13:44:40 waited 91s for global r=738 +13:44:45 global r=738 load 4.285 -> 4.284 +13:45:50 step 178520 loss=2.9506 lr=1.90e-05 tok=23.513B dt=5790 wait=297 +13:46:54 step 178540 loss=3.0092 lr=1.90e-05 tok=23.516B dt=5854 wait=297 +13:47:57 step 178560 loss=3.0074 lr=1.90e-05 tok=23.518B dt=5918 wait=297 +13:49:01 step 178580 loss=2.9707 lr=1.90e-05 tok=23.521B dt=5981 wait=297 +13:50:03 step 178600 loss=2.9284 lr=1.90e-05 tok=23.523B dt=6044 wait=297 +13:51:07 step 178620 loss=2.9838 lr=1.89e-05 tok=23.526B dt=6107 wait=297 +13:52:11 step 178640 loss=2.8633 lr=1.89e-05 tok=23.529B dt=6171 wait=297 +13:52:55 pushed ckpt_w177.pt,log_w177.txt +13:54:25 waited 91s for global r=739 +13:54:34 global r=739 load 4.277 -> 4.282 +13:55:08 step 178660 loss=3.1089 lr=1.89e-05 tok=23.531B dt=6348 wait=298 +13:56:11 step 178680 loss=2.8742 lr=1.89e-05 tok=23.534B dt=6412 wait=298 +13:57:15 step 178700 loss=3.0743 lr=1.89e-05 tok=23.537B dt=6475 wait=298 +13:58:19 step 178720 loss=3.0875 lr=1.89e-05 tok=23.539B dt=6539 wait=298 +13:59:22 step 178740 loss=2.8575 lr=1.89e-05 tok=23.542B dt=6603 wait=298 +14:00:26 step 178760 loss=3.0147 lr=1.89e-05 tok=23.544B dt=6666 wait=298 +14:01:30 step 178780 loss=2.9322 lr=1.89e-05 tok=23.547B dt=6730 wait=298 +14:02:34 step 178800 loss=2.9073 lr=1.89e-05 tok=23.550B dt=6794 wait=299 +14:02:48 pushed ckpt_w177.pt,log_w177.txt +14:04:19 waited 91s for global r=740 +14:04:24 global r=740 load 4.281 -> 4.279 +14:05:29 step 178820 loss=2.8374 lr=1.89e-05 tok=23.552B dt=6970 wait=299 +14:06:33 step 178840 loss=3.0058 lr=1.89e-05 tok=23.555B dt=7033 wait=299 +14:07:37 step 178860 loss=2.9067 lr=1.89e-05 tok=23.558B dt=7098 wait=299 +14:08:42 step 178880 loss=2.6644 lr=1.89e-05 tok=23.560B dt=7163 wait=299 +14:09:46 step 178900 loss=2.8626 lr=1.88e-05 tok=23.563B dt=7227 wait=299 +14:10:50 step 178920 loss=2.9004 lr=1.88e-05 tok=23.565B dt=7291 wait=299 +14:11:55 step 178940 loss=2.6642 lr=1.88e-05 tok=23.568B dt=7355 wait=300 +14:12:39 pushed ckpt_w177.pt,log_w177.txt +14:13:55 waited 76s for global r=741 +14:14:00 global r=741 load 4.288 -> 4.280 +14:14:34 step 178960 loss=2.8849 lr=1.88e-05 tok=23.571B dt=7514 wait=300 +14:15:38 step 178980 loss=2.8273 lr=1.88e-05 tok=23.573B dt=7578 wait=300 +14:16:41 step 179000 loss=2.8720 lr=1.88e-05 tok=23.576B dt=7642 wait=300 +14:17:46 step 179020 loss=2.7988 lr=1.88e-05 tok=23.579B dt=7706 wait=300 +14:18:50 step 179040 loss=2.7211 lr=1.88e-05 tok=23.581B dt=7770 wait=300 +14:19:54 step 179060 loss=2.8483 lr=1.88e-05 tok=23.584B dt=7834 wait=300 +14:20:58 step 179080 loss=2.8654 lr=1.88e-05 tok=23.586B dt=7899 wait=301 +14:22:03 step 179100 loss=2.7573 lr=1.88e-05 tok=23.589B dt=7963 wait=301 +14:22:15 pushed ckpt_w177.pt,log_w177.txt +14:23:30 waited 76s for global r=742 +14:23:38 global r=742 load 4.286 -> 4.282 +14:24:43 step 179120 loss=2.7764 lr=1.88e-05 tok=23.592B dt=8123 wait=301 +14:25:47 step 179140 loss=2.5909 lr=1.88e-05 tok=23.594B dt=8187 wait=301 +14:26:51 step 179160 loss=2.7924 lr=1.88e-05 tok=23.597B dt=8252 wait=301 +14:27:56 step 179180 loss=2.7002 lr=1.88e-05 tok=23.600B dt=8316 wait=301 +14:29:00 step 179200 loss=2.7261 lr=1.87e-05 tok=23.602B dt=8380 wait=301 +14:30:04 step 179220 loss=2.9196 lr=1.87e-05 tok=23.605B dt=8444 wait=302 +14:31:08 step 179240 loss=2.6750 lr=1.87e-05 tok=23.607B dt=8508 wait=302 +14:31:54 pushed ckpt_w177.pt,log_w177.txt +14:33:25 waited 91s for global r=743 +14:33:31 global r=743 load 4.285 -> 4.281 +14:34:04 step 179260 loss=2.8169 lr=1.87e-05 tok=23.610B dt=8684 wait=302 +14:35:08 step 179280 loss=2.6891 lr=1.87e-05 tok=23.613B dt=8748 wait=302 +14:36:13 step 179300 loss=2.6869 lr=1.87e-05 tok=23.615B dt=8813 wait=302 +14:37:17 step 179320 loss=2.7807 lr=1.87e-05 tok=23.618B dt=8877 wait=302 +14:38:20 step 179340 loss=2.7695 lr=1.87e-05 tok=23.620B dt=8941 wait=302 +14:39:24 step 179360 loss=2.8584 lr=1.87e-05 tok=23.623B dt=9004 wait=303 +14:40:27 step 179380 loss=2.7885 lr=1.87e-05 tok=23.626B dt=9068 wait=303 +14:41:32 step 179400 loss=2.2649 lr=1.87e-05 tok=23.628B dt=9133 wait=303 +14:41:48 pushed ckpt_w177.pt,log_w177.txt +14:43:04 waited 76s for global r=744 +14:43:09 global r=744 load 4.274 -> 4.280 +14:44:13 step 179420 loss=2.7615 lr=1.87e-05 tok=23.631B dt=9294 wait=303 +14:45:17 step 179440 loss=2.8493 lr=1.87e-05 tok=23.634B dt=9357 wait=303 +14:46:20 step 179460 loss=2.8343 lr=1.87e-05 tok=23.636B dt=9421 wait=303 +14:47:23 step 179480 loss=2.8341 lr=1.86e-05 tok=23.639B dt=9484 wait=303 +14:48:27 step 179500 loss=2.7647 lr=1.86e-05 tok=23.641B dt=9547 wait=303 +14:49:31 step 179520 loss=2.8734 lr=1.86e-05 tok=23.644B dt=9611 wait=304 +14:50:34 step 179540 loss=2.8206 lr=1.86e-05 tok=23.647B dt=9675 wait=304 diff --git a/log_w178.txt b/log_w178.txt new file mode 100644 index 0000000000000000000000000000000000000000..80570182a72b64b80c5d42f2bc42a37a6e2c51be --- /dev/null +++ b/log_w178.txt @@ -0,0 +1,176 @@ +launch +12:07:08 name=w178 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 125/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:07:14 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:07:14 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:07:23 params 1.08B +12:07:25 resumed step=177150 seen=23.33B round=729 +12:07:25 stream salt=177150 +12:09:41 blocks compiled +12:09:41 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +12:15:12 step 177151 loss=2.5566 lr=1.95e-05 tok=23.332B dt=331 wait=302 +12:15:28 step 177152 loss=2.6316 lr=1.95e-05 tok=23.332B dt=348 wait=302 +12:15:53 step 177160 loss=3.1062 lr=1.95e-05 tok=23.333B dt=373 wait=302 +12:16:58 step 177180 loss=2.8612 lr=1.95e-05 tok=23.336B dt=437 wait=302 +12:18:02 step 177200 loss=2.9574 lr=1.95e-05 tok=23.339B dt=501 wait=302 +12:19:05 step 177220 loss=3.0019 lr=1.95e-05 tok=23.341B dt=564 wait=302 +12:20:08 step 177240 loss=3.0404 lr=1.95e-05 tok=23.344B dt=627 wait=302 +12:21:12 step 177260 loss=3.1076 lr=1.94e-05 tok=23.347B dt=691 wait=302 +12:22:15 step 177280 loss=3.0147 lr=1.94e-05 tok=23.349B dt=754 wait=302 +12:23:19 step 177300 loss=3.0150 lr=1.94e-05 tok=23.352B dt=818 wait=303 +12:23:33 pushed ckpt_w178.pt,log_w178.txt +12:25:04 waited 91s for global r=730 +12:25:24 global r=730 load 4.283 -> 4.280 +12:26:31 step 177320 loss=2.9658 lr=1.94e-05 tok=23.354B dt=1010 wait=303 +12:27:35 step 177340 loss=3.0717 lr=1.94e-05 tok=23.357B dt=1074 wait=303 +12:28:39 step 177360 loss=3.0696 lr=1.94e-05 tok=23.360B dt=1138 wait=303 +12:29:44 step 177380 loss=3.0502 lr=1.94e-05 tok=23.362B dt=1203 wait=303 +12:30:48 step 177400 loss=3.0367 lr=1.94e-05 tok=23.365B dt=1267 wait=303 +12:31:53 step 177420 loss=3.0766 lr=1.94e-05 tok=23.368B dt=1332 wait=304 +12:32:56 step 177440 loss=3.0245 lr=1.94e-05 tok=23.370B dt=1396 wait=304 +12:33:40 pushed ckpt_w178.pt,log_w178.txt +12:37:16 step 177460 loss=3.1884 lr=1.94e-05 tok=23.373B dt=1656 wait=304 +12:37:23 global r=731 load 4.280 -> 4.278 +12:37:55 step 177460 loss=3.1693 lr=1.94e-05 tok=23.374B dt=1694 wait=304 +12:38:58 step 177480 loss=3.0394 lr=1.94e-05 tok=23.377B dt=1758 wait=304 +12:40:02 step 177500 loss=3.0957 lr=1.94e-05 tok=23.379B dt=1821 wait=304 +12:41:06 step 177520 loss=3.1637 lr=1.94e-05 tok=23.382B dt=1885 wait=304 +12:42:10 step 177540 loss=2.9390 lr=1.93e-05 tok=23.385B dt=1949 wait=304 +12:43:13 step 177560 loss=3.1312 lr=1.93e-05 tok=23.387B dt=2013 wait=305 +12:44:17 step 177580 loss=3.0113 lr=1.93e-05 tok=23.390B dt=2077 wait=305 +12:45:21 step 177600 loss=3.0819 lr=1.93e-05 tok=23.392B dt=2140 wait=305 +12:45:33 pushed ckpt_w178.pt,log_w178.txt +12:46:49 waited 76s for global r=732 +12:46:56 global r=732 load 4.289 -> 4.280 +12:48:02 step 177620 loss=3.0418 lr=1.93e-05 tok=23.395B dt=2301 wait=305 +12:49:06 step 177640 loss=2.9552 lr=1.93e-05 tok=23.398B dt=2365 wait=305 +12:50:10 step 177660 loss=3.1016 lr=1.93e-05 tok=23.400B dt=2430 wait=305 +12:51:15 step 177680 loss=2.9545 lr=1.93e-05 tok=23.403B dt=2494 wait=306 +12:52:19 step 177700 loss=3.0031 lr=1.93e-05 tok=23.406B dt=2559 wait=306 +12:53:24 step 177720 loss=3.0036 lr=1.93e-05 tok=23.408B dt=2623 wait=306 +12:54:27 step 177740 loss=2.9915 lr=1.93e-05 tok=23.411B dt=2686 wait=306 +12:55:11 pushed ckpt_w178.pt,log_w178.txt +12:56:11 waited 60s for global r=733 +12:56:18 global r=733 load 4.280 -> 4.282 +12:56:51 step 177760 loss=3.1614 lr=1.93e-05 tok=23.413B dt=2831 wait=306 +12:57:55 step 177780 loss=2.9463 lr=1.93e-05 tok=23.416B dt=2895 wait=306 +12:58:59 step 177800 loss=2.9442 lr=1.92e-05 tok=23.419B dt=2958 wait=307 +13:00:03 step 177820 loss=3.0876 lr=1.92e-05 tok=23.421B dt=3022 wait=307 +13:01:06 step 177840 loss=3.0049 lr=1.92e-05 tok=23.424B dt=3086 wait=307 +13:02:11 step 177860 loss=3.0636 lr=1.92e-05 tok=23.426B dt=3150 wait=307 +13:03:16 step 177880 loss=3.0375 lr=1.92e-05 tok=23.429B dt=3215 wait=307 +13:04:21 step 177900 loss=3.2111 lr=1.92e-05 tok=23.432B dt=3280 wait=307 +13:04:33 pushed ckpt_w178.pt,log_w178.txt +13:06:03 waited 91s for global r=734 +13:06:10 global r=734 load 4.289 -> 4.283 +13:07:16 step 177920 loss=3.0791 lr=1.92e-05 tok=23.434B dt=3455 wait=307 +13:08:20 step 177940 loss=3.1448 lr=1.92e-05 tok=23.437B dt=3520 wait=308 +13:09:24 step 177960 loss=3.0625 lr=1.92e-05 tok=23.440B dt=3584 wait=308 +13:10:29 step 177980 loss=3.1563 lr=1.92e-05 tok=23.442B dt=3648 wait=308 +13:11:32 step 178000 loss=3.1115 lr=1.92e-05 tok=23.445B dt=3712 wait=308 +13:12:36 step 178020 loss=3.0206 lr=1.92e-05 tok=23.447B dt=3776 wait=308 +13:13:39 step 178040 loss=2.9910 lr=1.92e-05 tok=23.450B dt=3839 wait=308 +13:14:24 pushed ckpt_w178.pt,log_w178.txt +13:15:40 waited 76s for global r=735 +13:15:46 global r=735 load 4.282 -> 4.283 +13:16:19 step 178060 loss=3.2325 lr=1.92e-05 tok=23.453B dt=3998 wait=308 +13:17:22 step 178080 loss=3.0333 lr=1.91e-05 tok=23.455B dt=4061 wait=309 +13:18:26 step 178100 loss=2.9189 lr=1.91e-05 tok=23.458B dt=4125 wait=309 +13:19:30 step 178120 loss=3.1015 lr=1.91e-05 tok=23.461B dt=4189 wait=309 +13:20:33 step 178140 loss=2.9361 lr=1.91e-05 tok=23.463B dt=4253 wait=309 +13:21:37 step 178160 loss=3.1322 lr=1.91e-05 tok=23.466B dt=4316 wait=309 +13:22:42 step 178180 loss=2.9772 lr=1.91e-05 tok=23.468B dt=4381 wait=309 +13:23:46 step 178200 loss=3.0928 lr=1.91e-05 tok=23.471B dt=4445 wait=310 +13:23:58 pushed ckpt_w178.pt,log_w178.txt +13:25:29 waited 91s for global r=736 +13:25:36 global r=736 load 4.283 -> 4.283 +13:26:41 step 178220 loss=3.0483 lr=1.91e-05 tok=23.474B dt=4620 wait=310 +13:27:45 step 178240 loss=3.0905 lr=1.91e-05 tok=23.476B dt=4684 wait=310 +13:28:49 step 178260 loss=3.2254 lr=1.91e-05 tok=23.479B dt=4748 wait=310 +13:29:53 step 178280 loss=3.0794 lr=1.91e-05 tok=23.482B dt=4812 wait=310 +13:30:57 step 178300 loss=3.1984 lr=1.91e-05 tok=23.484B dt=4877 wait=310 +13:32:02 step 178320 loss=3.2235 lr=1.91e-05 tok=23.487B dt=4941 wait=311 +13:33:05 step 178340 loss=3.1897 lr=1.90e-05 tok=23.489B dt=5005 wait=311 +13:33:53 pushed ckpt_w178.pt,log_w178.txt +13:35:09 waited 76s for global r=737 +13:35:20 global r=737 load 4.276 -> 4.282 +13:35:53 step 178360 loss=3.3964 lr=1.90e-05 tok=23.492B dt=5173 wait=311 +13:36:57 step 178380 loss=3.1383 lr=1.90e-05 tok=23.495B dt=5236 wait=311 +13:38:02 step 178400 loss=3.1206 lr=1.90e-05 tok=23.497B dt=5301 wait=311 +13:39:06 step 178420 loss=3.1097 lr=1.90e-05 tok=23.500B dt=5366 wait=311 +13:40:10 step 178440 loss=3.1653 lr=1.90e-05 tok=23.503B dt=5429 wait=311 +13:41:14 step 178460 loss=3.2174 lr=1.90e-05 tok=23.505B dt=5493 wait=312 +13:42:19 step 178480 loss=2.5327 lr=1.90e-05 tok=23.508B dt=5558 wait=312 +13:43:23 step 178500 loss=3.0564 lr=1.90e-05 tok=23.510B dt=5622 wait=312 +13:43:35 pushed ckpt_w178.pt,log_w178.txt +13:44:36 waited 61s for global r=738 +13:44:41 global r=738 load 4.287 -> 4.284 +13:45:47 step 178520 loss=3.1180 lr=1.90e-05 tok=23.513B dt=5767 wait=312 +13:46:51 step 178540 loss=3.1426 lr=1.90e-05 tok=23.516B dt=5831 wait=312 +13:47:56 step 178560 loss=3.0915 lr=1.90e-05 tok=23.518B dt=5895 wait=312 +13:48:59 step 178580 loss=3.0367 lr=1.90e-05 tok=23.521B dt=5959 wait=312 +13:50:03 step 178600 loss=3.1156 lr=1.90e-05 tok=23.523B dt=6023 wait=313 +13:51:07 step 178620 loss=3.1894 lr=1.89e-05 tok=23.526B dt=6087 wait=313 +13:52:12 step 178640 loss=3.0249 lr=1.89e-05 tok=23.529B dt=6151 wait=313 +13:52:58 pushed ckpt_w178.pt,log_w178.txt +13:54:29 waited 91s for global r=739 +13:54:36 global r=739 load 4.273 -> 4.282 +13:55:10 step 178660 loss=3.2036 lr=1.89e-05 tok=23.531B dt=6329 wait=313 +13:56:14 step 178680 loss=3.1764 lr=1.89e-05 tok=23.534B dt=6393 wait=313 +13:57:18 step 178700 loss=3.0977 lr=1.89e-05 tok=23.537B dt=6458 wait=313 +13:58:22 step 178720 loss=3.2241 lr=1.89e-05 tok=23.539B dt=6522 wait=313 +13:59:27 step 178740 loss=3.1184 lr=1.89e-05 tok=23.542B dt=6587 wait=314 +14:00:31 step 178760 loss=3.1469 lr=1.89e-05 tok=23.544B dt=6651 wait=314 +14:01:36 step 178780 loss=3.1038 lr=1.89e-05 tok=23.547B dt=6715 wait=314 +14:02:39 step 178800 loss=3.0626 lr=1.89e-05 tok=23.550B dt=6778 wait=314 +14:02:52 pushed ckpt_w178.pt,log_w178.txt +14:04:08 waited 76s for global r=740 +14:04:15 global r=740 load 4.280 -> 4.279 +14:05:20 step 178820 loss=3.0460 lr=1.89e-05 tok=23.552B dt=6940 wait=314 +14:06:24 step 178840 loss=3.0769 lr=1.89e-05 tok=23.555B dt=7003 wait=314 +14:07:28 step 178860 loss=3.0215 lr=1.89e-05 tok=23.558B dt=7067 wait=314 +14:08:32 step 178880 loss=2.9450 lr=1.89e-05 tok=23.560B dt=7132 wait=315 +14:09:36 step 178900 loss=3.0058 lr=1.88e-05 tok=23.563B dt=7195 wait=315 +14:10:40 step 178920 loss=2.9350 lr=1.88e-05 tok=23.565B dt=7260 wait=315 +14:11:44 step 178940 loss=2.8223 lr=1.88e-05 tok=23.568B dt=7323 wait=315 +14:12:30 pushed ckpt_w178.pt,log_w178.txt +14:14:01 waited 91s for global r=741 +14:14:07 global r=741 load 4.278 -> 4.280 +14:14:40 step 178960 loss=2.9307 lr=1.88e-05 tok=23.571B dt=7500 wait=315 +14:15:44 step 178980 loss=2.9058 lr=1.88e-05 tok=23.573B dt=7563 wait=315 +14:16:48 step 179000 loss=2.8958 lr=1.88e-05 tok=23.576B dt=7627 wait=315 +14:17:51 step 179020 loss=3.0432 lr=1.88e-05 tok=23.579B dt=7691 wait=316 +14:18:55 step 179040 loss=2.9841 lr=1.88e-05 tok=23.581B dt=7754 wait=316 +14:19:58 step 179060 loss=3.0406 lr=1.88e-05 tok=23.584B dt=7817 wait=316 +14:21:01 step 179080 loss=3.0293 lr=1.88e-05 tok=23.586B dt=7880 wait=316 +14:22:04 step 179100 loss=2.9104 lr=1.88e-05 tok=23.589B dt=7944 wait=316 +14:22:16 pushed ckpt_w178.pt,log_w178.txt +14:23:32 waited 76s for global r=742 +14:23:40 global r=742 load 4.285 -> 4.282 +14:24:44 step 179120 loss=2.9258 lr=1.88e-05 tok=23.592B dt=8103 wait=316 +14:25:46 step 179140 loss=3.0471 lr=1.88e-05 tok=23.594B dt=8166 wait=316 +14:26:49 step 179160 loss=2.9741 lr=1.88e-05 tok=23.597B dt=8229 wait=317 +14:27:53 step 179180 loss=2.8964 lr=1.88e-05 tok=23.600B dt=8292 wait=317 +14:28:56 step 179200 loss=2.8785 lr=1.87e-05 tok=23.602B dt=8355 wait=317 +14:29:59 step 179220 loss=2.9937 lr=1.87e-05 tok=23.605B dt=8418 wait=317 +14:31:02 step 179240 loss=2.8439 lr=1.87e-05 tok=23.607B dt=8481 wait=317 +14:31:49 pushed ckpt_w178.pt,log_w178.txt +14:33:20 waited 91s for global r=743 +14:33:27 global r=743 load 4.285 -> 4.281 +14:34:00 step 179260 loss=2.9263 lr=1.87e-05 tok=23.610B dt=8659 wait=317 +14:35:03 step 179280 loss=2.8128 lr=1.87e-05 tok=23.613B dt=8722 wait=317 +14:36:06 step 179300 loss=2.8516 lr=1.87e-05 tok=23.615B dt=8786 wait=318 +14:37:10 step 179320 loss=2.9648 lr=1.87e-05 tok=23.618B dt=8849 wait=318 +14:38:13 step 179340 loss=2.8679 lr=1.87e-05 tok=23.620B dt=8912 wait=318 +14:39:16 step 179360 loss=2.9633 lr=1.87e-05 tok=23.623B dt=8976 wait=318 +14:40:20 step 179380 loss=2.8845 lr=1.87e-05 tok=23.626B dt=9039 wait=318 +14:41:24 step 179400 loss=2.9021 lr=1.87e-05 tok=23.628B dt=9103 wait=318 +14:41:37 pushed ckpt_w178.pt,log_w178.txt +14:42:52 waited 76s for global r=744 +14:42:58 global r=744 load 4.284 -> 4.280 +14:44:03 step 179420 loss=2.9017 lr=1.87e-05 tok=23.631B dt=9263 wait=318 +14:45:07 step 179440 loss=2.9259 lr=1.87e-05 tok=23.634B dt=9326 wait=319 +14:46:11 step 179460 loss=2.9653 lr=1.87e-05 tok=23.636B dt=9390 wait=319 +14:47:14 step 179480 loss=2.8829 lr=1.86e-05 tok=23.639B dt=9454 wait=319 +14:48:18 step 179500 loss=2.8346 lr=1.86e-05 tok=23.641B dt=9517 wait=319 +14:49:22 step 179520 loss=2.9776 lr=1.86e-05 tok=23.644B dt=9581 wait=319 +14:50:26 step 179540 loss=2.8224 lr=1.86e-05 tok=23.647B dt=9645 wait=319 diff --git a/log_w179.txt b/log_w179.txt new file mode 100644 index 0000000000000000000000000000000000000000..d4e41c127a0e11567b4c122147cb06eeb790a2f0 --- /dev/null +++ b/log_w179.txt @@ -0,0 +1,687 @@ +launch +15:27:56 name=w179 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 126/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:28:03 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:28:03 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:28:12 params 1.08B +15:28:14 resumed step=179850 seen=23.69B round=746 +15:28:14 stream salt=179850 +15:30:17 blocks compiled +15:30:17 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +15:35:58 step 179851 loss=3.0008 lr=1.85e-05 tok=23.687B dt=341 wait=312 +15:36:14 step 179852 loss=3.0135 lr=1.85e-05 tok=23.688B dt=357 wait=312 +15:36:39 step 179860 loss=3.2962 lr=1.85e-05 tok=23.689B dt=382 wait=312 +15:36:55 global r=747 load 4.280 -> 4.283 +15:37:59 step 180020 loss=3.0258 lr=1.85e-05 tok=23.691B dt=462 wait=312 +15:39:03 step 180040 loss=2.9376 lr=1.85e-05 tok=23.694B dt=526 wait=312 +15:40:07 step 180060 loss=3.0234 lr=1.84e-05 tok=23.697B dt=590 wait=312 +15:41:10 step 180080 loss=3.0000 lr=1.84e-05 tok=23.699B dt=653 wait=313 +15:42:14 step 180100 loss=2.9738 lr=1.84e-05 tok=23.702B dt=717 wait=313 +15:42:54 global r=748 load 4.280 -> 4.280 +15:43:26 step 180160 loss=3.0487 lr=1.84e-05 tok=23.704B dt=789 wait=313 +15:44:30 step 180180 loss=3.0185 lr=1.84e-05 tok=23.707B dt=853 wait=313 +15:45:34 step 180200 loss=2.9325 lr=1.84e-05 tok=23.710B dt=917 wait=313 +15:46:38 step 180220 loss=3.0940 lr=1.84e-05 tok=23.712B dt=981 wait=313 +15:47:42 step 180240 loss=2.9182 lr=1.84e-05 tok=23.715B dt=1045 wait=313 +15:48:46 step 180260 loss=2.9180 lr=1.84e-05 tok=23.717B dt=1109 wait=314 +15:49:50 step 180280 loss=2.9668 lr=1.84e-05 tok=23.720B dt=1173 wait=314 +15:50:54 step 180300 loss=2.9983 lr=1.84e-05 tok=23.723B dt=1237 wait=314 +15:51:06 pushed ckpt_w179.pt,log_w179.txt +15:54:07 waited 181s for global r=749 +15:54:15 global r=749 load 4.284 -> 4.282 +15:55:21 step 180320 loss=2.9881 lr=1.84e-05 tok=23.725B dt=1504 wait=314 +15:56:25 step 180340 loss=3.0334 lr=1.84e-05 tok=23.728B dt=1568 wait=314 +15:57:28 step 180360 loss=2.9781 lr=1.84e-05 tok=23.731B dt=1631 wait=314 +15:58:32 step 180380 loss=2.7439 lr=1.83e-05 tok=23.733B dt=1695 wait=314 +15:59:37 step 180400 loss=2.7316 lr=1.83e-05 tok=23.736B dt=1760 wait=315 +16:00:40 step 180420 loss=3.0896 lr=1.83e-05 tok=23.738B dt=1823 wait=315 +16:01:43 step 180440 loss=3.0194 lr=1.83e-05 tok=23.741B dt=1886 wait=315 +16:02:28 pushed ckpt_w179.pt,log_w179.txt +16:06:02 step 180460 loss=3.0552 lr=1.83e-05 tok=23.744B dt=2145 wait=315 +16:06:11 global r=750 load 4.279 -> 4.283 +16:06:43 step 180460 loss=3.1054 lr=1.83e-05 tok=23.745B dt=2186 wait=315 +16:07:46 step 180480 loss=2.8948 lr=1.83e-05 tok=23.748B dt=2249 wait=315 +16:08:50 step 180500 loss=2.9475 lr=1.83e-05 tok=23.750B dt=2313 wait=315 +16:09:54 step 180520 loss=2.9639 lr=1.83e-05 tok=23.753B dt=2377 wait=316 +16:10:59 step 180540 loss=2.9447 lr=1.83e-05 tok=23.755B dt=2442 wait=316 +16:12:04 step 180560 loss=2.9875 lr=1.83e-05 tok=23.758B dt=2507 wait=316 +16:13:08 step 180580 loss=3.0370 lr=1.83e-05 tok=23.761B dt=2571 wait=316 +16:14:13 step 180600 loss=2.9036 lr=1.83e-05 tok=23.763B dt=2636 wait=316 +16:14:24 pushed ckpt_w179.pt,log_w179.txt +16:15:55 waited 91s for global r=751 +16:16:03 global r=751 load 4.285 -> 4.282 +16:17:09 step 180620 loss=2.9906 lr=1.83e-05 tok=23.766B dt=2812 wait=316 +16:18:12 step 180640 loss=3.1858 lr=1.83e-05 tok=23.769B dt=2875 wait=316 +16:19:17 step 180660 loss=3.0606 lr=1.83e-05 tok=23.771B dt=2940 wait=317 +16:20:21 step 180680 loss=3.0363 lr=1.82e-05 tok=23.774B dt=3004 wait=317 +16:21:25 step 180700 loss=2.9514 lr=1.82e-05 tok=23.776B dt=3068 wait=317 +16:22:29 step 180720 loss=3.1065 lr=1.82e-05 tok=23.779B dt=3132 wait=317 +16:23:34 step 180740 loss=2.8793 lr=1.82e-05 tok=23.782B dt=3197 wait=317 +16:24:23 pushed ckpt_w179.pt,log_w179.txt +launch +16:26:31 name=w179 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 126/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:26:36 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:26:36 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:26:45 params 1.08B +16:26:47 resumed step=180600 seen=23.80B round=751 +16:26:47 stream salt=180600 +16:28:42 blocks compiled +16:28:42 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:32:57 step 180601 loss=3.0187 lr=1.79e-05 tok=23.799B dt=255 wait=235 +16:33:04 step 180602 loss=3.0903 lr=1.79e-05 tok=23.799B dt=262 wait=235 +16:33:41 global r=752 load 4.274 -> 4.281 +16:34:14 step 180760 loss=3.2018 lr=1.78e-05 tok=23.801B dt=331 wait=235 +16:35:18 step 180780 loss=3.0607 lr=1.78e-05 tok=23.804B dt=396 wait=235 +16:36:21 step 180800 loss=3.0726 lr=1.77e-05 tok=23.807B dt=459 wait=235 +16:37:26 step 180820 loss=3.2026 lr=1.77e-05 tok=23.809B dt=524 wait=235 +16:38:30 step 180840 loss=3.1060 lr=1.77e-05 tok=23.812B dt=588 wait=235 +16:39:09 global r=753 load 4.284 -> 4.284 +16:40:12 step 180920 loss=3.0222 lr=1.76e-05 tok=23.816B dt=690 wait=235 +16:41:16 step 180940 loss=3.1073 lr=1.76e-05 tok=23.818B dt=754 wait=236 +16:42:20 step 180960 loss=3.0992 lr=1.76e-05 tok=23.821B dt=818 wait=236 +16:43:24 step 180980 loss=2.9667 lr=1.76e-05 tok=23.824B dt=882 wait=236 +16:44:29 step 181000 loss=3.0414 lr=1.75e-05 tok=23.826B dt=947 wait=236 +16:45:33 step 181020 loss=3.1143 lr=1.75e-05 tok=23.829B dt=1011 wait=236 +16:46:37 step 181040 loss=2.9476 lr=1.75e-05 tok=23.832B dt=1075 wait=236 +16:47:20 pushed ckpt_w179.pt,log_w179.txt +16:50:56 step 181060 loss=3.1226 lr=1.75e-05 tok=23.834B dt=1334 wait=236 +16:51:07 global r=754 load 4.282 -> 4.281 +16:51:39 step 181060 loss=3.1061 lr=1.75e-05 tok=23.835B dt=1377 wait=237 +16:52:42 step 181080 loss=3.0747 lr=1.75e-05 tok=23.838B dt=1440 wait=237 +16:53:46 step 181100 loss=3.0926 lr=1.75e-05 tok=23.841B dt=1504 wait=237 +16:54:50 step 181120 loss=3.1723 lr=1.74e-05 tok=23.843B dt=1568 wait=237 +16:55:54 step 181140 loss=3.1425 lr=1.74e-05 tok=23.846B dt=1632 wait=237 +16:56:58 step 181160 loss=3.1285 lr=1.74e-05 tok=23.849B dt=1696 wait=237 +16:58:01 step 181180 loss=3.1735 lr=1.74e-05 tok=23.851B dt=1759 wait=237 +16:59:06 step 181200 loss=3.0942 lr=1.74e-05 tok=23.854B dt=1824 wait=237 +16:59:17 pushed ckpt_w179.pt,log_w179.txt +17:01:03 waited 106s for global r=755 +17:01:11 global r=755 load 4.280 -> 4.280 +17:02:16 step 181220 loss=3.2042 lr=1.73e-05 tok=23.856B dt=2014 wait=238 +17:03:20 step 181240 loss=3.2526 lr=1.73e-05 tok=23.859B dt=2078 wait=238 +17:04:24 step 181260 loss=3.1677 lr=1.73e-05 tok=23.862B dt=2142 wait=238 +17:05:29 step 181280 loss=3.0264 lr=1.73e-05 tok=23.864B dt=2207 wait=238 +17:06:33 step 181300 loss=3.0317 lr=1.73e-05 tok=23.867B dt=2271 wait=238 +17:07:38 step 181320 loss=3.0932 lr=1.73e-05 tok=23.870B dt=2336 wait=238 +17:08:42 step 181340 loss=2.9766 lr=1.72e-05 tok=23.872B dt=2400 wait=239 +17:09:25 pushed ckpt_w179.pt,log_w179.txt +17:10:56 waited 91s for global r=756 +17:11:02 global r=756 load 4.274 -> 4.281 +17:11:36 step 181360 loss=3.0401 lr=1.72e-05 tok=23.875B dt=2574 wait=239 +17:12:39 step 181380 loss=3.0105 lr=1.72e-05 tok=23.877B dt=2637 wait=239 +17:13:42 step 181400 loss=2.9924 lr=1.72e-05 tok=23.880B dt=2700 wait=239 +17:14:47 step 181420 loss=2.9930 lr=1.72e-05 tok=23.883B dt=2765 wait=239 +17:15:50 step 181440 loss=2.9704 lr=1.71e-05 tok=23.885B dt=2828 wait=239 +17:16:55 step 181460 loss=2.9190 lr=1.71e-05 tok=23.888B dt=2893 wait=239 +17:17:59 step 181480 loss=3.0343 lr=1.71e-05 tok=23.890B dt=2957 wait=239 +17:19:03 step 181500 loss=2.9524 lr=1.71e-05 tok=23.893B dt=3021 wait=240 +17:19:15 pushed ckpt_w179.pt,log_w179.txt +17:22:56 global r=757 load 4.284 -> 4.282 +17:23:59 step 181520 loss=2.9711 lr=1.71e-05 tok=23.897B dt=3317 wait=240 +17:25:02 step 181540 loss=3.0359 lr=1.70e-05 tok=23.900B dt=3380 wait=240 +17:26:06 step 181560 loss=2.8813 lr=1.70e-05 tok=23.902B dt=3444 wait=240 +17:27:11 step 181580 loss=2.9516 lr=1.70e-05 tok=23.905B dt=3509 wait=240 +17:28:16 step 181600 loss=3.0691 lr=1.70e-05 tok=23.908B dt=3574 wait=241 +17:29:27 step 181620 loss=3.0134 lr=1.70e-05 tok=23.910B dt=3645 wait=241 +17:30:44 step 181640 loss=3.0718 lr=1.70e-05 tok=23.913B dt=3722 wait=241 +17:31:28 pushed ckpt_w179.pt,log_w179.txt +17:32:59 waited 91s for global r=758 +17:33:05 global r=758 load 4.283 -> 4.283 +17:33:38 step 181660 loss=3.1457 lr=1.69e-05 tok=23.915B dt=3896 wait=241 +17:34:43 step 181680 loss=3.0016 lr=1.69e-05 tok=23.918B dt=3961 wait=241 +17:36:08 step 181700 loss=2.9746 lr=1.69e-05 tok=23.921B dt=4045 wait=241 +17:37:13 step 181720 loss=3.0141 lr=1.69e-05 tok=23.923B dt=4111 wait=241 +17:38:19 step 181740 loss=2.9137 lr=1.69e-05 tok=23.926B dt=4177 wait=242 +17:39:24 step 181760 loss=3.0216 lr=1.68e-05 tok=23.929B dt=4242 wait=242 +17:40:28 step 181780 loss=3.0958 lr=1.68e-05 tok=23.931B dt=4306 wait=242 +17:41:33 step 181800 loss=3.0286 lr=1.68e-05 tok=23.934B dt=4371 wait=242 +17:41:48 pushed ckpt_w179.pt,log_w179.txt +17:42:33 waited 45s for global r=759 +17:42:43 global r=759 load 4.287 -> 4.283 +17:43:49 step 181820 loss=3.1520 lr=1.68e-05 tok=23.936B dt=4507 wait=242 +17:45:02 step 181840 loss=3.1201 lr=1.68e-05 tok=23.939B dt=4580 wait=242 +17:46:19 step 181860 loss=3.1362 lr=1.68e-05 tok=23.942B dt=4657 wait=242 +17:47:23 step 181880 loss=3.0400 lr=1.67e-05 tok=23.944B dt=4721 wait=243 +17:48:28 step 181900 loss=3.0451 lr=1.67e-05 tok=23.947B dt=4786 wait=243 +17:49:33 step 181920 loss=3.0983 lr=1.67e-05 tok=23.949B dt=4851 wait=243 +17:50:37 step 181940 loss=3.0937 lr=1.67e-05 tok=23.952B dt=4915 wait=243 +17:51:22 pushed ckpt_w179.pt,log_w179.txt +17:52:52 waited 91s for global r=760 +17:52:57 global r=760 load 4.280 -> 4.281 +17:53:31 step 181960 loss=3.0918 lr=1.67e-05 tok=23.955B dt=5089 wait=243 +17:54:43 step 181980 loss=2.9947 lr=1.66e-05 tok=23.957B dt=5161 wait=243 +17:55:59 step 182000 loss=3.0261 lr=1.66e-05 tok=23.960B dt=5237 wait=243 +17:57:02 step 182020 loss=3.0991 lr=1.66e-05 tok=23.963B dt=5300 wait=244 +17:58:06 step 182040 loss=2.9423 lr=1.66e-05 tok=23.965B dt=5364 wait=244 +17:59:10 step 182060 loss=3.0509 lr=1.66e-05 tok=23.968B dt=5428 wait=244 +18:00:13 step 182080 loss=3.0954 lr=1.65e-05 tok=23.970B dt=5491 wait=244 +18:01:17 step 182100 loss=3.1183 lr=1.65e-05 tok=23.973B dt=5555 wait=244 +18:01:28 pushed ckpt_w179.pt,log_w179.txt +18:02:43 waited 76s for global r=761 +18:02:50 global r=761 load 4.281 -> 4.281 +18:04:01 step 182120 loss=3.0780 lr=1.65e-05 tok=23.976B dt=5719 wait=244 +18:05:18 step 182140 loss=3.0067 lr=1.65e-05 tok=23.978B dt=5796 wait=244 +18:06:21 step 182160 loss=3.0462 lr=1.65e-05 tok=23.981B dt=5859 wait=245 +18:07:25 step 182180 loss=2.9394 lr=1.65e-05 tok=23.984B dt=5923 wait=245 +18:08:28 step 182200 loss=2.9907 lr=1.64e-05 tok=23.986B dt=5986 wait=245 +18:09:31 step 182220 loss=3.0275 lr=1.64e-05 tok=23.989B dt=6049 wait=245 +18:10:34 step 182240 loss=2.8358 lr=1.64e-05 tok=23.991B dt=6112 wait=245 +18:11:17 pushed ckpt_w179.pt,log_w179.txt +18:12:33 waited 76s for global r=762 +18:12:41 global r=762 load 4.282 -> 4.281 +18:13:14 step 182260 loss=3.0068 lr=1.64e-05 tok=23.994B dt=6272 wait=245 +18:14:35 step 182280 loss=2.9512 lr=1.64e-05 tok=23.997B dt=6353 wait=246 +18:15:40 step 182300 loss=2.9546 lr=1.63e-05 tok=23.999B dt=6418 wait=246 +18:16:43 step 182320 loss=3.0663 lr=1.63e-05 tok=24.002B dt=6481 wait=246 +18:17:45 step 182340 loss=2.9006 lr=1.63e-05 tok=24.005B dt=6543 wait=246 +18:18:49 step 182360 loss=3.0138 lr=1.63e-05 tok=24.007B dt=6607 wait=246 +18:19:51 step 182380 loss=3.0000 lr=1.63e-05 tok=24.010B dt=6669 wait=246 +18:20:55 step 182400 loss=2.9857 lr=1.63e-05 tok=24.012B dt=6733 wait=246 +18:21:06 pushed ckpt_w179.pt,log_w179.txt +18:24:07 waited 181s for global r=763 +18:24:14 global r=763 load 4.281 -> 4.280 +18:25:26 step 182420 loss=2.9916 lr=1.62e-05 tok=24.015B dt=7004 wait=247 +18:26:42 step 182440 loss=2.9501 lr=1.62e-05 tok=24.018B dt=7080 wait=247 +18:27:45 step 182460 loss=3.0948 lr=1.62e-05 tok=24.020B dt=7143 wait=247 +18:28:49 step 182480 loss=2.9670 lr=1.62e-05 tok=24.023B dt=7207 wait=247 +18:29:53 step 182500 loss=3.0370 lr=1.62e-05 tok=24.025B dt=7271 wait=247 +18:30:57 step 182520 loss=3.0678 lr=1.61e-05 tok=24.028B dt=7335 wait=247 +18:32:00 step 182540 loss=3.0032 lr=1.61e-05 tok=24.031B dt=7398 wait=247 +18:32:46 pushed ckpt_w179.pt,log_w179.txt +18:35:47 waited 181s for global r=764 +18:35:54 global r=764 load 4.285 -> 4.280 +18:36:27 step 182560 loss=2.9425 lr=1.61e-05 tok=24.033B dt=7665 wait=247 +18:37:45 step 182580 loss=2.8401 lr=1.61e-05 tok=24.036B dt=7743 wait=248 +18:38:55 step 182600 loss=2.8630 lr=1.61e-05 tok=24.039B dt=7813 wait=248 +18:40:00 step 182620 loss=2.6696 lr=1.61e-05 tok=24.041B dt=7878 wait=248 +18:41:03 step 182640 loss=2.8989 lr=1.60e-05 tok=24.044B dt=7941 wait=248 +18:42:06 step 182660 loss=3.0122 lr=1.60e-05 tok=24.046B dt=8004 wait=248 +18:43:10 step 182680 loss=2.7978 lr=1.60e-05 tok=24.049B dt=8068 wait=248 +18:44:13 step 182700 loss=2.9556 lr=1.60e-05 tok=24.052B dt=8131 wait=249 +18:44:24 pushed ckpt_w179.pt,log_w179.txt +18:45:25 waited 61s for global r=765 +18:45:39 global r=765 load 4.281 -> 4.280 +18:46:51 step 182720 loss=2.8256 lr=1.60e-05 tok=24.054B dt=8289 wait=249 +18:48:07 step 182740 loss=2.8597 lr=1.59e-05 tok=24.057B dt=8365 wait=249 +18:49:10 step 182760 loss=2.9327 lr=1.59e-05 tok=24.060B dt=8428 wait=249 +18:50:12 step 182780 loss=2.8543 lr=1.59e-05 tok=24.062B dt=8490 wait=249 +18:51:15 step 182800 loss=2.8657 lr=1.59e-05 tok=24.065B dt=8553 wait=249 +18:52:18 step 182820 loss=2.9336 lr=1.59e-05 tok=24.067B dt=8616 wait=249 +18:53:21 step 182840 loss=2.7795 lr=1.58e-05 tok=24.070B dt=8679 wait=250 +18:54:05 pushed ckpt_w179.pt,log_w179.txt +18:54:51 waited 45s for global r=766 +18:54:56 global r=766 load 4.285 -> 4.281 +18:55:29 step 182860 loss=2.9795 lr=1.58e-05 tok=24.073B dt=8807 wait=250 +18:56:40 step 182880 loss=2.8162 lr=1.58e-05 tok=24.075B dt=8878 wait=250 +18:57:56 step 182900 loss=2.9017 lr=1.58e-05 tok=24.078B dt=8954 wait=250 +18:58:59 step 182920 loss=2.9062 lr=1.58e-05 tok=24.081B dt=9017 wait=250 +19:00:03 step 182940 loss=2.8045 lr=1.58e-05 tok=24.083B dt=9081 wait=250 +19:01:07 step 182960 loss=2.8496 lr=1.57e-05 tok=24.086B dt=9145 wait=250 +19:02:10 step 182980 loss=2.8094 lr=1.57e-05 tok=24.088B dt=9208 wait=251 +19:03:13 step 183000 loss=2.8948 lr=1.57e-05 tok=24.091B dt=9271 wait=251 +19:03:27 pushed ckpt_w179.pt,log_w179.txt +19:04:28 waited 61s for global r=767 +19:04:39 global r=767 load 4.280 -> 4.283 +19:05:44 step 183020 loss=2.8310 lr=1.57e-05 tok=24.094B dt=9422 wait=251 +19:07:06 step 183040 loss=2.9054 lr=1.57e-05 tok=24.096B dt=9504 wait=251 +19:08:11 step 183060 loss=2.8645 lr=1.56e-05 tok=24.099B dt=9569 wait=251 +19:09:15 step 183080 loss=2.8568 lr=1.56e-05 tok=24.102B dt=9633 wait=251 +19:10:19 step 183100 loss=2.9292 lr=1.56e-05 tok=24.104B dt=9697 wait=252 +19:11:23 step 183120 loss=3.0041 lr=1.56e-05 tok=24.107B dt=9761 wait=252 +19:12:27 step 183140 loss=2.8896 lr=1.56e-05 tok=24.109B dt=9825 wait=252 +19:13:12 pushed ckpt_w179.pt,log_w179.txt +19:13:57 waited 45s for global r=768 +19:14:04 global r=768 load 4.282 -> 4.284 +19:14:37 step 183160 loss=3.0459 lr=1.56e-05 tok=24.112B dt=9954 wait=252 +19:15:47 step 183180 loss=2.9024 lr=1.55e-05 tok=24.115B dt=10025 wait=252 +19:17:02 step 183200 loss=2.9861 lr=1.55e-05 tok=24.117B dt=10100 wait=252 +19:18:06 step 183220 loss=3.0248 lr=1.55e-05 tok=24.120B dt=10164 wait=252 +19:19:11 step 183240 loss=2.7133 lr=1.55e-05 tok=24.122B dt=10229 wait=253 +19:20:14 step 183260 loss=3.0452 lr=1.55e-05 tok=24.125B dt=10292 wait=253 +19:21:18 step 183280 loss=2.9577 lr=1.54e-05 tok=24.128B dt=10356 wait=253 +19:22:22 step 183300 loss=2.9947 lr=1.54e-05 tok=24.130B dt=10420 wait=253 +19:22:33 pushed ckpt_w179.pt,log_w179.txt +19:23:34 waited 61s for global r=769 +19:23:42 global r=769 load 4.280 -> 4.282 +19:24:47 step 183320 loss=2.9352 lr=1.54e-05 tok=24.133B dt=10565 wait=253 +19:26:07 step 183340 loss=2.9943 lr=1.54e-05 tok=24.136B dt=10645 wait=253 +19:27:14 step 183360 loss=2.9959 lr=1.54e-05 tok=24.138B dt=10712 wait=253 +19:28:17 step 183380 loss=2.9755 lr=1.53e-05 tok=24.141B dt=10775 wait=254 +19:29:21 step 183400 loss=2.9238 lr=1.53e-05 tok=24.143B dt=10839 wait=254 +19:30:24 step 183420 loss=3.0681 lr=1.53e-05 tok=24.146B dt=10902 wait=254 +19:31:27 step 183440 loss=2.9675 lr=1.53e-05 tok=24.149B dt=10965 wait=254 +19:32:13 pushed ckpt_w179.pt,log_w179.txt +19:32:58 waited 45s for global r=770 +19:33:08 global r=770 load 4.281 -> 4.281 +19:33:41 step 183460 loss=3.0135 lr=1.53e-05 tok=24.151B dt=11099 wait=254 +19:34:45 step 183480 loss=2.9335 lr=1.53e-05 tok=24.154B dt=11163 wait=254 +19:36:08 step 183500 loss=2.8949 lr=1.52e-05 tok=24.157B dt=11246 wait=254 +19:37:11 step 183520 loss=2.9675 lr=1.52e-05 tok=24.159B dt=11309 wait=255 +19:38:14 step 183540 loss=2.8524 lr=1.52e-05 tok=24.162B dt=11372 wait=255 +19:39:18 step 183560 loss=2.9958 lr=1.52e-05 tok=24.164B dt=11436 wait=255 +19:40:22 step 183580 loss=2.9309 lr=1.52e-05 tok=24.167B dt=11500 wait=255 +19:41:25 step 183600 loss=2.9812 lr=1.51e-05 tok=24.170B dt=11563 wait=255 +19:41:41 pushed ckpt_w179.pt,log_w179.txt +19:44:27 waited 166s for global r=771 +19:44:33 global r=771 load 4.284 -> 4.282 +19:45:38 step 183620 loss=3.0346 lr=1.51e-05 tok=24.172B dt=11815 wait=255 +19:46:49 step 183640 loss=2.9939 lr=1.51e-05 tok=24.175B dt=11887 wait=255 +19:48:05 step 183660 loss=2.9702 lr=1.51e-05 tok=24.178B dt=11963 wait=256 +19:49:08 step 183680 loss=2.9510 lr=1.51e-05 tok=24.180B dt=12026 wait=256 +19:50:12 step 183700 loss=2.9668 lr=1.51e-05 tok=24.183B dt=12090 wait=256 +19:51:15 step 183720 loss=3.1235 lr=1.50e-05 tok=24.185B dt=12153 wait=256 +19:52:19 step 183740 loss=2.7137 lr=1.50e-05 tok=24.188B dt=12217 wait=256 +19:53:02 pushed ckpt_w179.pt,log_w179.txt +19:56:37 step 183760 loss=3.0922 lr=1.50e-05 tok=24.191B dt=12475 wait=256 +19:56:44 global r=772 load 4.281 -> 4.280 +19:57:16 step 183760 loss=3.0841 lr=1.50e-05 tok=24.192B dt=12514 wait=256 +19:58:27 step 183780 loss=3.0060 lr=1.50e-05 tok=24.195B dt=12585 wait=257 +19:59:42 step 183800 loss=3.1018 lr=1.50e-05 tok=24.197B dt=12660 wait=257 +20:00:46 step 183820 loss=3.0535 lr=1.49e-05 tok=24.200B dt=12724 wait=257 +20:01:49 step 183840 loss=2.9518 lr=1.49e-05 tok=24.202B dt=12787 wait=257 +20:02:52 step 183860 loss=2.7752 lr=1.49e-05 tok=24.205B dt=12850 wait=257 +20:03:56 step 183880 loss=2.9694 lr=1.49e-05 tok=24.208B dt=12914 wait=257 +20:04:59 step 183900 loss=3.0093 lr=1.49e-05 tok=24.210B dt=12977 wait=257 +20:05:10 pushed ckpt_w179.pt,log_w179.txt +20:07:57 waited 166s for global r=773 +20:08:01 global r=773 load 4.279 -> 4.282 +20:09:06 step 183920 loss=2.9002 lr=1.49e-05 tok=24.213B dt=13224 wait=258 +20:10:17 step 183940 loss=3.0013 lr=1.48e-05 tok=24.216B dt=13295 wait=258 +20:11:34 step 183960 loss=2.8929 lr=1.48e-05 tok=24.218B dt=13372 wait=258 +20:12:37 step 183980 loss=3.0610 lr=1.48e-05 tok=24.221B dt=13435 wait=258 +20:13:42 step 184000 loss=2.6085 lr=1.48e-05 tok=24.223B dt=13500 wait=258 +20:14:46 step 184020 loss=3.0870 lr=1.48e-05 tok=24.226B dt=13564 wait=258 +20:15:49 step 184040 loss=2.9689 lr=1.47e-05 tok=24.229B dt=13627 wait=258 +20:16:38 pushed ckpt_w179.pt,log_w179.txt +20:18:54 waited 136s for global r=774 +20:19:00 global r=774 load 4.283 -> 4.282 +20:19:41 step 184060 loss=2.9892 lr=1.47e-05 tok=24.231B dt=13859 wait=259 +20:20:57 step 184080 loss=2.9639 lr=1.47e-05 tok=24.234B dt=13935 wait=259 +20:22:00 step 184100 loss=3.0688 lr=1.47e-05 tok=24.237B dt=13998 wait=259 +20:23:03 step 184120 loss=3.0427 lr=1.47e-05 tok=24.239B dt=14061 wait=259 +20:24:06 step 184140 loss=2.9007 lr=1.46e-05 tok=24.242B dt=14124 wait=259 +20:25:09 step 184160 loss=2.9897 lr=1.46e-05 tok=24.244B dt=14187 wait=259 +20:26:11 step 184180 loss=2.9543 lr=1.46e-05 tok=24.247B dt=14249 wait=260 +20:27:14 step 184200 loss=2.8903 lr=1.46e-05 tok=24.250B dt=14312 wait=260 +20:27:27 pushed ckpt_w179.pt,log_w179.txt +20:28:27 waited 61s for global r=775 +20:28:34 global r=775 load 4.282 -> 4.281 +20:29:52 step 184220 loss=2.9582 lr=1.46e-05 tok=24.252B dt=14470 wait=260 +20:31:00 step 184240 loss=2.9815 lr=1.46e-05 tok=24.255B dt=14538 wait=260 +20:32:03 step 184260 loss=2.9455 lr=1.45e-05 tok=24.257B dt=14601 wait=260 +20:33:06 step 184280 loss=2.9246 lr=1.45e-05 tok=24.260B dt=14664 wait=260 +20:34:09 step 184300 loss=2.9098 lr=1.45e-05 tok=24.263B dt=14727 wait=261 +20:35:12 step 184320 loss=3.0868 lr=1.45e-05 tok=24.265B dt=14789 wait=261 +20:36:15 step 184340 loss=2.8131 lr=1.45e-05 tok=24.268B dt=14853 wait=261 +20:36:57 pushed ckpt_w179.pt,log_w179.txt +20:38:13 waited 76s for global r=776 +20:38:21 global r=776 load 4.280 -> 4.281 +20:39:04 step 184360 loss=2.9477 lr=1.44e-05 tok=24.271B dt=15022 wait=261 +20:40:25 step 184380 loss=2.8634 lr=1.44e-05 tok=24.273B dt=15103 wait=261 +20:41:30 step 184400 loss=2.8314 lr=1.44e-05 tok=24.276B dt=15168 wait=261 +20:42:35 step 184420 loss=2.7887 lr=1.44e-05 tok=24.278B dt=15233 wait=261 +20:43:40 step 184440 loss=2.8309 lr=1.44e-05 tok=24.281B dt=15298 wait=262 +20:44:45 step 184460 loss=2.8754 lr=1.44e-05 tok=24.284B dt=15363 wait=262 +20:45:50 step 184480 loss=2.8564 lr=1.43e-05 tok=24.286B dt=15428 wait=262 +20:46:56 step 184500 loss=2.8786 lr=1.43e-05 tok=24.289B dt=15494 wait=262 +20:47:07 pushed ckpt_w179.pt,log_w179.txt +20:48:07 waited 61s for global r=777 +20:48:13 global r=777 load 4.292 -> 4.280 +20:49:22 step 184520 loss=2.7996 lr=1.43e-05 tok=24.292B dt=15640 wait=262 +20:50:27 step 184540 loss=2.9346 lr=1.43e-05 tok=24.294B dt=15705 wait=262 +20:51:45 step 184560 loss=2.8552 lr=1.43e-05 tok=24.297B dt=15783 wait=262 +20:52:50 step 184580 loss=2.9132 lr=1.42e-05 tok=24.299B dt=15848 wait=263 +20:53:55 step 184600 loss=2.9247 lr=1.42e-05 tok=24.302B dt=15913 wait=263 +20:55:01 step 184620 loss=2.9340 lr=1.42e-05 tok=24.305B dt=15979 wait=263 +20:56:08 step 184640 loss=2.7440 lr=1.42e-05 tok=24.307B dt=16046 wait=263 +20:56:53 pushed ckpt_w179.pt,log_w179.txt +20:57:54 waited 61s for global r=778 +20:58:00 global r=778 load 4.291 -> 4.280 +20:58:34 step 184660 loss=2.9395 lr=1.42e-05 tok=24.310B dt=16192 wait=263 +20:59:43 step 184680 loss=2.8672 lr=1.41e-05 tok=24.313B dt=16261 wait=263 +21:01:04 step 184700 loss=2.8630 lr=1.41e-05 tok=24.315B dt=16342 wait=264 +21:02:08 step 184720 loss=3.0113 lr=1.41e-05 tok=24.318B dt=16406 wait=264 +21:03:13 step 184740 loss=2.8574 lr=1.41e-05 tok=24.320B dt=16471 wait=264 +21:04:19 step 184760 loss=2.9655 lr=1.41e-05 tok=24.323B dt=16537 wait=264 +21:05:24 step 184780 loss=2.7491 lr=1.41e-05 tok=24.326B dt=16602 wait=264 +21:06:29 step 184800 loss=2.8764 lr=1.40e-05 tok=24.328B dt=16667 wait=264 +21:06:44 pushed ckpt_w179.pt,log_w179.txt +21:07:44 waited 61s for global r=779 +21:07:49 global r=779 load 4.295 -> 4.280 +21:08:56 step 184820 loss=2.7963 lr=1.40e-05 tok=24.331B dt=16814 wait=264 +21:10:18 step 184840 loss=2.7884 lr=1.40e-05 tok=24.334B dt=16896 wait=265 +21:11:27 step 184860 loss=2.7686 lr=1.40e-05 tok=24.336B dt=16965 wait=265 +21:12:32 step 184880 loss=2.7198 lr=1.40e-05 tok=24.339B dt=17030 wait=265 +21:13:37 step 184900 loss=2.8143 lr=1.39e-05 tok=24.341B dt=17095 wait=265 +21:14:43 step 184920 loss=2.8351 lr=1.39e-05 tok=24.344B dt=17161 wait=265 +21:15:47 step 184940 loss=2.7852 lr=1.39e-05 tok=24.347B dt=17225 wait=265 +21:16:33 pushed ckpt_w179.pt,log_w179.txt +21:17:04 waited 30s for global r=780 +21:17:10 global r=780 load 4.286 -> 4.280 +21:17:44 step 184960 loss=2.8873 lr=1.39e-05 tok=24.349B dt=17342 wait=265 +21:18:50 step 184980 loss=2.8608 lr=1.39e-05 tok=24.352B dt=17408 wait=266 +21:20:12 step 185000 loss=2.9071 lr=1.39e-05 tok=24.354B dt=17490 wait=266 +21:21:16 step 185020 loss=2.9520 lr=1.38e-05 tok=24.357B dt=17554 wait=266 +21:22:20 step 185040 loss=2.8527 lr=1.38e-05 tok=24.360B dt=17617 wait=266 +21:23:23 step 185060 loss=2.9621 lr=1.38e-05 tok=24.362B dt=17681 wait=266 +21:24:26 step 185080 loss=2.9721 lr=1.38e-05 tok=24.365B dt=17744 wait=266 +21:25:30 step 185100 loss=2.9767 lr=1.38e-05 tok=24.368B dt=17808 wait=266 +21:25:49 pushed ckpt_w179.pt,log_w179.txt +21:26:49 waited 61s for global r=781 +21:26:58 global r=781 load 4.281 -> 4.282 +21:28:02 step 185120 loss=2.9132 lr=1.37e-05 tok=24.370B dt=17960 wait=266 +21:29:19 step 185140 loss=2.9621 lr=1.37e-05 tok=24.373B dt=18037 wait=267 +21:30:28 step 185160 loss=2.9241 lr=1.37e-05 tok=24.375B dt=18106 wait=267 +21:31:31 step 185180 loss=2.9293 lr=1.37e-05 tok=24.378B dt=18169 wait=267 +21:32:35 step 185200 loss=2.9075 lr=1.37e-05 tok=24.381B dt=18233 wait=267 +21:33:38 step 185220 loss=3.0170 lr=1.36e-05 tok=24.383B dt=18296 wait=267 +21:34:41 step 185240 loss=2.9121 lr=1.36e-05 tok=24.386B dt=18358 wait=267 +21:35:23 pushed ckpt_w179.pt,log_w179.txt +21:36:24 waited 61s for global r=782 +21:36:32 global r=782 load 4.281 -> 4.280 +21:37:04 step 185260 loss=3.0603 lr=1.36e-05 tok=24.389B dt=18502 wait=268 +21:38:13 step 185280 loss=2.9500 lr=1.36e-05 tok=24.391B dt=18571 wait=268 +21:39:31 step 185300 loss=2.9226 lr=1.36e-05 tok=24.394B dt=18649 wait=268 +21:40:35 step 185320 loss=3.0302 lr=1.36e-05 tok=24.396B dt=18713 wait=268 +21:41:39 step 185340 loss=2.9772 lr=1.35e-05 tok=24.399B dt=18777 wait=268 +21:42:42 step 185360 loss=3.0476 lr=1.35e-05 tok=24.402B dt=18840 wait=268 +21:43:45 step 185380 loss=2.9447 lr=1.35e-05 tok=24.404B dt=18903 wait=268 +21:44:49 step 185400 loss=2.9356 lr=1.35e-05 tok=24.407B dt=18967 wait=269 +21:45:02 pushed ckpt_w179.pt,log_w179.txt +21:46:03 waited 61s for global r=783 +21:46:12 global r=783 load 4.281 -> 4.279 +21:47:17 step 185420 loss=2.9605 lr=1.35e-05 tok=24.410B dt=19115 wait=269 +21:48:35 step 185440 loss=2.9467 lr=1.34e-05 tok=24.412B dt=19193 wait=269 +21:49:43 step 185460 loss=2.9407 lr=1.34e-05 tok=24.415B dt=19261 wait=269 +21:50:47 step 185480 loss=2.9235 lr=1.34e-05 tok=24.417B dt=19325 wait=269 +21:51:50 step 185500 loss=2.9328 lr=1.34e-05 tok=24.420B dt=19388 wait=269 +21:52:53 step 185520 loss=3.0072 lr=1.34e-05 tok=24.423B dt=19451 wait=269 +21:53:56 step 185540 loss=2.8727 lr=1.34e-05 tok=24.425B dt=19514 wait=270 +21:54:40 pushed ckpt_w179.pt,log_w179.txt +21:55:41 waited 61s for global r=784 +21:55:46 global r=784 load 4.281 -> 4.280 +21:56:19 step 185560 loss=3.0265 lr=1.33e-05 tok=24.428B dt=19657 wait=270 +21:57:29 step 185580 loss=2.9082 lr=1.33e-05 tok=24.431B dt=19727 wait=270 +21:58:47 step 185600 loss=2.9035 lr=1.33e-05 tok=24.433B dt=19805 wait=270 +21:59:50 step 185620 loss=3.0736 lr=1.33e-05 tok=24.436B dt=19868 wait=270 +22:00:54 step 185640 loss=2.8704 lr=1.33e-05 tok=24.438B dt=19932 wait=270 +22:01:57 step 185660 loss=2.9957 lr=1.32e-05 tok=24.441B dt=19995 wait=271 +22:03:00 step 185680 loss=2.8829 lr=1.32e-05 tok=24.444B dt=20058 wait=271 +22:04:04 step 185700 loss=2.8955 lr=1.32e-05 tok=24.446B dt=20122 wait=271 +22:04:15 pushed ckpt_w179.pt,log_w179.txt +22:05:15 waited 61s for global r=785 +22:05:31 global r=785 load 4.281 -> 4.279 +22:06:36 step 185720 loss=2.9481 lr=1.32e-05 tok=24.449B dt=20274 wait=271 +22:08:00 step 185740 loss=2.9784 lr=1.32e-05 tok=24.451B dt=20358 wait=271 +22:09:07 step 185760 loss=2.9591 lr=1.32e-05 tok=24.454B dt=20425 wait=271 +22:10:11 step 185780 loss=2.8819 lr=1.31e-05 tok=24.457B dt=20489 wait=271 +22:11:15 step 185800 loss=2.9236 lr=1.31e-05 tok=24.459B dt=20553 wait=272 +22:12:19 step 185820 loss=2.9752 lr=1.31e-05 tok=24.462B dt=20617 wait=272 +22:13:23 step 185840 loss=2.9638 lr=1.31e-05 tok=24.465B dt=20681 wait=272 +22:14:07 pushed ckpt_w179.pt,log_w179.txt +22:14:52 waited 45s for global r=786 +22:14:59 global r=786 load 4.275 -> 4.279 +22:15:32 step 185860 loss=3.0592 lr=1.31e-05 tok=24.467B dt=20810 wait=272 +22:16:41 step 185880 loss=3.0067 lr=1.30e-05 tok=24.470B dt=20879 wait=272 +22:18:00 step 185900 loss=2.9591 lr=1.30e-05 tok=24.472B dt=20958 wait=272 +22:19:04 step 185920 loss=3.1149 lr=1.30e-05 tok=24.475B dt=21022 wait=272 +22:20:08 step 185940 loss=2.9442 lr=1.30e-05 tok=24.478B dt=21085 wait=273 +22:21:11 step 185960 loss=3.0083 lr=1.30e-05 tok=24.480B dt=21149 wait=273 +22:22:15 step 185980 loss=2.9944 lr=1.29e-05 tok=24.483B dt=21213 wait=273 +22:23:19 step 186000 loss=3.0310 lr=1.29e-05 tok=24.486B dt=21277 wait=273 +22:23:31 pushed ckpt_w179.pt,log_w179.txt +22:24:17 waited 45s for global r=787 +22:24:22 global r=787 load 4.285 -> 4.280 +22:25:26 step 186020 loss=2.9894 lr=1.29e-05 tok=24.488B dt=21404 wait=273 +22:26:39 step 186040 loss=2.9462 lr=1.29e-05 tok=24.491B dt=21477 wait=273 +22:27:54 step 186060 loss=3.0265 lr=1.29e-05 tok=24.493B dt=21552 wait=273 +22:28:59 step 186080 loss=2.9385 lr=1.29e-05 tok=24.496B dt=21617 wait=274 +22:30:02 step 186100 loss=2.9712 lr=1.28e-05 tok=24.499B dt=21680 wait=274 +22:31:06 step 186120 loss=3.0276 lr=1.28e-05 tok=24.501B dt=21744 wait=274 +22:32:11 step 186140 loss=2.8180 lr=1.28e-05 tok=24.504B dt=21809 wait=274 +22:32:58 pushed ckpt_w179.pt,log_w179.txt +22:36:36 step 186160 loss=3.1878 lr=1.28e-05 tok=24.507B dt=22074 wait=274 +22:37:41 step 186180 loss=2.9522 lr=1.28e-05 tok=24.509B dt=22139 wait=274 +22:38:44 step 186200 loss=2.8876 lr=1.27e-05 tok=24.512B dt=22202 wait=275 +22:39:49 step 186220 loss=2.8463 lr=1.27e-05 tok=24.514B dt=22267 wait=275 +22:40:26 global r=788 load 4.286 -> 4.280 +22:40:58 step 186160 loss=3.0224 lr=1.28e-05 tok=24.517B dt=22336 wait=275 +22:42:09 step 186180 loss=2.9269 lr=1.28e-05 tok=24.520B dt=22407 wait=275 +22:43:26 step 186200 loss=2.8713 lr=1.27e-05 tok=24.522B dt=22484 wait=275 +22:44:31 step 186220 loss=2.9875 lr=1.27e-05 tok=24.525B dt=22549 wait=275 +22:45:35 step 186240 loss=2.8919 lr=1.27e-05 tok=24.528B dt=22612 wait=276 +22:46:39 step 186260 loss=2.9373 lr=1.27e-05 tok=24.530B dt=22677 wait=276 +22:47:43 step 186280 loss=2.9080 lr=1.27e-05 tok=24.533B dt=22741 wait=276 +22:48:48 step 186300 loss=2.8583 lr=1.27e-05 tok=24.535B dt=22806 wait=276 +22:49:01 pushed ckpt_w179.pt,log_w179.txt +22:51:33 waited 151s for global r=789 +22:51:42 global r=789 load 4.284 -> 4.283 +22:52:48 step 186320 loss=2.9208 lr=1.26e-05 tok=24.538B dt=23045 wait=276 +22:54:00 step 186340 loss=2.9035 lr=1.26e-05 tok=24.541B dt=23118 wait=276 +22:55:17 step 186360 loss=3.0356 lr=1.26e-05 tok=24.543B dt=23194 wait=277 +22:56:21 step 186380 loss=2.9525 lr=1.26e-05 tok=24.546B dt=23259 wait=277 +22:57:26 step 186400 loss=2.9003 lr=1.26e-05 tok=24.548B dt=23324 wait=277 +22:58:29 step 186420 loss=3.1434 lr=1.25e-05 tok=24.551B dt=23387 wait=277 +22:59:34 step 186440 loss=2.9077 lr=1.25e-05 tok=24.554B dt=23452 wait=277 +23:00:23 pushed ckpt_w179.pt,log_w179.txt +23:03:25 waited 181s for global r=790 +23:03:33 global r=790 load 4.283 -> 4.280 +23:04:07 step 186460 loss=3.0073 lr=1.25e-05 tok=24.556B dt=23725 wait=277 +23:05:14 step 186480 loss=2.9604 lr=1.25e-05 tok=24.559B dt=23792 wait=277 +23:06:37 step 186500 loss=2.8944 lr=1.25e-05 tok=24.562B dt=23875 wait=278 +23:07:43 step 186520 loss=3.0164 lr=1.24e-05 tok=24.564B dt=23941 wait=278 +23:08:47 step 186540 loss=2.9515 lr=1.24e-05 tok=24.567B dt=24004 wait=278 +23:09:51 step 186560 loss=3.1061 lr=1.24e-05 tok=24.569B dt=24069 wait=278 +23:10:55 step 186580 loss=3.0286 lr=1.24e-05 tok=24.572B dt=24133 wait=278 +23:11:59 step 186600 loss=2.9336 lr=1.24e-05 tok=24.575B dt=24197 wait=278 +23:12:16 pushed ckpt_w179.pt,log_w179.txt +23:15:23 waited 187s for global r=791 +23:15:44 global r=791 load 4.281 -> 4.278 +23:16:53 step 186620 loss=2.9092 lr=1.24e-05 tok=24.577B dt=24491 wait=279 +23:18:14 step 186640 loss=3.0545 lr=1.23e-05 tok=24.580B dt=24572 wait=279 +23:19:18 step 186660 loss=2.9679 lr=1.23e-05 tok=24.583B dt=24636 wait=279 +23:20:22 step 186680 loss=2.9618 lr=1.23e-05 tok=24.585B dt=24700 wait=279 +23:21:26 step 186700 loss=2.9151 lr=1.23e-05 tok=24.588B dt=24764 wait=279 +23:22:29 step 186720 loss=2.9720 lr=1.23e-05 tok=24.590B dt=24827 wait=279 +23:23:32 step 186740 loss=2.9111 lr=1.22e-05 tok=24.593B dt=24890 wait=280 +23:24:20 pushed ckpt_w179.pt,log_w179.txt +23:27:06 waited 166s for global r=792 +23:27:16 global r=792 load 4.282 -> 4.279 +23:27:50 step 186760 loss=2.8479 lr=1.22e-05 tok=24.596B dt=25148 wait=280 +23:29:02 step 186780 loss=2.7725 lr=1.22e-05 tok=24.598B dt=25220 wait=280 +23:30:18 step 186800 loss=2.8773 lr=1.22e-05 tok=24.601B dt=25296 wait=280 +23:31:21 step 186820 loss=3.0227 lr=1.22e-05 tok=24.604B dt=25359 wait=280 +23:32:25 step 186840 loss=2.8661 lr=1.22e-05 tok=24.606B dt=25423 wait=280 +23:33:29 step 186860 loss=2.9932 lr=1.21e-05 tok=24.609B dt=25487 wait=281 +23:34:32 step 186880 loss=3.0290 lr=1.21e-05 tok=24.611B dt=25550 wait=281 +23:35:36 step 186900 loss=2.9362 lr=1.21e-05 tok=24.614B dt=25614 wait=281 +23:35:46 pushed ckpt_w179.pt,log_w179.txt +23:39:30 global r=793 load 4.281 -> 4.283 +23:40:41 step 186920 loss=3.0780 lr=1.21e-05 tok=24.618B dt=25919 wait=281 +23:41:57 step 186940 loss=3.0311 lr=1.21e-05 tok=24.621B dt=25995 wait=281 +23:43:01 step 186960 loss=2.9890 lr=1.20e-05 tok=24.623B dt=26059 wait=281 +23:44:06 step 186980 loss=3.0668 lr=1.20e-05 tok=24.626B dt=26124 wait=282 +23:45:09 step 187000 loss=3.0007 lr=1.20e-05 tok=24.628B dt=26187 wait=282 +23:46:13 step 187020 loss=3.0654 lr=1.20e-05 tok=24.631B dt=26251 wait=282 +23:47:17 step 187040 loss=2.9662 lr=1.20e-05 tok=24.634B dt=26315 wait=282 +23:48:06 pushed ckpt_w179.pt,log_w179.txt +23:50:52 waited 166s for global r=794 +23:51:00 global r=794 load 4.279 -> 4.279 +23:51:37 step 187060 loss=3.0889 lr=1.20e-05 tok=24.636B dt=26575 wait=282 +23:53:01 step 187080 loss=3.0391 lr=1.19e-05 tok=24.639B dt=26659 wait=282 +23:54:05 step 187100 loss=2.9960 lr=1.19e-05 tok=24.642B dt=26723 wait=282 +23:55:09 step 187120 loss=3.0826 lr=1.19e-05 tok=24.644B dt=26787 wait=283 +23:56:13 step 187140 loss=2.9495 lr=1.19e-05 tok=24.647B dt=26851 wait=283 +23:57:17 step 187160 loss=3.0083 lr=1.19e-05 tok=24.649B dt=26915 wait=283 +23:58:21 step 187180 loss=3.0328 lr=1.18e-05 tok=24.652B dt=26979 wait=283 +23:59:25 step 187200 loss=3.0210 lr=1.18e-05 tok=24.655B dt=27043 wait=283 +23:59:35 pushed ckpt_w179.pt,log_w179.txt +00:02:42 waited 187s for global r=795 +00:02:52 global r=795 load 4.277 -> 4.279 +00:04:12 step 187220 loss=3.0449 lr=1.18e-05 tok=24.657B dt=27330 wait=283 +00:05:21 step 187240 loss=3.0855 lr=1.18e-05 tok=24.660B dt=27399 wait=283 +00:06:25 step 187260 loss=3.0156 lr=1.18e-05 tok=24.663B dt=27463 wait=284 +00:07:29 step 187280 loss=3.0200 lr=1.17e-05 tok=24.665B dt=27527 wait=284 +00:08:33 step 187300 loss=2.9723 lr=1.17e-05 tok=24.668B dt=27591 wait=284 +00:09:37 step 187320 loss=3.1716 lr=1.17e-05 tok=24.670B dt=27655 wait=284 +00:10:41 step 187340 loss=3.0164 lr=1.17e-05 tok=24.673B dt=27719 wait=284 +00:11:30 pushed ckpt_w179.pt,log_w179.txt +00:14:31 waited 181s for global r=796 +00:14:41 global r=796 load 4.276 -> 4.277 +00:15:22 step 187360 loss=3.0759 lr=1.17e-05 tok=24.676B dt=28000 wait=284 +00:16:37 step 187380 loss=2.9731 lr=1.17e-05 tok=24.678B dt=28075 wait=285 +00:17:41 step 187400 loss=3.1383 lr=1.16e-05 tok=24.681B dt=28139 wait=285 +00:18:44 step 187420 loss=3.2224 lr=1.16e-05 tok=24.683B dt=28202 wait=285 +00:19:48 step 187440 loss=3.0997 lr=1.16e-05 tok=24.686B dt=28266 wait=285 +00:20:52 step 187460 loss=3.1293 lr=1.16e-05 tok=24.689B dt=28330 wait=285 +00:21:55 step 187480 loss=3.0290 lr=1.16e-05 tok=24.691B dt=28393 wait=285 +00:22:59 step 187500 loss=3.0982 lr=1.15e-05 tok=24.694B dt=28457 wait=286 +00:23:09 pushed ckpt_w179.pt,log_w179.txt +00:27:16 step 187520 loss=3.1837 lr=1.15e-05 tok=24.697B dt=28714 wait=286 +00:28:19 step 187540 loss=2.9832 lr=1.15e-05 tok=24.699B dt=28777 wait=286 +00:29:23 step 187560 loss=2.9828 lr=1.15e-05 tok=24.702B dt=28841 wait=286 +00:30:26 step 187580 loss=2.9106 lr=1.15e-05 tok=24.704B dt=28904 wait=286 +00:31:29 step 187600 loss=2.9458 lr=1.15e-05 tok=24.707B dt=28967 wait=286 +00:32:14 global r=797 load 4.281 -> 4.278 +00:33:26 step 187520 loss=2.9623 lr=1.15e-05 tok=24.711B dt=29084 wait=287 +00:34:40 step 187540 loss=3.0102 lr=1.15e-05 tok=24.714B dt=29158 wait=287 +00:35:43 step 187560 loss=3.0709 lr=1.15e-05 tok=24.716B dt=29221 wait=287 +00:36:46 step 187580 loss=2.9824 lr=1.15e-05 tok=24.719B dt=29284 wait=287 +00:37:49 step 187600 loss=2.9686 lr=1.15e-05 tok=24.721B dt=29347 wait=287 +00:38:53 step 187620 loss=3.1109 lr=1.14e-05 tok=24.724B dt=29411 wait=287 +00:39:56 step 187640 loss=3.0032 lr=1.14e-05 tok=24.727B dt=29474 wait=288 +00:40:39 pushed ckpt_w179.pt,log_w179.txt +00:43:41 waited 181s for global r=798 +00:43:50 global r=798 load 4.279 -> 4.279 +00:44:30 step 187660 loss=3.0696 lr=1.14e-05 tok=24.729B dt=29748 wait=288 +00:45:47 step 187680 loss=3.0142 lr=1.14e-05 tok=24.732B dt=29825 wait=288 +00:46:52 step 187700 loss=2.9804 lr=1.14e-05 tok=24.735B dt=29890 wait=288 +00:47:55 step 187720 loss=3.0346 lr=1.13e-05 tok=24.737B dt=29953 wait=288 +00:48:58 step 187740 loss=2.9984 lr=1.13e-05 tok=24.740B dt=30016 wait=288 +00:50:02 step 187760 loss=3.0218 lr=1.13e-05 tok=24.742B dt=30080 wait=289 +00:51:05 step 187780 loss=2.9686 lr=1.13e-05 tok=24.745B dt=30143 wait=289 +00:52:09 step 187800 loss=3.0530 lr=1.13e-05 tok=24.748B dt=30207 wait=289 +00:52:19 pushed ckpt_w179.pt,log_w179.txt +00:55:20 waited 181s for global r=799 +00:55:39 global r=799 load 4.281 -> 4.281 +00:56:59 step 187820 loss=3.0267 lr=1.12e-05 tok=24.750B dt=30497 wait=289 +00:58:06 step 187840 loss=3.0787 lr=1.12e-05 tok=24.753B dt=30564 wait=289 +00:59:09 step 187860 loss=3.1222 lr=1.12e-05 tok=24.756B dt=30627 wait=289 +01:00:12 step 187880 loss=3.0608 lr=1.12e-05 tok=24.758B dt=30690 wait=289 +01:01:16 step 187900 loss=3.0004 lr=1.12e-05 tok=24.761B dt=30754 wait=290 +01:02:20 step 187920 loss=3.1340 lr=1.12e-05 tok=24.763B dt=30818 wait=290 +01:03:23 step 187940 loss=2.9291 lr=1.11e-05 tok=24.766B dt=30881 wait=290 +01:04:06 pushed ckpt_w179.pt,log_w179.txt +01:07:07 waited 182s for global r=800 +01:07:21 global r=800 load 4.280 -> 4.280 +01:08:02 step 187960 loss=3.0526 lr=1.11e-05 tok=24.769B dt=31160 wait=290 +01:09:17 step 187980 loss=2.9803 lr=1.11e-05 tok=24.771B dt=31235 wait=290 +01:10:20 step 188000 loss=2.8862 lr=1.11e-05 tok=24.774B dt=31298 wait=290 +01:11:23 step 188020 loss=3.2084 lr=1.11e-05 tok=24.777B dt=31361 wait=291 +01:12:27 step 188040 loss=2.9521 lr=1.10e-05 tok=24.779B dt=31425 wait=291 +01:13:30 step 188060 loss=3.1146 lr=1.10e-05 tok=24.782B dt=31488 wait=291 +01:14:33 step 188080 loss=2.9608 lr=1.10e-05 tok=24.784B dt=31551 wait=291 +01:15:38 step 188100 loss=2.8790 lr=1.10e-05 tok=24.787B dt=31616 wait=291 +01:15:49 pushed ckpt_w179.pt,log_w179.txt +01:19:55 global r=801 load 4.277 -> 4.276 +01:21:11 step 188120 loss=2.9204 lr=1.10e-05 tok=24.791B dt=31949 wait=291 +01:22:16 step 188140 loss=3.1732 lr=1.10e-05 tok=24.794B dt=32014 wait=292 +01:23:20 step 188160 loss=2.9797 lr=1.09e-05 tok=24.796B dt=32078 wait=292 +01:24:26 step 188180 loss=2.9171 lr=1.09e-05 tok=24.799B dt=32144 wait=292 +01:25:30 step 188200 loss=2.9392 lr=1.09e-05 tok=24.801B dt=32208 wait=292 +01:26:35 step 188220 loss=2.9447 lr=1.09e-05 tok=24.804B dt=32273 wait=292 +01:27:40 step 188240 loss=2.8677 lr=1.09e-05 tok=24.807B dt=32337 wait=292 +01:28:28 pushed ckpt_w179.pt,log_w179.txt +01:31:29 waited 181s for global r=802 +01:31:38 global r=802 load 4.281 -> 4.281 +01:32:13 step 188260 loss=2.9337 lr=1.08e-05 tok=24.809B dt=32611 wait=293 +01:33:26 step 188280 loss=2.8468 lr=1.08e-05 tok=24.812B dt=32684 wait=293 +01:34:42 step 188300 loss=2.9255 lr=1.08e-05 tok=24.815B dt=32760 wait=293 +01:35:47 step 188320 loss=3.0037 lr=1.08e-05 tok=24.817B dt=32824 wait=293 +01:36:51 step 188340 loss=2.8791 lr=1.08e-05 tok=24.820B dt=32889 wait=293 +01:37:56 step 188360 loss=2.9473 lr=1.08e-05 tok=24.822B dt=32954 wait=293 +01:39:01 step 188380 loss=2.8904 lr=1.07e-05 tok=24.825B dt=33018 wait=293 +01:40:05 step 188400 loss=2.8308 lr=1.07e-05 tok=24.828B dt=33083 wait=293 +01:40:18 pushed ckpt_w179.pt,log_w179.txt +01:43:59 global r=803 load 4.282 -> 4.283 +01:45:12 step 188420 loss=2.9281 lr=1.07e-05 tok=24.832B dt=33390 wait=294 +01:46:31 step 188440 loss=2.9510 lr=1.07e-05 tok=24.834B dt=33469 wait=294 +01:47:36 step 188460 loss=2.8885 lr=1.07e-05 tok=24.837B dt=33534 wait=294 +01:48:40 step 188480 loss=3.0235 lr=1.06e-05 tok=24.839B dt=33598 wait=294 +01:49:44 step 188500 loss=2.9713 lr=1.06e-05 tok=24.842B dt=33662 wait=294 +01:50:49 step 188520 loss=3.0018 lr=1.06e-05 tok=24.845B dt=33727 wait=294 +01:51:53 step 188540 loss=2.8417 lr=1.06e-05 tok=24.847B dt=33791 wait=295 +01:52:42 pushed ckpt_w179.pt,log_w179.txt +01:55:43 waited 181s for global r=804 +01:55:53 global r=804 load 4.281 -> 4.281 +01:56:31 step 188560 loss=2.9898 lr=1.06e-05 tok=24.850B dt=34069 wait=295 +01:57:51 step 188580 loss=2.8274 lr=1.05e-05 tok=24.853B dt=34149 wait=295 +01:58:55 step 188600 loss=2.8867 lr=1.05e-05 tok=24.855B dt=34213 wait=295 +01:59:59 step 188620 loss=2.9419 lr=1.05e-05 tok=24.858B dt=34277 wait=295 +02:01:03 step 188640 loss=2.8266 lr=1.05e-05 tok=24.860B dt=34341 wait=295 +02:02:07 step 188660 loss=2.9689 lr=1.05e-05 tok=24.863B dt=34405 wait=295 +02:03:11 step 188680 loss=2.8614 lr=1.05e-05 tok=24.866B dt=34469 wait=296 +02:04:15 step 188700 loss=2.7723 lr=1.04e-05 tok=24.868B dt=34533 wait=296 +02:04:28 pushed ckpt_w179.pt,log_w179.txt +02:07:30 waited 182s for global r=805 +02:07:43 global r=805 load 4.282 -> 4.282 +02:09:04 step 188720 loss=2.8254 lr=1.04e-05 tok=24.871B dt=34822 wait=296 +02:10:12 step 188740 loss=2.9133 lr=1.04e-05 tok=24.874B dt=34890 wait=296 +02:11:15 step 188760 loss=2.9212 lr=1.04e-05 tok=24.876B dt=34953 wait=296 +02:12:18 step 188780 loss=2.7944 lr=1.04e-05 tok=24.879B dt=35016 wait=296 +02:13:21 step 188800 loss=2.9042 lr=1.03e-05 tok=24.881B dt=35079 wait=297 +02:14:25 step 188820 loss=2.9527 lr=1.03e-05 tok=24.884B dt=35143 wait=297 +02:15:28 step 188840 loss=2.8041 lr=1.03e-05 tok=24.887B dt=35206 wait=297 +02:16:13 pushed ckpt_w179.pt,log_w179.txt +02:19:15 waited 182s for global r=806 +02:19:25 global r=806 load 4.282 -> 4.282 +02:20:06 step 188860 loss=3.0023 lr=1.03e-05 tok=24.889B dt=35484 wait=297 +02:21:22 step 188880 loss=2.8904 lr=1.03e-05 tok=24.892B dt=35560 wait=298 +02:22:24 step 188900 loss=2.9590 lr=1.03e-05 tok=24.895B dt=35622 wait=298 +02:23:28 step 188920 loss=3.0291 lr=1.02e-05 tok=24.897B dt=35685 wait=298 +02:24:31 step 188940 loss=2.9841 lr=1.02e-05 tok=24.900B dt=35749 wait=298 +02:25:35 step 188960 loss=3.0175 lr=1.02e-05 tok=24.902B dt=35813 wait=298 +02:26:38 step 188980 loss=2.9902 lr=1.02e-05 tok=24.905B dt=35876 wait=298 +02:27:42 step 189000 loss=3.0034 lr=1.02e-05 tok=24.908B dt=35940 wait=299 +02:27:52 pushed ckpt_w179.pt,log_w179.txt +02:30:54 waited 181s for global r=807 +02:31:03 global r=807 load 4.280 -> 4.280 +02:32:22 step 189020 loss=3.0064 lr=1.01e-05 tok=24.910B dt=36220 wait=299 +02:33:31 step 189040 loss=2.9678 lr=1.01e-05 tok=24.913B dt=36289 wait=299 +02:34:35 step 189060 loss=2.9260 lr=1.01e-05 tok=24.915B dt=36353 wait=299 +02:35:39 step 189080 loss=2.8983 lr=1.01e-05 tok=24.918B dt=36417 wait=299 +02:36:43 step 189100 loss=2.7337 lr=1.01e-05 tok=24.921B dt=36481 wait=299 +02:37:47 step 189120 loss=3.0211 lr=1.00e-05 tok=24.923B dt=36544 wait=300 +02:38:51 step 189140 loss=2.7914 lr=1.00e-05 tok=24.926B dt=36609 wait=300 +02:39:34 pushed ckpt_w179.pt,log_w179.txt +02:42:36 waited 181s for global r=808 +02:42:45 global r=808 load 4.278 -> 4.278 +02:43:26 step 189160 loss=3.0861 lr=1.00e-05 tok=24.929B dt=36884 wait=300 +02:44:43 step 189180 loss=2.9351 lr=9.99e-06 tok=24.931B dt=36961 wait=300 +02:45:47 step 189200 loss=3.0142 lr=9.97e-06 tok=24.934B dt=37025 wait=300 +02:46:52 step 189220 loss=3.0278 lr=9.96e-06 tok=24.936B dt=37090 wait=300 +02:47:56 step 189240 loss=2.9695 lr=9.94e-06 tok=24.939B dt=37154 wait=301 +02:49:01 step 189260 loss=3.0662 lr=9.92e-06 tok=24.942B dt=37219 wait=301 +02:50:04 step 189280 loss=2.9806 lr=9.90e-06 tok=24.944B dt=37282 wait=301 +02:51:08 step 189300 loss=2.9752 lr=9.88e-06 tok=24.947B dt=37346 wait=301 +02:51:18 pushed ckpt_w179.pt,log_w179.txt +02:54:20 waited 181s for global r=809 +02:54:38 global r=809 load 4.281 -> 4.281 +02:55:34 mix source 5 lap 1 +02:56:01 step 189320 loss=2.9531 lr=9.86e-06 tok=24.950B dt=37639 wait=301 +02:57:07 step 189340 loss=3.0134 lr=9.85e-06 tok=24.952B dt=37705 wait=301 +02:58:11 step 189360 loss=2.9567 lr=9.83e-06 tok=24.955B dt=37769 wait=301 +02:59:15 step 189380 loss=2.7680 lr=9.81e-06 tok=24.957B dt=37833 wait=302 +03:00:19 step 189400 loss=2.9474 lr=9.79e-06 tok=24.960B dt=37897 wait=302 +03:01:23 step 189420 loss=3.0870 lr=9.77e-06 tok=24.963B dt=37961 wait=302 +03:02:27 step 189440 loss=2.8342 lr=9.75e-06 tok=24.965B dt=38025 wait=302 +03:03:09 pushed ckpt_w179.pt,log_w179.txt +03:06:11 waited 182s for global r=810 +03:06:25 global r=810 load 4.281 -> 4.281 +03:07:08 step 189460 loss=3.0223 lr=9.73e-06 tok=24.968B dt=38306 wait=302 +03:08:24 step 189480 loss=2.9340 lr=9.72e-06 tok=24.971B dt=38382 wait=302 +03:09:28 step 189500 loss=2.8775 lr=9.70e-06 tok=24.973B dt=38446 wait=303 +03:10:32 step 189520 loss=3.0524 lr=9.68e-06 tok=24.976B dt=38510 wait=303 +03:11:35 step 189540 loss=2.9604 lr=9.66e-06 tok=24.978B dt=38573 wait=303 +03:12:38 step 189560 loss=3.0139 lr=9.64e-06 tok=24.981B dt=38636 wait=303 +03:13:41 step 189580 loss=2.9697 lr=9.62e-06 tok=24.984B dt=38699 wait=303 +03:14:44 step 189600 loss=2.9224 lr=9.61e-06 tok=24.986B dt=38762 wait=303 diff --git a/log_w180.txt b/log_w180.txt new file mode 100644 index 0000000000000000000000000000000000000000..96dc5dd0f51e14dc7f3474cba442aa2e0af7926c --- /dev/null +++ b/log_w180.txt @@ -0,0 +1,433 @@ +launch +15:23:53 name=w180 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 127/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:23:59 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:23:59 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:24:09 params 1.08B +15:24:10 resumed step=179850 seen=23.69B round=746 +15:24:10 stream salt=179850 +15:26:07 blocks compiled +15:26:07 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +15:30:30 step 179851 loss=2.7458 lr=1.85e-05 tok=23.687B dt=263 wait=226 +15:30:52 step 179852 loss=2.8255 lr=1.85e-05 tok=23.688B dt=285 wait=226 +15:31:19 step 179860 loss=3.1992 lr=1.85e-05 tok=23.689B dt=312 wait=226 +15:31:38 global r=747 load 4.283 -> 4.283 +15:32:43 step 180020 loss=3.0938 lr=1.85e-05 tok=23.691B dt=396 wait=226 +15:33:49 step 180040 loss=3.1348 lr=1.85e-05 tok=23.694B dt=462 wait=226 +15:34:53 step 180060 loss=3.1648 lr=1.84e-05 tok=23.697B dt=526 wait=226 +15:35:57 step 180080 loss=3.1587 lr=1.84e-05 tok=23.699B dt=590 wait=226 +15:37:02 step 180100 loss=3.1823 lr=1.84e-05 tok=23.702B dt=655 wait=226 +15:38:07 step 180120 loss=3.2121 lr=1.84e-05 tok=23.704B dt=720 wait=227 +15:39:12 step 180140 loss=2.9870 lr=1.84e-05 tok=23.707B dt=785 wait=227 +15:39:57 pushed ckpt_w180.pt,log_w180.txt +15:42:29 waited 151s for global r=748 +15:42:36 global r=748 load 4.281 -> 4.280 +15:43:11 step 180160 loss=3.0027 lr=1.84e-05 tok=23.710B dt=1025 wait=227 +15:44:17 step 180180 loss=2.9647 lr=1.84e-05 tok=23.712B dt=1090 wait=227 +15:45:21 step 180200 loss=3.0026 lr=1.84e-05 tok=23.715B dt=1154 wait=227 +15:46:26 step 180220 loss=3.1036 lr=1.84e-05 tok=23.717B dt=1219 wait=227 +15:47:30 step 180240 loss=2.9771 lr=1.84e-05 tok=23.720B dt=1283 wait=227 +15:48:35 step 180260 loss=3.0871 lr=1.84e-05 tok=23.723B dt=1348 wait=228 +15:49:39 step 180280 loss=3.0264 lr=1.84e-05 tok=23.725B dt=1412 wait=228 +15:50:44 step 180300 loss=3.0587 lr=1.84e-05 tok=23.728B dt=1477 wait=228 +15:50:58 pushed ckpt_w180.pt,log_w180.txt +15:54:41 global r=749 load 4.281 -> 4.282 +15:55:46 step 180320 loss=3.1078 lr=1.84e-05 tok=23.732B dt=1779 wait=228 +15:56:51 step 180340 loss=3.1532 lr=1.84e-05 tok=23.735B dt=1844 wait=228 +15:57:55 step 180360 loss=3.0663 lr=1.84e-05 tok=23.737B dt=1909 wait=228 +15:58:59 step 180380 loss=3.1081 lr=1.83e-05 tok=23.740B dt=1972 wait=229 +16:00:04 step 180400 loss=3.0178 lr=1.83e-05 tok=23.742B dt=2037 wait=229 +16:01:09 step 180420 loss=3.0807 lr=1.83e-05 tok=23.745B dt=2102 wait=229 +16:02:13 step 180440 loss=2.9722 lr=1.83e-05 tok=23.748B dt=2166 wait=229 +16:02:57 pushed ckpt_w180.pt,log_w180.txt +16:05:43 waited 166s for global r=750 +16:05:52 global r=750 load 4.280 -> 4.283 +16:06:25 step 180460 loss=3.0325 lr=1.83e-05 tok=23.750B dt=2419 wait=229 +16:07:30 step 180480 loss=2.9214 lr=1.83e-05 tok=23.753B dt=2483 wait=229 +16:08:34 step 180500 loss=2.9437 lr=1.83e-05 tok=23.755B dt=2548 wait=229 +16:09:39 step 180520 loss=2.9594 lr=1.83e-05 tok=23.758B dt=2612 wait=230 +16:10:43 step 180540 loss=2.8818 lr=1.83e-05 tok=23.761B dt=2677 wait=230 +16:11:48 step 180560 loss=2.9644 lr=1.83e-05 tok=23.763B dt=2741 wait=230 +16:12:51 step 180580 loss=3.0098 lr=1.83e-05 tok=23.766B dt=2804 wait=230 +16:13:55 step 180600 loss=2.9939 lr=1.83e-05 tok=23.769B dt=2868 wait=230 +16:14:09 pushed ckpt_w180.pt,log_w180.txt +16:15:55 waited 106s for global r=751 +16:16:00 global r=751 load 4.279 -> 4.282 +16:17:05 step 180620 loss=2.8938 lr=1.83e-05 tok=23.771B dt=3058 wait=230 +16:18:09 step 180640 loss=3.0118 lr=1.83e-05 tok=23.774B dt=3122 wait=230 +16:19:13 step 180660 loss=2.9308 lr=1.83e-05 tok=23.776B dt=3186 wait=231 +16:20:17 step 180680 loss=2.9908 lr=1.82e-05 tok=23.779B dt=3250 wait=231 +16:21:22 step 180700 loss=2.8774 lr=1.82e-05 tok=23.782B dt=3315 wait=231 +16:22:26 step 180720 loss=2.9993 lr=1.82e-05 tok=23.784B dt=3379 wait=231 +16:23:30 step 180740 loss=2.9553 lr=1.82e-05 tok=23.787B dt=3443 wait=231 +16:24:16 pushed ckpt_w180.pt,log_w180.txt +16:27:04 waited 167s for global r=752 +16:27:09 global r=752 load 4.291 -> 4.281 +16:27:43 step 180760 loss=2.9958 lr=1.82e-05 tok=23.790B dt=3696 wait=231 +launch +16:28:57 name=w180 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 127/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:29:02 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:29:02 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:29:11 params 1.08B +16:29:13 resumed step=180750 seen=23.82B round=752 +16:29:13 stream salt=180750 +16:31:05 blocks compiled +16:31:05 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:35:10 step 180751 loss=3.0108 lr=1.78e-05 tok=23.819B dt=245 wait=235 +16:35:17 step 180752 loss=2.9965 lr=1.78e-05 tok=23.819B dt=252 wait=235 +16:35:42 step 180760 loss=3.1461 lr=1.78e-05 tok=23.820B dt=277 wait=235 +16:36:47 step 180780 loss=2.9691 lr=1.78e-05 tok=23.822B dt=342 wait=235 +16:37:52 step 180800 loss=3.1280 lr=1.77e-05 tok=23.825B dt=407 wait=236 +16:38:57 step 180820 loss=3.2275 lr=1.77e-05 tok=23.828B dt=472 wait=236 +16:39:08 global r=753 load 4.286 -> 4.284 +16:40:13 step 180920 loss=2.6768 lr=1.76e-05 tok=23.830B dt=548 wait=236 +16:41:17 step 180940 loss=3.0520 lr=1.76e-05 tok=23.833B dt=612 wait=236 +16:42:22 step 180960 loss=3.1531 lr=1.76e-05 tok=23.835B dt=677 wait=236 +16:43:25 step 180980 loss=3.1169 lr=1.76e-05 tok=23.838B dt=740 wait=236 +16:44:30 step 181000 loss=3.1860 lr=1.75e-05 tok=23.841B dt=805 wait=236 +16:45:34 step 181020 loss=3.1078 lr=1.75e-05 tok=23.843B dt=869 wait=236 +16:46:37 step 181040 loss=3.0473 lr=1.75e-05 tok=23.846B dt=932 wait=237 +16:47:23 pushed ckpt_w180.pt,log_w180.txt +16:50:59 step 181060 loss=3.0689 lr=1.75e-05 tok=23.849B dt=1194 wait=237 +16:51:08 global r=754 load 4.282 -> 4.281 +16:51:40 step 181060 loss=3.1054 lr=1.75e-05 tok=23.850B dt=1235 wait=237 +16:52:45 step 181080 loss=3.1115 lr=1.75e-05 tok=23.852B dt=1300 wait=237 +16:53:50 step 181100 loss=2.9979 lr=1.75e-05 tok=23.855B dt=1364 wait=237 +16:54:53 step 181120 loss=3.2253 lr=1.74e-05 tok=23.858B dt=1428 wait=237 +16:55:57 step 181140 loss=3.0034 lr=1.74e-05 tok=23.860B dt=1492 wait=237 +16:57:02 step 181160 loss=3.0328 lr=1.74e-05 tok=23.863B dt=1557 wait=238 +16:58:06 step 181180 loss=3.0091 lr=1.74e-05 tok=23.866B dt=1621 wait=238 +16:59:11 step 181200 loss=2.9076 lr=1.74e-05 tok=23.868B dt=1686 wait=238 +16:59:23 pushed ckpt_w180.pt,log_w180.txt +17:01:09 waited 106s for global r=755 +17:01:16 global r=755 load 4.279 -> 4.280 +17:02:22 step 181220 loss=3.0221 lr=1.73e-05 tok=23.871B dt=1877 wait=238 +17:03:27 step 181240 loss=3.0360 lr=1.73e-05 tok=23.873B dt=1942 wait=238 +17:04:33 step 181260 loss=2.9679 lr=1.73e-05 tok=23.876B dt=2008 wait=238 +17:05:38 step 181280 loss=2.9265 lr=1.73e-05 tok=23.879B dt=2073 wait=238 +17:06:44 step 181300 loss=2.8908 lr=1.73e-05 tok=23.881B dt=2139 wait=239 +17:07:50 step 181320 loss=3.0025 lr=1.73e-05 tok=23.884B dt=2205 wait=239 +17:08:56 step 181340 loss=2.8741 lr=1.72e-05 tok=23.887B dt=2271 wait=239 +17:09:44 pushed ckpt_w180.pt,log_w180.txt +17:11:00 waited 76s for global r=756 +17:11:07 global r=756 load 4.291 -> 4.281 +17:11:41 step 181360 loss=2.9603 lr=1.72e-05 tok=23.889B dt=2436 wait=239 +17:12:47 step 181380 loss=2.8763 lr=1.72e-05 tok=23.892B dt=2502 wait=239 +17:13:53 step 181400 loss=2.8924 lr=1.72e-05 tok=23.894B dt=2568 wait=239 +17:14:59 step 181420 loss=2.9619 lr=1.72e-05 tok=23.897B dt=2634 wait=239 +17:16:04 step 181440 loss=2.8667 lr=1.71e-05 tok=23.900B dt=2699 wait=239 +17:17:10 step 181460 loss=2.9729 lr=1.71e-05 tok=23.902B dt=2765 wait=240 +17:18:15 step 181480 loss=2.9215 lr=1.71e-05 tok=23.905B dt=2830 wait=240 +17:19:21 step 181500 loss=2.9828 lr=1.71e-05 tok=23.908B dt=2896 wait=240 +17:19:32 pushed ckpt_w180.pt,log_w180.txt +17:22:33 waited 181s for global r=757 +17:22:40 global r=757 load 4.284 -> 4.282 +17:23:46 step 181520 loss=2.9518 lr=1.71e-05 tok=23.910B dt=3161 wait=240 +17:24:53 step 181540 loss=2.9699 lr=1.70e-05 tok=23.913B dt=3228 wait=240 +17:25:59 step 181560 loss=2.9458 lr=1.70e-05 tok=23.915B dt=3294 wait=240 +17:27:04 step 181580 loss=3.0382 lr=1.70e-05 tok=23.918B dt=3359 wait=240 +17:28:10 step 181600 loss=2.9379 lr=1.70e-05 tok=23.921B dt=3425 wait=241 +17:29:16 step 181620 loss=3.0225 lr=1.70e-05 tok=23.923B dt=3491 wait=241 +17:30:22 step 181640 loss=2.8923 lr=1.70e-05 tok=23.926B dt=3557 wait=241 +17:31:06 pushed ckpt_w180.pt,log_w180.txt +17:32:52 waited 106s for global r=758 +17:32:59 global r=758 load 4.290 -> 4.283 +17:33:33 step 181660 loss=3.0366 lr=1.69e-05 tok=23.929B dt=3748 wait=241 +17:34:40 step 181680 loss=2.4327 lr=1.69e-05 tok=23.931B dt=3815 wait=241 +17:35:46 step 181700 loss=3.0503 lr=1.69e-05 tok=23.934B dt=3881 wait=241 +17:36:52 step 181720 loss=3.0314 lr=1.69e-05 tok=23.936B dt=3946 wait=241 +17:37:58 step 181740 loss=3.0601 lr=1.69e-05 tok=23.939B dt=4013 wait=242 +17:39:03 step 181760 loss=3.1507 lr=1.68e-05 tok=23.942B dt=4078 wait=242 +17:40:09 step 181780 loss=2.9223 lr=1.68e-05 tok=23.944B dt=4144 wait=242 +17:41:15 step 181800 loss=3.0023 lr=1.68e-05 tok=23.947B dt=4210 wait=242 +17:41:27 pushed ckpt_w180.pt,log_w180.txt +17:42:42 waited 76s for global r=759 +17:42:50 global r=759 load 4.291 -> 4.283 +17:43:57 step 181820 loss=3.0494 lr=1.68e-05 tok=23.949B dt=4372 wait=242 +17:45:02 step 181840 loss=3.0331 lr=1.68e-05 tok=23.952B dt=4437 wait=242 +17:46:08 step 181860 loss=2.9828 lr=1.68e-05 tok=23.955B dt=4503 wait=242 +17:47:13 step 181880 loss=3.0510 lr=1.67e-05 tok=23.957B dt=4568 wait=242 +17:48:17 step 181900 loss=2.9519 lr=1.67e-05 tok=23.960B dt=4632 wait=243 +17:49:22 step 181920 loss=3.0711 lr=1.67e-05 tok=23.963B dt=4697 wait=243 +17:50:27 step 181940 loss=2.9950 lr=1.67e-05 tok=23.965B dt=4762 wait=243 +17:51:12 pushed ckpt_w180.pt,log_w180.txt +17:52:57 waited 106s for global r=760 +17:53:06 global r=760 load 4.284 -> 4.281 +17:53:40 step 181960 loss=3.0808 lr=1.67e-05 tok=23.968B dt=4955 wait=243 +17:54:44 step 181980 loss=2.9908 lr=1.66e-05 tok=23.970B dt=5019 wait=243 +17:55:48 step 182000 loss=3.0293 lr=1.66e-05 tok=23.973B dt=5083 wait=243 +17:56:52 step 182020 loss=3.1386 lr=1.66e-05 tok=23.976B dt=5147 wait=243 +17:57:56 step 182040 loss=2.9770 lr=1.66e-05 tok=23.978B dt=5211 wait=243 +17:58:59 step 182060 loss=3.1491 lr=1.66e-05 tok=23.981B dt=5274 wait=244 +18:00:03 step 182080 loss=2.9608 lr=1.65e-05 tok=23.984B dt=5338 wait=244 +18:01:08 step 182100 loss=2.9567 lr=1.65e-05 tok=23.986B dt=5403 wait=244 +18:01:20 pushed ckpt_w180.pt,log_w180.txt +18:02:50 waited 91s for global r=761 +18:03:00 global r=761 load 4.280 -> 4.281 +18:04:05 step 182120 loss=3.0175 lr=1.65e-05 tok=23.989B dt=5580 wait=244 +18:05:09 step 182140 loss=3.0499 lr=1.65e-05 tok=23.991B dt=5644 wait=244 +18:06:12 step 182160 loss=3.0995 lr=1.65e-05 tok=23.994B dt=5707 wait=244 +18:07:16 step 182180 loss=2.9541 lr=1.65e-05 tok=23.997B dt=5771 wait=244 +18:08:20 step 182200 loss=2.9780 lr=1.64e-05 tok=23.999B dt=5835 wait=245 +18:09:24 step 182220 loss=3.0254 lr=1.64e-05 tok=24.002B dt=5899 wait=245 +18:10:28 step 182240 loss=2.8958 lr=1.64e-05 tok=24.005B dt=5963 wait=245 +18:11:12 pushed ckpt_w180.pt,log_w180.txt +18:12:27 waited 76s for global r=762 +18:12:36 global r=762 load 4.283 -> 4.281 +18:13:09 step 182260 loss=2.9649 lr=1.64e-05 tok=24.007B dt=6124 wait=245 +18:14:13 step 182280 loss=3.0043 lr=1.64e-05 tok=24.010B dt=6188 wait=245 +18:15:17 step 182300 loss=3.0421 lr=1.63e-05 tok=24.012B dt=6252 wait=245 +18:16:21 step 182320 loss=3.1062 lr=1.63e-05 tok=24.015B dt=6316 wait=245 +18:17:25 step 182340 loss=3.0694 lr=1.63e-05 tok=24.018B dt=6380 wait=246 +18:18:29 step 182360 loss=3.1569 lr=1.63e-05 tok=24.020B dt=6444 wait=246 +18:19:32 step 182380 loss=3.0994 lr=1.63e-05 tok=24.023B dt=6507 wait=246 +18:20:37 step 182400 loss=3.0749 lr=1.63e-05 tok=24.025B dt=6572 wait=246 +18:20:56 pushed ckpt_w180.pt,log_w180.txt +18:24:43 global r=763 load 4.278 -> 4.280 +18:25:47 step 182420 loss=3.1215 lr=1.62e-05 tok=24.029B dt=6882 wait=246 +18:26:52 step 182440 loss=3.2193 lr=1.62e-05 tok=24.032B dt=6947 wait=246 +18:27:56 step 182460 loss=3.2069 lr=1.62e-05 tok=24.035B dt=7011 wait=247 +18:29:01 step 182480 loss=3.1849 lr=1.62e-05 tok=24.037B dt=7076 wait=247 +18:30:05 step 182500 loss=2.9981 lr=1.62e-05 tok=24.040B dt=7140 wait=247 +18:31:09 step 182520 loss=3.1705 lr=1.61e-05 tok=24.043B dt=7204 wait=247 +18:32:13 step 182540 loss=3.0204 lr=1.61e-05 tok=24.045B dt=7268 wait=247 +18:32:59 pushed ckpt_w180.pt,log_w180.txt +18:35:45 waited 166s for global r=764 +18:35:56 global r=764 load 4.275 -> 4.280 +18:36:29 step 182560 loss=3.1094 lr=1.61e-05 tok=24.048B dt=7524 wait=247 +18:37:33 step 182580 loss=2.9269 lr=1.61e-05 tok=24.050B dt=7588 wait=247 +18:38:37 step 182600 loss=2.9271 lr=1.61e-05 tok=24.053B dt=7652 wait=248 +18:39:42 step 182620 loss=3.0432 lr=1.61e-05 tok=24.056B dt=7717 wait=248 +18:40:46 step 182640 loss=3.0349 lr=1.60e-05 tok=24.058B dt=7781 wait=248 +18:41:51 step 182660 loss=3.1383 lr=1.60e-05 tok=24.061B dt=7846 wait=248 +18:42:56 step 182680 loss=3.0006 lr=1.60e-05 tok=24.064B dt=7911 wait=248 +18:44:01 step 182700 loss=3.0456 lr=1.60e-05 tok=24.066B dt=7976 wait=248 +18:44:16 pushed ckpt_w180.pt,log_w180.txt +18:45:17 waited 61s for global r=765 +18:45:26 global r=765 load 4.276 -> 4.280 +18:46:32 step 182720 loss=3.0352 lr=1.60e-05 tok=24.069B dt=8127 wait=249 +18:47:37 step 182740 loss=3.0231 lr=1.59e-05 tok=24.071B dt=8192 wait=249 +18:48:41 step 182760 loss=2.9433 lr=1.59e-05 tok=24.074B dt=8256 wait=249 +18:49:46 step 182780 loss=2.9959 lr=1.59e-05 tok=24.077B dt=8321 wait=249 +18:50:50 step 182800 loss=2.9937 lr=1.59e-05 tok=24.079B dt=8385 wait=249 +18:51:55 step 182820 loss=3.0096 lr=1.59e-05 tok=24.082B dt=8450 wait=249 +18:53:00 step 182840 loss=2.8795 lr=1.58e-05 tok=24.084B dt=8515 wait=249 +18:53:46 pushed ckpt_w180.pt,log_w180.txt +18:54:48 waited 62s for global r=766 +18:54:57 global r=766 load 4.286 -> 4.281 +18:55:31 step 182860 loss=2.9731 lr=1.58e-05 tok=24.087B dt=8666 wait=250 +18:56:37 step 182880 loss=2.9698 lr=1.58e-05 tok=24.090B dt=8732 wait=250 +18:57:41 step 182900 loss=2.9412 lr=1.58e-05 tok=24.092B dt=8796 wait=250 +18:58:46 step 182920 loss=2.9497 lr=1.58e-05 tok=24.095B dt=8861 wait=250 +18:59:50 step 182940 loss=3.0739 lr=1.58e-05 tok=24.098B dt=8925 wait=250 +19:00:56 step 182960 loss=3.0711 lr=1.57e-05 tok=24.100B dt=8991 wait=250 +19:02:01 step 182980 loss=2.9323 lr=1.57e-05 tok=24.103B dt=9055 wait=250 +19:03:07 step 183000 loss=2.3157 lr=1.57e-05 tok=24.105B dt=9122 wait=251 +19:03:19 pushed ckpt_w180.pt,log_w180.txt +19:04:36 waited 77s for global r=767 +19:04:42 global r=767 load 4.294 -> 4.283 +19:05:47 step 183020 loss=3.0273 lr=1.57e-05 tok=24.108B dt=9282 wait=251 +19:06:52 step 183040 loss=3.1545 lr=1.57e-05 tok=24.111B dt=9347 wait=251 +19:07:56 step 183060 loss=3.1077 lr=1.56e-05 tok=24.113B dt=9411 wait=251 +19:09:00 step 183080 loss=3.0737 lr=1.56e-05 tok=24.116B dt=9475 wait=251 +19:10:04 step 183100 loss=3.0156 lr=1.56e-05 tok=24.119B dt=9539 wait=251 +19:11:09 step 183120 loss=3.1823 lr=1.56e-05 tok=24.121B dt=9604 wait=251 +19:12:13 step 183140 loss=2.9456 lr=1.56e-05 tok=24.124B dt=9668 wait=252 +19:12:56 pushed ckpt_w180.pt,log_w180.txt +19:13:57 waited 61s for global r=768 +19:14:06 global r=768 load 4.286 -> 4.284 +19:14:40 step 183160 loss=3.0962 lr=1.56e-05 tok=24.126B dt=9815 wait=252 +19:15:44 step 183180 loss=3.1183 lr=1.55e-05 tok=24.129B dt=9879 wait=252 +19:16:48 step 183200 loss=3.0760 lr=1.55e-05 tok=24.132B dt=9943 wait=252 +19:17:52 step 183220 loss=3.2007 lr=1.55e-05 tok=24.134B dt=10007 wait=252 +19:18:56 step 183240 loss=3.1003 lr=1.55e-05 tok=24.137B dt=10071 wait=252 +19:20:00 step 183260 loss=3.1398 lr=1.55e-05 tok=24.140B dt=10135 wait=252 +19:21:04 step 183280 loss=2.8882 lr=1.54e-05 tok=24.142B dt=10199 wait=253 +19:22:08 step 183300 loss=2.9898 lr=1.54e-05 tok=24.145B dt=10263 wait=253 +19:22:23 pushed ckpt_w180.pt,log_w180.txt +19:23:24 waited 61s for global r=769 +19:23:31 global r=769 load 4.283 -> 4.282 +19:24:37 step 183320 loss=2.9187 lr=1.54e-05 tok=24.147B dt=10412 wait=253 +19:25:41 step 183340 loss=3.0407 lr=1.54e-05 tok=24.150B dt=10476 wait=253 +19:26:45 step 183360 loss=2.9847 lr=1.54e-05 tok=24.153B dt=10540 wait=253 +19:27:49 step 183380 loss=2.8984 lr=1.53e-05 tok=24.155B dt=10604 wait=253 +19:28:51 step 183400 loss=2.9240 lr=1.53e-05 tok=24.158B dt=10666 wait=253 +19:29:55 step 183420 loss=3.0651 lr=1.53e-05 tok=24.161B dt=10730 wait=254 +19:30:58 step 183440 loss=2.9267 lr=1.53e-05 tok=24.163B dt=10793 wait=254 +19:31:42 pushed ckpt_w180.pt,log_w180.txt +19:32:58 waited 76s for global r=770 +19:33:05 global r=770 load 4.281 -> 4.281 +19:33:38 step 183460 loss=3.0404 lr=1.53e-05 tok=24.166B dt=10953 wait=254 +19:34:42 step 183480 loss=3.0734 lr=1.53e-05 tok=24.168B dt=11017 wait=254 +19:35:45 step 183500 loss=3.0829 lr=1.52e-05 tok=24.171B dt=11080 wait=254 +19:36:49 step 183520 loss=3.0838 lr=1.52e-05 tok=24.174B dt=11144 wait=254 +19:37:53 step 183540 loss=3.0240 lr=1.52e-05 tok=24.176B dt=11208 wait=255 +19:38:57 step 183560 loss=3.0272 lr=1.52e-05 tok=24.179B dt=11272 wait=255 +19:40:01 step 183580 loss=2.9950 lr=1.52e-05 tok=24.181B dt=11336 wait=255 +19:41:05 step 183600 loss=3.0159 lr=1.51e-05 tok=24.184B dt=11400 wait=255 +19:41:18 pushed ckpt_w180.pt,log_w180.txt +19:44:59 global r=771 load 4.280 -> 4.282 +19:46:03 step 183620 loss=2.9885 lr=1.51e-05 tok=24.188B dt=11698 wait=255 +19:47:07 step 183640 loss=2.9731 lr=1.51e-05 tok=24.191B dt=11762 wait=255 +19:48:11 step 183660 loss=2.9413 lr=1.51e-05 tok=24.193B dt=11826 wait=256 +19:49:15 step 183680 loss=2.9828 lr=1.51e-05 tok=24.196B dt=11890 wait=256 +19:50:19 step 183700 loss=3.0009 lr=1.51e-05 tok=24.199B dt=11954 wait=256 +19:51:23 step 183720 loss=3.0954 lr=1.50e-05 tok=24.201B dt=12018 wait=256 +19:52:27 step 183740 loss=3.0424 lr=1.50e-05 tok=24.204B dt=12082 wait=256 +19:53:19 pushed ckpt_w180.pt,log_w180.txt +19:56:22 waited 182s for global r=772 +19:56:30 global r=772 load 4.280 -> 4.280 +19:57:04 step 183760 loss=3.1131 lr=1.50e-05 tok=24.206B dt=12359 wait=256 +19:58:07 step 183780 loss=3.0176 lr=1.50e-05 tok=24.209B dt=12422 wait=256 +19:59:11 step 183800 loss=3.0315 lr=1.50e-05 tok=24.212B dt=12486 wait=257 +20:00:16 step 183820 loss=3.0517 lr=1.49e-05 tok=24.214B dt=12551 wait=257 +20:01:20 step 183840 loss=3.0062 lr=1.49e-05 tok=24.217B dt=12615 wait=257 +20:02:24 step 183860 loss=3.0749 lr=1.49e-05 tok=24.219B dt=12679 wait=257 +20:03:29 step 183880 loss=2.8529 lr=1.49e-05 tok=24.222B dt=12744 wait=257 +20:04:33 step 183900 loss=3.1244 lr=1.49e-05 tok=24.225B dt=12808 wait=257 +20:04:47 pushed ckpt_w180.pt,log_w180.txt +20:07:54 waited 187s for global r=773 +20:08:03 global r=773 load 4.283 -> 4.282 +20:09:09 step 183920 loss=3.0364 lr=1.49e-05 tok=24.227B dt=13084 wait=257 +20:10:13 step 183940 loss=3.0444 lr=1.48e-05 tok=24.230B dt=13148 wait=258 +20:11:17 step 183960 loss=2.9553 lr=1.48e-05 tok=24.233B dt=13212 wait=258 +20:12:21 step 183980 loss=2.9297 lr=1.48e-05 tok=24.235B dt=13276 wait=258 +20:13:26 step 184000 loss=3.0169 lr=1.48e-05 tok=24.238B dt=13341 wait=258 +20:14:30 step 184020 loss=3.1312 lr=1.48e-05 tok=24.240B dt=13405 wait=258 +20:15:34 step 184040 loss=2.8879 lr=1.47e-05 tok=24.243B dt=13469 wait=258 +20:16:20 pushed ckpt_w180.pt,log_w180.txt +20:18:57 waited 158s for global r=774 +20:19:06 global r=774 load 4.283 -> 4.282 +20:19:40 step 184060 loss=3.1835 lr=1.47e-05 tok=24.246B dt=13715 wait=258 +20:20:45 step 184080 loss=2.9159 lr=1.47e-05 tok=24.248B dt=13780 wait=259 +20:21:49 step 184100 loss=3.0123 lr=1.47e-05 tok=24.251B dt=13844 wait=259 +20:22:53 step 184120 loss=3.0673 lr=1.47e-05 tok=24.254B dt=13908 wait=259 +20:23:57 step 184140 loss=3.0509 lr=1.46e-05 tok=24.256B dt=13972 wait=259 +20:25:02 step 184160 loss=3.1249 lr=1.46e-05 tok=24.259B dt=14037 wait=259 +20:26:06 step 184180 loss=3.1967 lr=1.46e-05 tok=24.261B dt=14101 wait=259 +20:27:11 step 184200 loss=3.1004 lr=1.46e-05 tok=24.264B dt=14166 wait=259 +20:27:31 pushed ckpt_w180.pt,log_w180.txt +20:28:34 waited 63s for global r=775 +20:28:45 global r=775 load 4.280 -> 4.281 +20:29:50 step 184220 loss=3.0825 lr=1.46e-05 tok=24.267B dt=14325 wait=260 +20:30:55 step 184240 loss=3.0380 lr=1.46e-05 tok=24.269B dt=14390 wait=260 +20:31:59 step 184260 loss=3.0999 lr=1.45e-05 tok=24.272B dt=14454 wait=260 +20:33:03 step 184280 loss=2.9857 lr=1.45e-05 tok=24.275B dt=14518 wait=260 +20:34:06 step 184300 loss=3.0093 lr=1.45e-05 tok=24.277B dt=14581 wait=261 +20:35:10 step 184320 loss=3.0978 lr=1.45e-05 tok=24.280B dt=14645 wait=261 +20:36:14 step 184340 loss=3.0012 lr=1.45e-05 tok=24.282B dt=14709 wait=261 +20:37:00 pushed ckpt_w180.pt,log_w180.txt +20:38:17 waited 76s for global r=776 +20:38:33 global r=776 load 4.282 -> 4.281 +20:39:06 step 184360 loss=3.0828 lr=1.44e-05 tok=24.285B dt=14881 wait=261 +20:40:10 step 184380 loss=3.0709 lr=1.44e-05 tok=24.288B dt=14945 wait=261 +20:41:14 step 184400 loss=3.0069 lr=1.44e-05 tok=24.290B dt=15008 wait=261 +20:42:18 step 184420 loss=3.1163 lr=1.44e-05 tok=24.293B dt=15073 wait=261 +20:43:21 step 184440 loss=3.1349 lr=1.44e-05 tok=24.296B dt=15136 wait=262 +20:44:26 step 184460 loss=3.1081 lr=1.44e-05 tok=24.298B dt=15201 wait=262 +20:45:31 step 184480 loss=3.1934 lr=1.43e-05 tok=24.301B dt=15266 wait=262 +20:46:36 step 184500 loss=3.2811 lr=1.43e-05 tok=24.303B dt=15331 wait=262 +20:46:52 pushed ckpt_w180.pt,log_w180.txt +20:48:08 waited 76s for global r=777 +20:48:15 global r=777 load 4.278 -> 4.280 +20:49:21 step 184520 loss=3.1297 lr=1.43e-05 tok=24.306B dt=15495 wait=262 +20:50:25 step 184540 loss=3.1742 lr=1.43e-05 tok=24.309B dt=15560 wait=262 +20:51:29 step 184560 loss=2.9707 lr=1.43e-05 tok=24.311B dt=15624 wait=262 +20:52:33 step 184580 loss=3.0486 lr=1.42e-05 tok=24.314B dt=15688 wait=263 +20:53:36 step 184600 loss=3.0072 lr=1.42e-05 tok=24.316B dt=15751 wait=263 +20:54:40 step 184620 loss=3.0777 lr=1.42e-05 tok=24.319B dt=15815 wait=263 +20:55:43 step 184640 loss=2.9036 lr=1.42e-05 tok=24.322B dt=15878 wait=263 +20:56:35 pushed ckpt_w180.pt,log_w180.txt +20:57:51 waited 76s for global r=778 +20:57:59 global r=778 load 4.275 -> 4.280 +20:58:33 step 184660 loss=3.0576 lr=1.42e-05 tok=24.324B dt=16048 wait=263 +20:59:38 step 184680 loss=3.0041 lr=1.41e-05 tok=24.327B dt=16113 wait=263 +21:00:42 step 184700 loss=3.1352 lr=1.41e-05 tok=24.330B dt=16177 wait=264 +21:01:45 step 184720 loss=3.1248 lr=1.41e-05 tok=24.332B dt=16240 wait=264 +21:02:50 step 184740 loss=2.9220 lr=1.41e-05 tok=24.335B dt=16304 wait=264 +21:03:53 step 184760 loss=3.1931 lr=1.41e-05 tok=24.337B dt=16368 wait=264 +21:04:56 step 184780 loss=2.9887 lr=1.41e-05 tok=24.340B dt=16431 wait=264 +21:05:59 step 184800 loss=3.0180 lr=1.40e-05 tok=24.343B dt=16494 wait=264 +21:06:14 pushed ckpt_w180.pt,log_w180.txt +21:07:30 waited 76s for global r=779 +21:07:39 global r=779 load 4.278 -> 4.280 +21:08:45 step 184820 loss=2.8689 lr=1.40e-05 tok=24.345B dt=16660 wait=264 +21:09:48 step 184840 loss=2.9236 lr=1.40e-05 tok=24.348B dt=16723 wait=265 +21:10:52 step 184860 loss=2.8984 lr=1.40e-05 tok=24.351B dt=16787 wait=265 +21:11:56 step 184880 loss=2.7671 lr=1.40e-05 tok=24.353B dt=16851 wait=265 +21:12:59 step 184900 loss=2.9174 lr=1.39e-05 tok=24.356B dt=16914 wait=265 +21:14:03 step 184920 loss=3.0021 lr=1.39e-05 tok=24.358B dt=16978 wait=265 +21:15:07 step 184940 loss=2.8969 lr=1.39e-05 tok=24.361B dt=17042 wait=265 +21:15:52 pushed ckpt_w180.pt,log_w180.txt +21:17:09 waited 77s for global r=780 +21:17:16 global r=780 load 4.280 -> 4.280 +21:17:50 step 184960 loss=3.0115 lr=1.39e-05 tok=24.364B dt=17204 wait=265 +21:18:54 step 184980 loss=3.0147 lr=1.39e-05 tok=24.366B dt=17269 wait=266 +21:19:58 step 185000 loss=2.9552 lr=1.39e-05 tok=24.369B dt=17333 wait=267 +21:21:02 step 185020 loss=3.1060 lr=1.38e-05 tok=24.372B dt=17397 wait=267 +21:22:06 step 185040 loss=2.8809 lr=1.38e-05 tok=24.374B dt=17461 wait=267 +21:23:11 step 185060 loss=3.0909 lr=1.38e-05 tok=24.377B dt=17526 wait=267 +21:24:15 step 185080 loss=2.9749 lr=1.38e-05 tok=24.379B dt=17590 wait=267 +21:25:19 step 185100 loss=3.0633 lr=1.38e-05 tok=24.382B dt=17654 wait=267 +21:25:34 pushed ckpt_w180.pt,log_w180.txt +21:26:51 waited 77s for global r=781 +21:27:07 global r=781 load 4.279 -> 4.282 +21:28:12 step 185120 loss=3.0889 lr=1.37e-05 tok=24.385B dt=17827 wait=268 +21:29:17 step 185140 loss=3.0783 lr=1.37e-05 tok=24.387B dt=17892 wait=268 +21:30:21 step 185160 loss=3.0129 lr=1.37e-05 tok=24.390B dt=17955 wait=268 +21:31:24 step 185180 loss=2.9333 lr=1.37e-05 tok=24.392B dt=18019 wait=268 +21:32:28 step 185200 loss=2.9449 lr=1.37e-05 tok=24.395B dt=18083 wait=268 +21:33:32 step 185220 loss=3.0259 lr=1.36e-05 tok=24.398B dt=18147 wait=268 +21:34:36 step 185240 loss=2.8871 lr=1.36e-05 tok=24.400B dt=18211 wait=268 +21:35:29 pushed ckpt_w180.pt,log_w180.txt +21:36:17 waited 47s for global r=782 +21:36:22 global r=782 load 4.278 -> 4.280 +21:36:56 step 185260 loss=3.0367 lr=1.36e-05 tok=24.403B dt=18350 wait=268 +21:38:00 step 185280 loss=2.8066 lr=1.36e-05 tok=24.406B dt=18415 wait=269 +21:39:03 step 185300 loss=2.8107 lr=1.36e-05 tok=24.408B dt=18478 wait=269 +21:40:07 step 185320 loss=2.8786 lr=1.36e-05 tok=24.411B dt=18542 wait=269 +21:41:11 step 185340 loss=2.8884 lr=1.35e-05 tok=24.413B dt=18606 wait=269 +21:42:14 step 185360 loss=2.8917 lr=1.35e-05 tok=24.416B dt=18669 wait=269 +21:43:18 step 185380 loss=2.8497 lr=1.35e-05 tok=24.419B dt=18733 wait=269 +21:44:21 step 185400 loss=2.9193 lr=1.35e-05 tok=24.421B dt=18796 wait=270 +21:44:33 pushed ckpt_w180.pt,log_w180.txt +21:45:56 waited 84s for global r=783 +21:46:07 global r=783 load 4.277 -> 4.279 +21:47:12 step 185420 loss=2.8335 lr=1.35e-05 tok=24.424B dt=18967 wait=270 +21:48:16 step 185440 loss=2.9105 lr=1.34e-05 tok=24.427B dt=19031 wait=270 +21:49:19 step 185460 loss=2.9445 lr=1.34e-05 tok=24.429B dt=19094 wait=270 +21:50:23 step 185480 loss=2.9445 lr=1.34e-05 tok=24.432B dt=19158 wait=270 +21:51:26 step 185500 loss=2.9601 lr=1.34e-05 tok=24.434B dt=19221 wait=270 +21:52:31 step 185520 loss=3.0118 lr=1.34e-05 tok=24.437B dt=19286 wait=271 +21:53:36 step 185540 loss=2.9671 lr=1.34e-05 tok=24.440B dt=19351 wait=271 +21:54:19 pushed ckpt_w180.pt,log_w180.txt +21:55:36 waited 76s for global r=784 +21:55:43 global r=784 load 4.281 -> 4.280 +21:56:17 step 185560 loss=3.0797 lr=1.33e-05 tok=24.442B dt=19512 wait=271 +21:57:21 step 185580 loss=2.9978 lr=1.33e-05 tok=24.445B dt=19576 wait=271 +21:58:24 step 185600 loss=2.9731 lr=1.33e-05 tok=24.448B dt=19639 wait=271 +21:59:28 step 185620 loss=2.7358 lr=1.33e-05 tok=24.450B dt=19703 wait=271 +22:00:32 step 185640 loss=2.8846 lr=1.33e-05 tok=24.453B dt=19767 wait=271 +22:01:36 step 185660 loss=2.9190 lr=1.32e-05 tok=24.455B dt=19831 wait=272 +22:02:39 step 185680 loss=2.9054 lr=1.32e-05 tok=24.458B dt=19894 wait=272 +22:03:42 step 185700 loss=2.9352 lr=1.32e-05 tok=24.461B dt=19957 wait=272 +22:03:55 pushed ckpt_w180.pt,log_w180.txt +22:05:11 waited 76s for global r=785 +22:05:19 global r=785 load 4.280 -> 4.279 +22:06:24 step 185720 loss=2.8975 lr=1.32e-05 tok=24.463B dt=20119 wait=272 +22:07:27 step 185740 loss=2.9272 lr=1.32e-05 tok=24.466B dt=20182 wait=272 +22:08:31 step 185760 loss=2.8715 lr=1.32e-05 tok=24.469B dt=20246 wait=272 +22:09:34 step 185780 loss=2.8682 lr=1.31e-05 tok=24.471B dt=20309 wait=272 +22:10:36 step 185800 loss=2.8469 lr=1.31e-05 tok=24.474B dt=20371 wait=273 +22:11:40 step 185820 loss=2.8878 lr=1.31e-05 tok=24.476B dt=20435 wait=273 +22:12:43 step 185840 loss=2.8094 lr=1.31e-05 tok=24.479B dt=20498 wait=273 +22:13:28 pushed ckpt_w180.pt,log_w180.txt +22:14:44 waited 76s for global r=786 +22:14:56 global r=786 load 4.280 -> 4.279 +22:15:29 step 185860 loss=2.8696 lr=1.31e-05 tok=24.482B dt=20664 wait=273 +22:16:32 step 185880 loss=2.8106 lr=1.30e-05 tok=24.484B dt=20727 wait=273 +22:17:34 step 185900 loss=2.8256 lr=1.30e-05 tok=24.487B dt=20789 wait=273 +22:18:38 step 185920 loss=2.8634 lr=1.30e-05 tok=24.489B dt=20853 wait=274 +22:19:41 step 185940 loss=2.7678 lr=1.30e-05 tok=24.492B dt=20916 wait=274 +22:20:44 step 185960 loss=2.8250 lr=1.30e-05 tok=24.495B dt=20979 wait=274 +22:21:47 step 185980 loss=2.8373 lr=1.29e-05 tok=24.497B dt=21042 wait=274 +22:22:50 step 186000 loss=2.7809 lr=1.29e-05 tok=24.500B dt=21105 wait=274 diff --git a/log_w181.txt b/log_w181.txt new file mode 100644 index 0000000000000000000000000000000000000000..5acc409f0fc43ccbf22ffe055f0a4d2fa5dc96b5 --- /dev/null +++ b/log_w181.txt @@ -0,0 +1,435 @@ +launch +15:23:52 name=w181 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 128/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:23:58 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:23:59 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:24:08 params 1.08B +15:24:10 resumed step=179850 seen=23.69B round=746 +15:24:10 stream salt=179850 +15:26:06 blocks compiled +15:26:06 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +15:31:12 step 179851 loss=3.1732 lr=1.85e-05 tok=23.687B dt=306 wait=280 +15:31:28 step 179852 loss=3.2012 lr=1.85e-05 tok=23.688B dt=322 wait=280 +15:31:54 step 179860 loss=3.3646 lr=1.85e-05 tok=23.689B dt=348 wait=281 +15:32:10 global r=747 load 4.279 -> 4.283 +15:33:16 step 180020 loss=3.2564 lr=1.85e-05 tok=23.691B dt=430 wait=281 +15:34:22 step 180040 loss=3.2959 lr=1.85e-05 tok=23.694B dt=496 wait=281 +15:35:27 step 180060 loss=3.2859 lr=1.84e-05 tok=23.697B dt=561 wait=281 +15:36:32 step 180080 loss=3.2333 lr=1.84e-05 tok=23.699B dt=626 wait=281 +15:37:37 step 180100 loss=3.1589 lr=1.84e-05 tok=23.702B dt=691 wait=281 +15:38:42 step 180120 loss=3.2432 lr=1.84e-05 tok=23.704B dt=756 wait=281 +15:39:47 step 180140 loss=3.0911 lr=1.84e-05 tok=23.707B dt=822 wait=282 +15:40:38 pushed ckpt_w181.pt,log_w181.txt +15:42:39 waited 121s for global r=748 +15:42:48 global r=748 load 4.283 -> 4.280 +15:43:23 step 180160 loss=3.1240 lr=1.84e-05 tok=23.710B dt=1037 wait=282 +15:44:28 step 180180 loss=3.0832 lr=1.84e-05 tok=23.712B dt=1102 wait=282 +15:45:33 step 180200 loss=3.0483 lr=1.84e-05 tok=23.715B dt=1167 wait=282 +15:46:39 step 180220 loss=3.1483 lr=1.84e-05 tok=23.717B dt=1233 wait=282 +15:47:44 step 180240 loss=3.0812 lr=1.84e-05 tok=23.720B dt=1298 wait=282 +15:48:50 step 180260 loss=3.2058 lr=1.84e-05 tok=23.723B dt=1364 wait=282 +15:49:55 step 180280 loss=3.1109 lr=1.84e-05 tok=23.725B dt=1429 wait=282 +15:51:00 step 180300 loss=3.0637 lr=1.84e-05 tok=23.728B dt=1494 wait=283 +15:51:12 pushed ckpt_w181.pt,log_w181.txt +15:54:13 waited 181s for global r=749 +15:54:22 global r=749 load 4.286 -> 4.282 +15:55:28 step 180320 loss=3.0427 lr=1.84e-05 tok=23.731B dt=1762 wait=283 +15:56:33 step 180340 loss=3.0305 lr=1.84e-05 tok=23.733B dt=1827 wait=283 +15:57:37 step 180360 loss=3.1180 lr=1.84e-05 tok=23.736B dt=1892 wait=283 +15:58:42 step 180380 loss=2.9995 lr=1.83e-05 tok=23.738B dt=1956 wait=283 +15:59:45 step 180400 loss=3.0195 lr=1.83e-05 tok=23.741B dt=2020 wait=283 +16:00:50 step 180420 loss=3.0960 lr=1.83e-05 tok=23.744B dt=2084 wait=283 +16:01:54 step 180440 loss=3.0106 lr=1.83e-05 tok=23.746B dt=2148 wait=284 +16:02:38 pushed ckpt_w181.pt,log_w181.txt +16:05:39 waited 181s for global r=750 +16:05:48 global r=750 load 4.281 -> 4.283 +16:06:22 step 180460 loss=3.1140 lr=1.83e-05 tok=23.749B dt=2416 wait=284 +16:07:26 step 180480 loss=2.9263 lr=1.83e-05 tok=23.752B dt=2480 wait=284 +16:08:31 step 180500 loss=3.0092 lr=1.83e-05 tok=23.754B dt=2546 wait=284 +16:09:36 step 180520 loss=3.1536 lr=1.83e-05 tok=23.757B dt=2610 wait=284 +16:10:41 step 180540 loss=3.1157 lr=1.83e-05 tok=23.759B dt=2675 wait=284 +16:11:45 step 180560 loss=3.1407 lr=1.83e-05 tok=23.762B dt=2739 wait=284 +16:12:49 step 180580 loss=3.1008 lr=1.83e-05 tok=23.765B dt=2804 wait=285 +16:13:54 step 180600 loss=3.1324 lr=1.83e-05 tok=23.767B dt=2868 wait=285 +16:14:05 pushed ckpt_w181.pt,log_w181.txt +16:15:51 waited 106s for global r=751 +16:15:57 global r=751 load 4.283 -> 4.282 +16:17:02 step 180620 loss=3.1355 lr=1.83e-05 tok=23.770B dt=3056 wait=285 +16:18:06 step 180640 loss=3.1557 lr=1.83e-05 tok=23.773B dt=3120 wait=285 +16:19:10 step 180660 loss=3.0432 lr=1.83e-05 tok=23.775B dt=3185 wait=285 +16:20:15 step 180680 loss=3.0118 lr=1.82e-05 tok=23.778B dt=3249 wait=285 +16:21:19 step 180700 loss=3.0025 lr=1.82e-05 tok=23.780B dt=3313 wait=285 +16:22:23 step 180720 loss=3.1874 lr=1.82e-05 tok=23.783B dt=3377 wait=286 +16:23:27 step 180740 loss=3.0836 lr=1.82e-05 tok=23.786B dt=3441 wait=286 +16:24:14 pushed ckpt_w181.pt,log_w181.txt +16:27:00 waited 166s for global r=752 +16:27:05 global r=752 load 4.284 -> 4.281 +16:27:38 step 180760 loss=3.1260 lr=1.82e-05 tok=23.788B dt=3692 wait=286 +16:28:42 step 180780 loss=2.9875 lr=1.82e-05 tok=23.791B dt=3756 wait=286 +16:29:47 step 180800 loss=3.0009 lr=1.82e-05 tok=23.794B dt=3821 wait=286 +16:30:51 step 180820 loss=3.1023 lr=1.82e-05 tok=23.796B dt=3885 wait=286 +launch +16:31:28 name=w181 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 128/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:31:32 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:31:32 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:31:42 params 1.08B +16:31:43 resumed step=180750 seen=23.82B round=752 +16:31:43 stream salt=180750 +16:33:30 blocks compiled +16:33:30 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:38:18 step 180751 loss=2.8364 lr=1.78e-05 tok=23.819B dt=287 wait=278 +16:38:25 step 180752 loss=2.9614 lr=1.78e-05 tok=23.819B dt=294 wait=278 +16:38:51 step 180760 loss=3.3742 lr=1.78e-05 tok=23.820B dt=320 wait=278 +16:39:06 global r=753 load 4.278 -> 4.284 +16:40:12 step 180920 loss=3.0513 lr=1.76e-05 tok=23.822B dt=402 wait=278 +16:41:18 step 180940 loss=3.1671 lr=1.76e-05 tok=23.825B dt=468 wait=278 +16:42:23 step 180960 loss=3.2008 lr=1.76e-05 tok=23.828B dt=533 wait=278 +16:43:28 step 180980 loss=3.1609 lr=1.76e-05 tok=23.830B dt=598 wait=279 +16:44:33 step 181000 loss=3.1253 lr=1.75e-05 tok=23.833B dt=663 wait=279 +16:45:38 step 181020 loss=3.2314 lr=1.75e-05 tok=23.835B dt=728 wait=279 +16:46:44 step 181040 loss=2.9590 lr=1.75e-05 tok=23.838B dt=794 wait=279 +16:47:30 pushed ckpt_w181.pt,log_w181.txt +16:51:07 step 181060 loss=3.1762 lr=1.75e-05 tok=23.841B dt=1056 wait=279 +16:51:14 global r=754 load 4.280 -> 4.281 +16:51:47 step 181060 loss=3.1735 lr=1.75e-05 tok=23.842B dt=1096 wait=279 +16:52:51 step 181080 loss=3.1002 lr=1.75e-05 tok=23.845B dt=1160 wait=279 +16:53:56 step 181100 loss=3.1408 lr=1.75e-05 tok=23.847B dt=1225 wait=279 +16:55:01 step 181120 loss=3.1689 lr=1.74e-05 tok=23.850B dt=1290 wait=280 +16:56:06 step 181140 loss=2.9874 lr=1.74e-05 tok=23.852B dt=1355 wait=280 +16:57:11 step 181160 loss=3.1207 lr=1.74e-05 tok=23.855B dt=1420 wait=280 +16:58:14 step 181180 loss=3.1504 lr=1.74e-05 tok=23.858B dt=1484 wait=280 +16:59:19 step 181200 loss=3.0859 lr=1.74e-05 tok=23.860B dt=1548 wait=280 +16:59:30 pushed ckpt_w181.pt,log_w181.txt +17:01:01 waited 91s for global r=755 +17:01:19 global r=755 load 4.284 -> 4.280 +17:02:25 step 181220 loss=3.1449 lr=1.73e-05 tok=23.863B dt=1734 wait=280 +17:03:29 step 181240 loss=3.2253 lr=1.73e-05 tok=23.866B dt=1799 wait=281 +17:04:33 step 181260 loss=3.1356 lr=1.73e-05 tok=23.868B dt=1863 wait=281 +17:05:36 step 181280 loss=3.0958 lr=1.73e-05 tok=23.871B dt=1926 wait=281 +17:06:41 step 181300 loss=2.9701 lr=1.73e-05 tok=23.873B dt=1990 wait=281 +17:07:45 step 181320 loss=3.0901 lr=1.73e-05 tok=23.876B dt=2054 wait=281 +17:08:48 step 181340 loss=3.0221 lr=1.72e-05 tok=23.879B dt=2118 wait=281 +17:09:32 pushed ckpt_w181.pt,log_w181.txt +17:11:03 waited 91s for global r=756 +17:11:09 global r=756 load 4.280 -> 4.281 +17:11:43 step 181360 loss=3.1101 lr=1.72e-05 tok=23.881B dt=2292 wait=281 +17:12:47 step 181380 loss=2.9512 lr=1.72e-05 tok=23.884B dt=2357 wait=282 +17:13:51 step 181400 loss=3.0851 lr=1.72e-05 tok=23.887B dt=2421 wait=282 +17:14:56 step 181420 loss=3.0970 lr=1.72e-05 tok=23.889B dt=2485 wait=282 +17:16:00 step 181440 loss=3.1101 lr=1.71e-05 tok=23.892B dt=2550 wait=282 +17:17:04 step 181460 loss=3.2020 lr=1.71e-05 tok=23.894B dt=2614 wait=282 +17:18:08 step 181480 loss=3.1549 lr=1.71e-05 tok=23.897B dt=2677 wait=282 +17:19:12 step 181500 loss=3.1471 lr=1.71e-05 tok=23.900B dt=2741 wait=282 +17:19:29 pushed ckpt_w181.pt,log_w181.txt +17:22:31 waited 181s for global r=757 +17:22:37 global r=757 load 4.281 -> 4.282 +17:23:43 step 181520 loss=3.0532 lr=1.71e-05 tok=23.902B dt=3012 wait=282 +17:24:47 step 181540 loss=3.2296 lr=1.70e-05 tok=23.905B dt=3077 wait=283 +17:25:51 step 181560 loss=3.2161 lr=1.70e-05 tok=23.908B dt=3141 wait=283 +17:26:55 step 181580 loss=3.2004 lr=1.70e-05 tok=23.910B dt=3205 wait=283 +17:28:00 step 181600 loss=3.1799 lr=1.70e-05 tok=23.913B dt=3270 wait=283 +17:29:05 step 181620 loss=3.2664 lr=1.70e-05 tok=23.915B dt=3335 wait=283 +17:30:10 step 181640 loss=3.1598 lr=1.70e-05 tok=23.918B dt=3399 wait=283 +17:30:54 pushed ckpt_w181.pt,log_w181.txt +17:32:55 waited 121s for global r=758 +17:33:02 global r=758 load 4.277 -> 4.283 +17:33:36 step 181660 loss=3.1934 lr=1.69e-05 tok=23.921B dt=3605 wait=283 +17:34:41 step 181680 loss=3.1570 lr=1.69e-05 tok=23.923B dt=3671 wait=284 +17:35:46 step 181700 loss=2.9665 lr=1.69e-05 tok=23.926B dt=3735 wait=284 +17:36:50 step 181720 loss=3.0397 lr=1.69e-05 tok=23.929B dt=3800 wait=284 +17:37:54 step 181740 loss=3.0297 lr=1.69e-05 tok=23.931B dt=3864 wait=284 +17:38:59 step 181760 loss=2.9522 lr=1.68e-05 tok=23.934B dt=3929 wait=284 +17:40:04 step 181780 loss=3.0076 lr=1.68e-05 tok=23.936B dt=3993 wait=284 +17:41:08 step 181800 loss=3.0635 lr=1.68e-05 tok=23.939B dt=4057 wait=285 +17:41:24 pushed ckpt_w181.pt,log_w181.txt +17:42:40 waited 76s for global r=759 +17:42:52 global r=759 load 4.283 -> 4.283 +17:43:58 step 181820 loss=2.9456 lr=1.68e-05 tok=23.942B dt=4227 wait=285 +17:45:03 step 181840 loss=2.9148 lr=1.68e-05 tok=23.944B dt=4293 wait=285 +17:46:07 step 181860 loss=2.9899 lr=1.68e-05 tok=23.947B dt=4357 wait=285 +17:47:12 step 181880 loss=2.9223 lr=1.67e-05 tok=23.949B dt=4422 wait=285 +17:48:17 step 181900 loss=3.0754 lr=1.67e-05 tok=23.952B dt=4486 wait=285 +17:49:21 step 181920 loss=3.0944 lr=1.67e-05 tok=23.955B dt=4551 wait=285 +17:50:26 step 181940 loss=3.0370 lr=1.67e-05 tok=23.957B dt=4616 wait=286 +17:51:10 pushed ckpt_w181.pt,log_w181.txt +17:52:56 waited 106s for global r=760 +17:53:07 global r=760 load 4.284 -> 4.281 +17:53:40 step 181960 loss=3.0614 lr=1.67e-05 tok=23.960B dt=4810 wait=286 +17:54:44 step 181980 loss=2.9389 lr=1.66e-05 tok=23.963B dt=4874 wait=286 +17:55:49 step 182000 loss=2.8661 lr=1.66e-05 tok=23.965B dt=4939 wait=286 +17:56:54 step 182020 loss=3.0537 lr=1.66e-05 tok=23.968B dt=5003 wait=286 +17:57:58 step 182040 loss=2.9005 lr=1.66e-05 tok=23.970B dt=5068 wait=286 +17:59:02 step 182060 loss=2.9645 lr=1.66e-05 tok=23.973B dt=5132 wait=286 +18:00:06 step 182080 loss=2.9255 lr=1.65e-05 tok=23.976B dt=5195 wait=287 +18:01:10 step 182100 loss=2.9272 lr=1.65e-05 tok=23.978B dt=5259 wait=287 +18:01:22 pushed ckpt_w181.pt,log_w181.txt +18:02:53 waited 91s for global r=761 +18:03:11 global r=761 load 4.286 -> 4.281 +18:04:15 step 182120 loss=2.8521 lr=1.65e-05 tok=23.981B dt=5445 wait=287 +18:05:19 step 182140 loss=2.8744 lr=1.65e-05 tok=23.984B dt=5509 wait=287 +18:06:23 step 182160 loss=2.8442 lr=1.65e-05 tok=23.986B dt=5573 wait=287 +18:07:27 step 182180 loss=2.8559 lr=1.65e-05 tok=23.989B dt=5637 wait=287 +18:08:31 step 182200 loss=2.9036 lr=1.64e-05 tok=23.991B dt=5700 wait=287 +18:09:35 step 182220 loss=2.9962 lr=1.64e-05 tok=23.994B dt=5764 wait=288 +18:10:39 step 182240 loss=2.8538 lr=1.64e-05 tok=23.997B dt=5828 wait=288 +18:11:24 pushed ckpt_w181.pt,log_w181.txt +18:12:39 waited 76s for global r=762 +18:12:49 global r=762 load 4.282 -> 4.281 +18:13:22 step 182260 loss=2.9598 lr=1.64e-05 tok=23.999B dt=5991 wait=288 +18:14:26 step 182280 loss=2.8738 lr=1.64e-05 tok=24.002B dt=6055 wait=288 +18:15:30 step 182300 loss=2.8653 lr=1.63e-05 tok=24.005B dt=6119 wait=288 +18:16:33 step 182320 loss=3.0002 lr=1.63e-05 tok=24.007B dt=6183 wait=288 +18:17:37 step 182340 loss=2.9188 lr=1.63e-05 tok=24.010B dt=6246 wait=288 +18:18:41 step 182360 loss=3.0659 lr=1.63e-05 tok=24.012B dt=6311 wait=288 +18:19:45 step 182380 loss=2.9640 lr=1.63e-05 tok=24.015B dt=6374 wait=289 +18:20:48 step 182400 loss=2.9230 lr=1.63e-05 tok=24.018B dt=6438 wait=289 +18:21:05 pushed ckpt_w181.pt,log_w181.txt +18:24:07 waited 181s for global r=763 +18:24:14 global r=763 load 4.282 -> 4.280 +18:25:18 step 182420 loss=2.9925 lr=1.62e-05 tok=24.020B dt=6708 wait=289 +18:26:21 step 182440 loss=2.9283 lr=1.62e-05 tok=24.023B dt=6771 wait=289 +18:27:25 step 182460 loss=2.9415 lr=1.62e-05 tok=24.025B dt=6834 wait=289 +18:28:28 step 182480 loss=2.8308 lr=1.62e-05 tok=24.028B dt=6898 wait=289 +18:29:31 step 182500 loss=2.9335 lr=1.62e-05 tok=24.031B dt=6961 wait=289 +18:30:35 step 182520 loss=2.8436 lr=1.61e-05 tok=24.033B dt=7025 wait=290 +18:31:39 step 182540 loss=2.7852 lr=1.61e-05 tok=24.036B dt=7089 wait=290 +18:32:24 pushed ckpt_w181.pt,log_w181.txt +18:35:58 step 182560 loss=2.9051 lr=1.61e-05 tok=24.039B dt=7348 wait=290 +18:36:17 global r=764 load 4.283 -> 4.280 +18:36:49 step 182560 loss=2.8871 lr=1.61e-05 tok=24.040B dt=7398 wait=290 +18:37:53 step 182580 loss=2.7835 lr=1.61e-05 tok=24.043B dt=7463 wait=290 +18:38:58 step 182600 loss=2.7718 lr=1.61e-05 tok=24.045B dt=7528 wait=290 +18:40:02 step 182620 loss=2.8885 lr=1.61e-05 tok=24.048B dt=7591 wait=290 +18:41:05 step 182640 loss=2.7928 lr=1.60e-05 tok=24.050B dt=7655 wait=290 +18:42:10 step 182660 loss=2.8562 lr=1.60e-05 tok=24.053B dt=7719 wait=291 +18:43:14 step 182680 loss=2.8123 lr=1.60e-05 tok=24.056B dt=7784 wait=291 +18:44:19 step 182700 loss=2.7119 lr=1.60e-05 tok=24.058B dt=7848 wait=291 +18:44:31 pushed ckpt_w181.pt,log_w181.txt +18:45:16 waited 45s for global r=765 +18:45:24 global r=765 load 4.283 -> 4.280 +18:46:30 step 182720 loss=2.5467 lr=1.60e-05 tok=24.061B dt=7979 wait=291 +18:47:33 step 182740 loss=2.8571 lr=1.59e-05 tok=24.064B dt=8043 wait=291 +18:48:38 step 182760 loss=2.3343 lr=1.59e-05 tok=24.066B dt=8108 wait=291 +18:49:42 step 182780 loss=2.7171 lr=1.59e-05 tok=24.069B dt=8172 wait=291 +18:50:46 step 182800 loss=2.6913 lr=1.59e-05 tok=24.071B dt=8235 wait=292 +18:51:50 step 182820 loss=2.8189 lr=1.59e-05 tok=24.074B dt=8299 wait=292 +18:52:54 step 182840 loss=2.6297 lr=1.58e-05 tok=24.077B dt=8363 wait=292 +18:53:39 pushed ckpt_w181.pt,log_w181.txt +18:54:55 waited 76s for global r=766 +18:55:05 global r=766 load 4.284 -> 4.281 +18:55:38 step 182860 loss=2.7783 lr=1.58e-05 tok=24.079B dt=8528 wait=292 +18:56:42 step 182880 loss=2.6936 lr=1.58e-05 tok=24.082B dt=8591 wait=292 +18:57:46 step 182900 loss=2.7128 lr=1.58e-05 tok=24.084B dt=8655 wait=292 +18:58:49 step 182920 loss=2.7903 lr=1.58e-05 tok=24.087B dt=8719 wait=292 +18:59:53 step 182940 loss=2.7089 lr=1.58e-05 tok=24.090B dt=8783 wait=292 +19:00:57 step 182960 loss=2.8194 lr=1.57e-05 tok=24.092B dt=8846 wait=293 +19:02:01 step 182980 loss=2.6707 lr=1.57e-05 tok=24.095B dt=8911 wait=293 +19:03:05 step 183000 loss=2.7726 lr=1.57e-05 tok=24.098B dt=8975 wait=293 +19:03:19 pushed ckpt_w181.pt,log_w181.txt +19:04:35 waited 76s for global r=767 +19:04:45 global r=767 load 4.286 -> 4.283 +19:05:51 step 183020 loss=2.7481 lr=1.57e-05 tok=24.100B dt=9141 wait=293 +19:06:55 step 183040 loss=2.7534 lr=1.57e-05 tok=24.103B dt=9205 wait=293 +19:08:00 step 183060 loss=2.7576 lr=1.56e-05 tok=24.105B dt=9270 wait=293 +19:09:04 step 183080 loss=2.7499 lr=1.56e-05 tok=24.108B dt=9333 wait=293 +19:10:08 step 183100 loss=2.7037 lr=1.56e-05 tok=24.111B dt=9397 wait=294 +19:11:12 step 183120 loss=2.7810 lr=1.56e-05 tok=24.113B dt=9462 wait=294 +19:12:15 step 183140 loss=2.8110 lr=1.56e-05 tok=24.116B dt=9525 wait=294 +19:13:00 pushed ckpt_w181.pt,log_w181.txt +19:14:00 waited 61s for global r=768 +19:14:06 global r=768 load 4.280 -> 4.284 +19:14:40 step 183160 loss=3.0330 lr=1.56e-05 tok=24.119B dt=9669 wait=294 +19:15:44 step 183180 loss=2.7362 lr=1.55e-05 tok=24.121B dt=9734 wait=294 +19:16:48 step 183200 loss=2.7485 lr=1.55e-05 tok=24.124B dt=9797 wait=294 +19:17:52 step 183220 loss=2.9203 lr=1.55e-05 tok=24.126B dt=9861 wait=295 +19:18:57 step 183240 loss=2.7299 lr=1.55e-05 tok=24.129B dt=9926 wait=295 +19:20:01 step 183260 loss=2.9067 lr=1.55e-05 tok=24.132B dt=9990 wait=295 +19:21:05 step 183280 loss=2.7960 lr=1.54e-05 tok=24.134B dt=10055 wait=295 +19:22:09 step 183300 loss=2.8544 lr=1.54e-05 tok=24.137B dt=10119 wait=295 +19:22:23 pushed ckpt_w181.pt,log_w181.txt +19:23:24 waited 61s for global r=769 +19:23:31 global r=769 load 4.290 -> 4.282 +19:24:37 step 183320 loss=2.8298 lr=1.54e-05 tok=24.140B dt=10267 wait=295 +19:25:42 step 183340 loss=2.8062 lr=1.54e-05 tok=24.142B dt=10331 wait=295 +19:26:46 step 183360 loss=2.8599 lr=1.54e-05 tok=24.145B dt=10395 wait=296 +19:27:50 step 183380 loss=2.7386 lr=1.53e-05 tok=24.147B dt=10459 wait=296 +19:28:54 step 183400 loss=2.8938 lr=1.53e-05 tok=24.150B dt=10523 wait=296 +19:29:59 step 183420 loss=2.5574 lr=1.53e-05 tok=24.153B dt=10588 wait=296 +19:31:03 step 183440 loss=2.7566 lr=1.53e-05 tok=24.155B dt=10652 wait=296 +19:31:46 pushed ckpt_w181.pt,log_w181.txt +19:32:47 waited 61s for global r=770 +19:32:53 global r=770 load 4.288 -> 4.281 +19:33:26 step 183460 loss=2.9057 lr=1.53e-05 tok=24.158B dt=10796 wait=296 +19:34:30 step 183480 loss=2.7276 lr=1.53e-05 tok=24.161B dt=10860 wait=296 +19:35:34 step 183500 loss=2.7826 lr=1.52e-05 tok=24.163B dt=10924 wait=297 +19:36:38 step 183520 loss=2.8662 lr=1.52e-05 tok=24.166B dt=10988 wait=297 +19:37:42 step 183540 loss=2.7842 lr=1.52e-05 tok=24.168B dt=11052 wait=297 +19:38:46 step 183560 loss=2.8691 lr=1.52e-05 tok=24.171B dt=11116 wait=297 +19:39:51 step 183580 loss=2.8090 lr=1.52e-05 tok=24.174B dt=11180 wait=297 +19:40:55 step 183600 loss=2.8083 lr=1.51e-05 tok=24.176B dt=11244 wait=297 +19:41:08 pushed ckpt_w181.pt,log_w181.txt +19:44:51 global r=771 load 4.282 -> 4.282 +19:45:55 step 183620 loss=2.6865 lr=1.51e-05 tok=24.180B dt=11544 wait=297 +19:46:59 step 183640 loss=2.8032 lr=1.51e-05 tok=24.183B dt=11608 wait=297 +19:48:03 step 183660 loss=2.7901 lr=1.51e-05 tok=24.185B dt=11672 wait=298 +19:49:06 step 183680 loss=2.7283 lr=1.51e-05 tok=24.188B dt=11736 wait=298 +19:50:11 step 183700 loss=2.6383 lr=1.51e-05 tok=24.191B dt=11801 wait=298 +19:51:16 step 183720 loss=2.7821 lr=1.50e-05 tok=24.193B dt=11865 wait=298 +19:52:20 step 183740 loss=2.7005 lr=1.50e-05 tok=24.196B dt=11929 wait=298 +19:53:05 pushed ckpt_w181.pt,log_w181.txt +19:56:39 step 183760 loss=2.8203 lr=1.50e-05 tok=24.199B dt=12189 wait=298 +19:56:45 global r=772 load 4.284 -> 4.280 +19:57:17 step 183760 loss=2.8576 lr=1.50e-05 tok=24.200B dt=12226 wait=298 +19:58:20 step 183780 loss=2.7448 lr=1.50e-05 tok=24.202B dt=12290 wait=299 +19:59:24 step 183800 loss=2.6548 lr=1.50e-05 tok=24.205B dt=12354 wait=299 +20:00:27 step 183820 loss=2.8588 lr=1.49e-05 tok=24.208B dt=12417 wait=299 +20:01:31 step 183840 loss=2.8380 lr=1.49e-05 tok=24.210B dt=12480 wait=299 +20:02:35 step 183860 loss=2.8577 lr=1.49e-05 tok=24.213B dt=12544 wait=299 +20:03:38 step 183880 loss=2.8306 lr=1.49e-05 tok=24.216B dt=12608 wait=299 +20:04:42 step 183900 loss=3.0023 lr=1.49e-05 tok=24.218B dt=12671 wait=299 +20:04:54 pushed ckpt_w181.pt,log_w181.txt +20:07:55 waited 181s for global r=773 +20:08:04 global r=773 load 4.280 -> 4.282 +20:09:08 step 183920 loss=2.8291 lr=1.49e-05 tok=24.221B dt=12938 wait=300 +20:10:12 step 183940 loss=3.0029 lr=1.48e-05 tok=24.223B dt=13002 wait=300 +20:11:16 step 183960 loss=3.0278 lr=1.48e-05 tok=24.226B dt=13065 wait=300 +20:12:19 step 183980 loss=2.9482 lr=1.48e-05 tok=24.229B dt=13129 wait=300 +20:13:23 step 184000 loss=2.8572 lr=1.48e-05 tok=24.231B dt=13193 wait=300 +20:14:27 step 184020 loss=3.0003 lr=1.48e-05 tok=24.234B dt=13257 wait=300 +20:15:31 step 184040 loss=2.9454 lr=1.47e-05 tok=24.237B dt=13321 wait=301 +20:16:16 pushed ckpt_w181.pt,log_w181.txt +20:18:48 waited 151s for global r=774 +20:18:54 global r=774 load 4.284 -> 4.282 +20:19:27 step 184060 loss=3.0685 lr=1.47e-05 tok=24.239B dt=13556 wait=301 +20:20:31 step 184080 loss=2.8987 lr=1.47e-05 tok=24.242B dt=13621 wait=301 +20:21:35 step 184100 loss=3.0484 lr=1.47e-05 tok=24.244B dt=13684 wait=301 +20:22:39 step 184120 loss=3.0378 lr=1.47e-05 tok=24.247B dt=13748 wait=301 +20:23:43 step 184140 loss=2.9296 lr=1.46e-05 tok=24.250B dt=13813 wait=301 +20:24:47 step 184160 loss=3.0043 lr=1.46e-05 tok=24.252B dt=13876 wait=301 +20:25:51 step 184180 loss=3.0191 lr=1.46e-05 tok=24.255B dt=13940 wait=301 +20:26:54 step 184200 loss=3.1682 lr=1.46e-05 tok=24.257B dt=14004 wait=302 +20:27:08 pushed ckpt_w181.pt,log_w181.txt +20:28:39 waited 91s for global r=775 +20:28:49 global r=775 load 4.280 -> 4.281 +20:29:55 step 184220 loss=3.0482 lr=1.46e-05 tok=24.260B dt=14184 wait=302 +20:30:58 step 184240 loss=3.0872 lr=1.46e-05 tok=24.263B dt=14248 wait=302 +20:32:02 step 184260 loss=2.9455 lr=1.45e-05 tok=24.265B dt=14312 wait=302 +20:33:06 step 184280 loss=2.9543 lr=1.45e-05 tok=24.268B dt=14376 wait=302 +20:34:10 step 184300 loss=3.0055 lr=1.45e-05 tok=24.271B dt=14439 wait=302 +20:35:14 step 184320 loss=3.1337 lr=1.45e-05 tok=24.273B dt=14503 wait=303 +20:36:17 step 184340 loss=2.8546 lr=1.45e-05 tok=24.276B dt=14567 wait=303 +20:37:01 pushed ckpt_w181.pt,log_w181.txt +20:38:16 waited 76s for global r=776 +20:38:22 global r=776 load 4.283 -> 4.281 +20:38:55 step 184360 loss=3.0203 lr=1.44e-05 tok=24.278B dt=14725 wait=303 +20:39:59 step 184380 loss=2.8410 lr=1.44e-05 tok=24.281B dt=14788 wait=303 +20:41:02 step 184400 loss=2.9608 lr=1.44e-05 tok=24.284B dt=14852 wait=303 +20:42:05 step 184420 loss=3.0531 lr=1.44e-05 tok=24.286B dt=14915 wait=303 +20:43:08 step 184440 loss=3.0366 lr=1.44e-05 tok=24.289B dt=14978 wait=303 +20:44:12 step 184460 loss=3.0568 lr=1.44e-05 tok=24.292B dt=15041 wait=304 +20:45:15 step 184480 loss=3.0835 lr=1.43e-05 tok=24.294B dt=15105 wait=304 +20:46:19 step 184500 loss=2.9615 lr=1.43e-05 tok=24.297B dt=15169 wait=304 +20:46:31 pushed ckpt_w181.pt,log_w181.txt +20:48:02 waited 91s for global r=777 +20:48:07 global r=777 load 4.274 -> 4.280 +20:49:13 step 184520 loss=3.0712 lr=1.43e-05 tok=24.299B dt=15342 wait=304 +20:50:17 step 184540 loss=3.0894 lr=1.43e-05 tok=24.302B dt=15407 wait=304 +20:51:22 step 184560 loss=3.0567 lr=1.43e-05 tok=24.305B dt=15472 wait=304 +20:52:26 step 184580 loss=3.1145 lr=1.42e-05 tok=24.307B dt=15536 wait=304 +20:53:31 step 184600 loss=3.1248 lr=1.42e-05 tok=24.310B dt=15600 wait=305 +20:54:35 step 184620 loss=3.1593 lr=1.42e-05 tok=24.313B dt=15665 wait=305 +20:55:40 step 184640 loss=3.0753 lr=1.42e-05 tok=24.315B dt=15729 wait=305 +20:56:24 pushed ckpt_w181.pt,log_w181.txt +20:57:55 waited 91s for global r=778 +20:58:01 global r=778 load 4.277 -> 4.280 +20:58:35 step 184660 loss=3.3416 lr=1.42e-05 tok=24.318B dt=15905 wait=305 +20:59:39 step 184680 loss=3.2008 lr=1.41e-05 tok=24.320B dt=15969 wait=305 +21:00:44 step 184700 loss=3.2133 lr=1.41e-05 tok=24.323B dt=16034 wait=305 +21:01:49 step 184720 loss=3.2761 lr=1.41e-05 tok=24.326B dt=16099 wait=305 +21:02:54 step 184740 loss=3.1173 lr=1.41e-05 tok=24.328B dt=16164 wait=306 +21:03:58 step 184760 loss=3.1672 lr=1.41e-05 tok=24.331B dt=16228 wait=306 +21:05:03 step 184780 loss=3.1003 lr=1.41e-05 tok=24.334B dt=16292 wait=306 +21:06:07 step 184800 loss=3.0880 lr=1.40e-05 tok=24.336B dt=16356 wait=306 +21:06:19 pushed ckpt_w181.pt,log_w181.txt +21:07:34 waited 76s for global r=779 +21:07:40 global r=779 load 4.278 -> 4.280 +21:08:45 step 184820 loss=3.1892 lr=1.40e-05 tok=24.339B dt=16514 wait=306 +21:09:48 step 184840 loss=3.0764 lr=1.40e-05 tok=24.341B dt=16578 wait=306 +21:10:52 step 184860 loss=3.0518 lr=1.40e-05 tok=24.344B dt=16642 wait=307 +21:11:56 step 184880 loss=3.0091 lr=1.40e-05 tok=24.347B dt=16706 wait=307 +21:13:00 step 184900 loss=3.0248 lr=1.39e-05 tok=24.349B dt=16769 wait=307 +21:14:04 step 184920 loss=3.0933 lr=1.39e-05 tok=24.352B dt=16833 wait=307 +21:15:07 step 184940 loss=2.9773 lr=1.39e-05 tok=24.354B dt=16897 wait=307 +21:15:52 pushed ckpt_w181.pt,log_w181.txt +21:17:07 waited 76s for global r=780 +21:17:15 global r=780 load 4.279 -> 4.280 +21:17:48 step 184960 loss=3.0708 lr=1.39e-05 tok=24.357B dt=17057 wait=307 +21:18:51 step 184980 loss=3.0474 lr=1.39e-05 tok=24.360B dt=17121 wait=308 +21:19:55 step 185000 loss=3.1119 lr=1.39e-05 tok=24.362B dt=17184 wait=308 +21:20:59 step 185020 loss=3.2607 lr=1.38e-05 tok=24.365B dt=17249 wait=308 +21:22:04 step 185040 loss=3.1870 lr=1.38e-05 tok=24.368B dt=17313 wait=308 +21:23:08 step 185060 loss=3.2433 lr=1.38e-05 tok=24.370B dt=17378 wait=308 +21:24:12 step 185080 loss=3.1753 lr=1.38e-05 tok=24.373B dt=17442 wait=308 +21:25:17 step 185100 loss=3.2013 lr=1.38e-05 tok=24.375B dt=17506 wait=308 +21:25:31 pushed ckpt_w181.pt,log_w181.txt +21:26:47 waited 76s for global r=781 +21:26:55 global r=781 load 4.280 -> 4.282 +21:28:01 step 185120 loss=3.2155 lr=1.37e-05 tok=24.378B dt=17671 wait=309 +21:29:05 step 185140 loss=3.1586 lr=1.37e-05 tok=24.381B dt=17734 wait=309 +21:30:09 step 185160 loss=3.1512 lr=1.37e-05 tok=24.383B dt=17798 wait=309 +21:31:13 step 185180 loss=3.1971 lr=1.37e-05 tok=24.386B dt=17862 wait=309 +21:32:18 step 185200 loss=3.0339 lr=1.37e-05 tok=24.389B dt=17928 wait=309 +21:33:22 step 185220 loss=3.1915 lr=1.36e-05 tok=24.391B dt=17991 wait=309 +21:34:26 step 185240 loss=3.0552 lr=1.36e-05 tok=24.394B dt=18055 wait=309 +21:35:09 pushed ckpt_w181.pt,log_w181.txt +21:36:25 waited 76s for global r=782 +21:36:31 global r=782 load 4.282 -> 4.280 +21:37:04 step 185260 loss=3.1733 lr=1.36e-05 tok=24.396B dt=18214 wait=310 +21:38:09 step 185280 loss=3.0347 lr=1.36e-05 tok=24.399B dt=18278 wait=310 +21:39:13 step 185300 loss=2.9790 lr=1.36e-05 tok=24.402B dt=18342 wait=310 +21:40:17 step 185320 loss=3.0452 lr=1.36e-05 tok=24.404B dt=18406 wait=310 +21:41:21 step 185340 loss=2.9467 lr=1.35e-05 tok=24.407B dt=18471 wait=310 +21:42:25 step 185360 loss=3.0792 lr=1.35e-05 tok=24.410B dt=18535 wait=310 +21:43:30 step 185380 loss=3.0524 lr=1.35e-05 tok=24.412B dt=18599 wait=311 +21:44:34 step 185400 loss=2.9217 lr=1.35e-05 tok=24.415B dt=18664 wait=311 +21:44:47 pushed ckpt_w181.pt,log_w181.txt +21:46:02 waited 76s for global r=783 +21:46:10 global r=783 load 4.281 -> 4.279 +21:47:16 step 185420 loss=2.9620 lr=1.35e-05 tok=24.417B dt=18825 wait=311 +21:48:19 step 185440 loss=3.0745 lr=1.34e-05 tok=24.420B dt=18889 wait=311 +21:49:23 step 185460 loss=3.0195 lr=1.34e-05 tok=24.423B dt=18953 wait=311 +21:50:27 step 185480 loss=3.0101 lr=1.34e-05 tok=24.425B dt=19017 wait=311 +21:51:31 step 185500 loss=2.9703 lr=1.34e-05 tok=24.428B dt=19081 wait=312 +21:52:35 step 185520 loss=3.1164 lr=1.34e-05 tok=24.431B dt=19144 wait=312 +21:53:38 step 185540 loss=2.9242 lr=1.34e-05 tok=24.433B dt=19208 wait=312 +21:54:22 pushed ckpt_w181.pt,log_w181.txt +21:55:37 waited 76s for global r=784 +21:55:43 global r=784 load 4.279 -> 4.280 +21:56:17 step 185560 loss=3.0392 lr=1.33e-05 tok=24.436B dt=19366 wait=312 +21:57:21 step 185580 loss=3.0059 lr=1.33e-05 tok=24.438B dt=19430 wait=312 +21:58:25 step 185600 loss=2.9748 lr=1.33e-05 tok=24.441B dt=19494 wait=312 +21:59:29 step 185620 loss=3.1466 lr=1.33e-05 tok=24.444B dt=19558 wait=313 +22:00:33 step 185640 loss=2.9318 lr=1.33e-05 tok=24.446B dt=19623 wait=313 +22:01:37 step 185660 loss=3.1930 lr=1.32e-05 tok=24.449B dt=19687 wait=313 +22:02:42 step 185680 loss=3.0679 lr=1.32e-05 tok=24.451B dt=19751 wait=313 +22:03:46 step 185700 loss=3.2130 lr=1.32e-05 tok=24.454B dt=19816 wait=313 +22:04:02 pushed ckpt_w181.pt,log_w181.txt +22:05:18 waited 76s for global r=785 +22:05:24 global r=785 load 4.282 -> 4.279 +22:06:30 step 185720 loss=2.9492 lr=1.32e-05 tok=24.457B dt=19979 wait=313 +22:07:34 step 185740 loss=3.1198 lr=1.32e-05 tok=24.459B dt=20043 wait=313 +22:08:37 step 185760 loss=3.1055 lr=1.32e-05 tok=24.462B dt=20107 wait=314 +22:09:42 step 185780 loss=2.9018 lr=1.31e-05 tok=24.465B dt=20171 wait=314 +22:10:45 step 185800 loss=2.9325 lr=1.31e-05 tok=24.467B dt=20235 wait=314 +22:11:49 step 185820 loss=3.0385 lr=1.31e-05 tok=24.470B dt=20299 wait=314 +22:12:53 step 185840 loss=2.9573 lr=1.31e-05 tok=24.472B dt=20363 wait=314 +22:13:40 pushed ckpt_w181.pt,log_w181.txt +22:14:56 waited 76s for global r=786 +22:15:03 global r=786 load 4.283 -> 4.279 +22:15:37 step 185860 loss=3.0035 lr=1.31e-05 tok=24.475B dt=20527 wait=314 +22:16:41 step 185880 loss=3.0521 lr=1.30e-05 tok=24.478B dt=20591 wait=315 +22:17:45 step 185900 loss=2.8635 lr=1.30e-05 tok=24.480B dt=20655 wait=315 +22:18:50 step 185920 loss=3.0342 lr=1.30e-05 tok=24.483B dt=20719 wait=315 +22:19:55 step 185940 loss=3.0089 lr=1.30e-05 tok=24.486B dt=20784 wait=315 +22:20:59 step 185960 loss=3.0693 lr=1.30e-05 tok=24.488B dt=20849 wait=315 +22:22:04 step 185980 loss=3.1018 lr=1.29e-05 tok=24.491B dt=20914 wait=315 +22:23:09 step 186000 loss=3.1289 lr=1.29e-05 tok=24.493B dt=20979 wait=316 diff --git a/log_w182.txt b/log_w182.txt new file mode 100644 index 0000000000000000000000000000000000000000..2cdf71c385e4adbeee68c5f15dba65c930c1cc6a --- /dev/null +++ b/log_w182.txt @@ -0,0 +1,174 @@ +launch +15:23:18 name=w182 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 129/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:23:25 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:23:25 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:23:34 params 1.08B +15:23:36 resumed step=179850 seen=23.69B round=746 +15:23:36 stream salt=179850 +15:25:41 blocks compiled +15:25:41 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +15:30:54 step 179851 loss=2.9905 lr=1.85e-05 tok=23.687B dt=313 wait=284 +15:31:09 step 179852 loss=2.9367 lr=1.85e-05 tok=23.688B dt=329 wait=284 +15:31:35 step 179860 loss=3.4407 lr=1.85e-05 tok=23.689B dt=355 wait=284 +15:31:50 global r=747 load 4.275 -> 4.283 +15:32:55 step 180020 loss=3.1239 lr=1.85e-05 tok=23.691B dt=434 wait=284 +15:34:00 step 180040 loss=3.0965 lr=1.85e-05 tok=23.694B dt=499 wait=285 +15:35:05 step 180060 loss=3.0657 lr=1.84e-05 tok=23.697B dt=564 wait=285 +15:36:09 step 180080 loss=3.1530 lr=1.84e-05 tok=23.699B dt=628 wait=285 +15:37:14 step 180100 loss=3.1832 lr=1.84e-05 tok=23.702B dt=693 wait=285 +15:38:19 step 180120 loss=3.1849 lr=1.84e-05 tok=23.704B dt=758 wait=285 +15:39:24 step 180140 loss=3.0454 lr=1.84e-05 tok=23.707B dt=823 wait=285 +15:40:10 pushed ckpt_w182.pt,log_w182.txt +15:42:26 waited 136s for global r=748 +15:42:34 global r=748 load 4.271 -> 4.280 +15:43:08 step 180160 loss=3.1394 lr=1.84e-05 tok=23.710B dt=1048 wait=285 +15:44:13 step 180180 loss=2.9588 lr=1.84e-05 tok=23.712B dt=1113 wait=286 +15:45:18 step 180200 loss=3.0513 lr=1.84e-05 tok=23.715B dt=1178 wait=286 +15:46:24 step 180220 loss=3.1141 lr=1.84e-05 tok=23.717B dt=1243 wait=286 +15:47:29 step 180240 loss=3.0520 lr=1.84e-05 tok=23.720B dt=1308 wait=286 +15:48:34 step 180260 loss=3.1187 lr=1.84e-05 tok=23.723B dt=1374 wait=286 +15:49:39 step 180280 loss=3.1060 lr=1.84e-05 tok=23.725B dt=1438 wait=286 +15:50:43 step 180300 loss=3.1762 lr=1.84e-05 tok=23.728B dt=1503 wait=286 +15:50:55 pushed ckpt_w182.pt,log_w182.txt +15:54:41 global r=749 load 4.275 -> 4.282 +15:55:45 step 180320 loss=2.9805 lr=1.84e-05 tok=23.732B dt=1805 wait=287 +15:56:51 step 180340 loss=2.7509 lr=1.84e-05 tok=23.735B dt=1871 wait=287 +15:57:56 step 180360 loss=3.0707 lr=1.84e-05 tok=23.737B dt=1936 wait=287 +15:59:00 step 180380 loss=3.1507 lr=1.83e-05 tok=23.740B dt=2000 wait=287 +16:00:06 step 180400 loss=3.1426 lr=1.83e-05 tok=23.742B dt=2065 wait=287 +16:01:11 step 180420 loss=3.0408 lr=1.83e-05 tok=23.745B dt=2131 wait=287 +16:02:15 step 180440 loss=2.9595 lr=1.83e-05 tok=23.748B dt=2195 wait=288 +16:03:00 pushed ckpt_w182.pt,log_w182.txt +16:05:46 waited 166s for global r=750 +16:05:54 global r=750 load 4.279 -> 4.283 +16:06:27 step 180460 loss=3.0371 lr=1.83e-05 tok=23.750B dt=2447 wait=288 +16:07:33 step 180480 loss=2.8059 lr=1.83e-05 tok=23.753B dt=2512 wait=288 +16:08:37 step 180500 loss=2.8496 lr=1.83e-05 tok=23.755B dt=2577 wait=288 +16:09:42 step 180520 loss=3.0726 lr=1.83e-05 tok=23.758B dt=2642 wait=288 +16:10:47 step 180540 loss=2.9810 lr=1.83e-05 tok=23.761B dt=2707 wait=288 +16:11:52 step 180560 loss=3.0218 lr=1.83e-05 tok=23.763B dt=2771 wait=288 +16:12:56 step 180580 loss=3.0840 lr=1.83e-05 tok=23.766B dt=2836 wait=289 +16:14:01 step 180600 loss=3.0958 lr=1.83e-05 tok=23.769B dt=2900 wait=289 +16:14:15 pushed ckpt_w182.pt,log_w182.txt +16:15:45 waited 91s for global r=751 +16:15:51 global r=751 load 4.286 -> 4.282 +16:16:56 step 180620 loss=3.0679 lr=1.83e-05 tok=23.771B dt=3076 wait=289 +16:18:01 step 180640 loss=2.6726 lr=1.83e-05 tok=23.774B dt=3141 wait=289 +16:19:05 step 180660 loss=3.0367 lr=1.83e-05 tok=23.776B dt=3205 wait=289 +16:20:10 step 180680 loss=2.9143 lr=1.82e-05 tok=23.779B dt=3269 wait=289 +16:21:16 step 180700 loss=2.6730 lr=1.82e-05 tok=23.782B dt=3335 wait=290 +16:22:21 step 180720 loss=3.0217 lr=1.82e-05 tok=23.784B dt=3400 wait=290 +16:23:26 step 180740 loss=2.8270 lr=1.82e-05 tok=23.787B dt=3465 wait=290 +16:24:10 pushed ckpt_w182.pt,log_w182.txt +16:26:56 waited 166s for global r=752 +16:27:02 global r=752 load 4.278 -> 4.281 +16:27:35 step 180760 loss=2.8434 lr=1.82e-05 tok=23.790B dt=3715 wait=290 +16:28:40 step 180780 loss=2.8419 lr=1.82e-05 tok=23.792B dt=3779 wait=290 +16:29:45 step 180800 loss=2.6260 lr=1.82e-05 tok=23.795B dt=3845 wait=290 +16:30:50 step 180820 loss=3.0122 lr=1.82e-05 tok=23.797B dt=3909 wait=290 +16:31:54 step 180840 loss=2.8735 lr=1.82e-05 tok=23.800B dt=3973 wait=290 +16:32:59 step 180860 loss=2.9545 lr=1.82e-05 tok=23.803B dt=4038 wait=291 +launch +16:33:58 name=w182 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 129/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:34:02 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:34:02 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:34:11 params 1.08B +16:34:13 resumed step=180750 seen=23.82B round=752 +16:34:13 stream salt=180750 +16:36:01 blocks compiled +16:36:01 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:40:55 step 180751 loss=2.6508 lr=1.78e-05 tok=23.819B dt=293 wait=285 +16:41:02 step 180752 loss=2.8564 lr=1.78e-05 tok=23.819B dt=301 wait=285 +16:41:27 step 180760 loss=3.1256 lr=1.78e-05 tok=23.820B dt=326 wait=285 +16:41:42 global r=753 load 4.282 -> 4.284 +16:42:47 step 180920 loss=2.9891 lr=1.76e-05 tok=23.822B dt=406 wait=285 +16:43:52 step 180940 loss=3.0437 lr=1.76e-05 tok=23.825B dt=471 wait=285 +16:44:57 step 180960 loss=3.0017 lr=1.76e-05 tok=23.828B dt=536 wait=285 +16:46:01 step 180980 loss=2.9653 lr=1.76e-05 tok=23.830B dt=600 wait=285 +16:47:07 step 181000 loss=2.8954 lr=1.75e-05 tok=23.833B dt=665 wait=285 +16:48:11 step 181020 loss=3.0341 lr=1.75e-05 tok=23.835B dt=730 wait=286 +16:49:16 step 181040 loss=2.8697 lr=1.75e-05 tok=23.838B dt=795 wait=286 +16:50:03 pushed ckpt_w182.pt,log_w182.txt +16:50:48 waited 45s for global r=754 +16:50:59 global r=754 load 4.283 -> 4.281 +16:51:33 step 181060 loss=2.9461 lr=1.75e-05 tok=23.841B dt=932 wait=286 +16:52:37 step 181080 loss=2.8667 lr=1.75e-05 tok=23.843B dt=996 wait=286 +16:53:42 step 181100 loss=2.9867 lr=1.75e-05 tok=23.846B dt=1060 wait=286 +16:54:47 step 181120 loss=3.0069 lr=1.74e-05 tok=23.849B dt=1126 wait=286 +16:55:52 step 181140 loss=2.8123 lr=1.74e-05 tok=23.851B dt=1191 wait=286 +16:56:57 step 181160 loss=2.8580 lr=1.74e-05 tok=23.854B dt=1256 wait=287 +16:58:02 step 181180 loss=2.8819 lr=1.74e-05 tok=23.856B dt=1321 wait=287 +16:59:07 step 181200 loss=3.0167 lr=1.74e-05 tok=23.859B dt=1385 wait=287 +16:59:18 pushed ckpt_w182.pt,log_w182.txt +17:01:04 waited 106s for global r=755 +17:01:12 global r=755 load 4.280 -> 4.280 +17:02:18 step 181220 loss=2.9692 lr=1.73e-05 tok=23.862B dt=1576 wait=287 +17:03:22 step 181240 loss=3.0431 lr=1.73e-05 tok=23.864B dt=1641 wait=287 +17:04:27 step 181260 loss=3.0825 lr=1.73e-05 tok=23.867B dt=1706 wait=287 +17:05:32 step 181280 loss=3.1306 lr=1.73e-05 tok=23.870B dt=1771 wait=288 +17:06:37 step 181300 loss=3.2024 lr=1.73e-05 tok=23.872B dt=1835 wait=288 +17:07:43 step 181320 loss=3.1913 lr=1.73e-05 tok=23.875B dt=1902 wait=288 +17:08:49 step 181340 loss=3.1041 lr=1.72e-05 tok=23.877B dt=1967 wait=288 +17:09:41 pushed ckpt_w182.pt,log_w182.txt +17:10:56 waited 76s for global r=756 +17:11:03 global r=756 load 4.284 -> 4.281 +17:11:37 step 181360 loss=3.1824 lr=1.72e-05 tok=23.880B dt=2136 wait=288 +17:12:43 step 181380 loss=3.1044 lr=1.72e-05 tok=23.883B dt=2202 wait=288 +17:13:50 step 181400 loss=3.1043 lr=1.72e-05 tok=23.885B dt=2268 wait=288 +17:14:56 step 181420 loss=3.1451 lr=1.72e-05 tok=23.888B dt=2334 wait=288 +17:16:01 step 181440 loss=3.0826 lr=1.71e-05 tok=23.890B dt=2400 wait=289 +17:17:07 step 181460 loss=3.1897 lr=1.71e-05 tok=23.893B dt=2466 wait=289 +17:18:12 step 181480 loss=3.0262 lr=1.71e-05 tok=23.896B dt=2531 wait=289 +17:19:17 step 181500 loss=3.0664 lr=1.71e-05 tok=23.898B dt=2596 wait=289 +17:19:32 pushed ckpt_w182.pt,log_w182.txt +17:22:33 waited 181s for global r=757 +17:22:39 global r=757 load 4.284 -> 4.282 +17:23:45 step 181520 loss=3.0199 lr=1.71e-05 tok=23.901B dt=2864 wait=289 +17:24:50 step 181540 loss=3.0073 lr=1.70e-05 tok=23.904B dt=2929 wait=289 +17:25:55 step 181560 loss=2.8319 lr=1.70e-05 tok=23.906B dt=2994 wait=290 +17:27:00 step 181580 loss=2.9883 lr=1.70e-05 tok=23.909B dt=3058 wait=290 +17:28:05 step 181600 loss=2.9666 lr=1.70e-05 tok=23.911B dt=3123 wait=290 +17:29:10 step 181620 loss=3.0303 lr=1.70e-05 tok=23.914B dt=3189 wait=290 +17:30:15 step 181640 loss=2.9078 lr=1.70e-05 tok=23.917B dt=3254 wait=290 +17:31:01 pushed ckpt_w182.pt,log_w182.txt +17:33:02 waited 121s for global r=758 +17:33:09 global r=758 load 4.292 -> 4.283 +17:33:43 step 181660 loss=3.0321 lr=1.69e-05 tok=23.919B dt=3462 wait=290 +17:34:48 step 181680 loss=2.8751 lr=1.69e-05 tok=23.922B dt=3527 wait=290 +17:35:54 step 181700 loss=2.8977 lr=1.69e-05 tok=23.925B dt=3593 wait=291 +17:36:59 step 181720 loss=2.9693 lr=1.69e-05 tok=23.927B dt=3658 wait=291 +17:38:05 step 181740 loss=2.9595 lr=1.69e-05 tok=23.930B dt=3723 wait=291 +17:39:10 step 181760 loss=2.9091 lr=1.68e-05 tok=23.932B dt=3789 wait=291 +17:40:16 step 181780 loss=2.9863 lr=1.68e-05 tok=23.935B dt=3854 wait=291 +17:41:21 step 181800 loss=2.8890 lr=1.68e-05 tok=23.938B dt=3920 wait=291 +17:41:35 pushed ckpt_w182.pt,log_w182.txt +17:42:35 waited 60s for global r=759 +17:42:45 global r=759 load 4.284 -> 4.283 +17:43:51 step 181820 loss=2.9065 lr=1.68e-05 tok=23.940B dt=4070 wait=291 +17:44:57 step 181840 loss=2.9337 lr=1.68e-05 tok=23.943B dt=4135 wait=292 +17:46:02 step 181860 loss=2.8241 lr=1.68e-05 tok=23.946B dt=4201 wait=292 +17:47:08 step 181880 loss=2.9273 lr=1.67e-05 tok=23.948B dt=4266 wait=292 +17:48:13 step 181900 loss=2.8407 lr=1.67e-05 tok=23.951B dt=4331 wait=292 +17:49:19 step 181920 loss=3.0156 lr=1.67e-05 tok=23.953B dt=4397 wait=292 +17:50:24 step 181940 loss=2.8764 lr=1.67e-05 tok=23.956B dt=4463 wait=292 +17:51:09 pushed ckpt_w182.pt,log_w182.txt +17:52:55 waited 106s for global r=760 +17:53:03 global r=760 load 4.283 -> 4.281 +17:53:36 step 181960 loss=2.9874 lr=1.67e-05 tok=23.959B dt=4655 wait=292 +17:54:41 step 181980 loss=2.9215 lr=1.66e-05 tok=23.961B dt=4720 wait=293 +17:55:46 step 182000 loss=2.9052 lr=1.66e-05 tok=23.964B dt=4785 wait=293 +17:56:53 step 182020 loss=2.8813 lr=1.66e-05 tok=23.967B dt=4851 wait=293 +17:57:59 step 182040 loss=2.8133 lr=1.66e-05 tok=23.969B dt=4917 wait=293 +17:59:04 step 182060 loss=2.8435 lr=1.66e-05 tok=23.972B dt=4983 wait=293 +18:00:11 step 182080 loss=2.3076 lr=1.65e-05 tok=23.974B dt=5049 wait=293 +18:01:16 step 182100 loss=2.7940 lr=1.65e-05 tok=23.977B dt=5115 wait=294 +18:01:30 pushed ckpt_w182.pt,log_w182.txt +18:02:45 waited 76s for global r=761 +18:02:53 global r=761 load 4.289 -> 4.281 +18:03:59 step 182120 loss=2.8314 lr=1.65e-05 tok=23.980B dt=5278 wait=294 +18:05:04 step 182140 loss=2.8607 lr=1.65e-05 tok=23.982B dt=5343 wait=294 +18:06:10 step 182160 loss=2.7741 lr=1.65e-05 tok=23.985B dt=5409 wait=294 +18:07:17 step 182180 loss=2.8288 lr=1.65e-05 tok=23.987B dt=5475 wait=294 +18:08:22 step 182200 loss=2.8792 lr=1.64e-05 tok=23.990B dt=5540 wait=294 +18:09:28 step 182220 loss=2.8611 lr=1.64e-05 tok=23.993B dt=5606 wait=294 +18:10:33 step 182240 loss=2.7787 lr=1.64e-05 tok=23.995B dt=5671 wait=295 diff --git a/log_w183.txt b/log_w183.txt new file mode 100644 index 0000000000000000000000000000000000000000..f688838972a00d0b17bd48f1b9a60f1b901749cf --- /dev/null +++ b/log_w183.txt @@ -0,0 +1,175 @@ +launch +15:26:01 name=w183 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 130/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:26:08 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:26:08 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:26:17 params 1.08B +15:26:19 resumed step=179850 seen=23.69B round=746 +15:26:19 stream salt=179850 +15:28:15 blocks compiled +15:28:15 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +15:33:27 step 179851 loss=2.9748 lr=1.85e-05 tok=23.687B dt=312 wait=281 +15:33:43 step 179852 loss=3.0714 lr=1.85e-05 tok=23.688B dt=327 wait=281 +15:34:08 step 179860 loss=3.2837 lr=1.85e-05 tok=23.689B dt=353 wait=281 +15:34:27 global r=747 load 4.272 -> 4.283 +15:35:32 step 180020 loss=3.0279 lr=1.85e-05 tok=23.691B dt=436 wait=282 +15:36:36 step 180040 loss=3.1332 lr=1.85e-05 tok=23.694B dt=500 wait=282 +15:37:39 step 180060 loss=3.1452 lr=1.84e-05 tok=23.697B dt=564 wait=282 +15:38:43 step 180080 loss=3.1896 lr=1.84e-05 tok=23.699B dt=628 wait=282 +15:39:48 step 180100 loss=3.1854 lr=1.84e-05 tok=23.702B dt=692 wait=282 +15:40:52 step 180120 loss=3.2814 lr=1.84e-05 tok=23.704B dt=757 wait=282 +15:41:56 step 180140 loss=3.1126 lr=1.84e-05 tok=23.707B dt=821 wait=283 +15:42:42 pushed ckpt_w183.pt,log_w183.txt +15:42:42 waited 0s for global r=748 +15:42:52 global r=748 load 4.283 -> 4.280 +15:43:26 step 180160 loss=3.2346 lr=1.84e-05 tok=23.710B dt=911 wait=283 +15:44:31 step 180180 loss=3.2666 lr=1.84e-05 tok=23.712B dt=976 wait=283 +15:45:36 step 180200 loss=3.1835 lr=1.84e-05 tok=23.715B dt=1041 wait=283 +15:46:42 step 180220 loss=3.4890 lr=1.84e-05 tok=23.717B dt=1106 wait=283 +15:47:46 step 180240 loss=3.2671 lr=1.84e-05 tok=23.720B dt=1171 wait=283 +15:48:51 step 180260 loss=3.2969 lr=1.84e-05 tok=23.723B dt=1236 wait=283 +15:49:55 step 180280 loss=3.2045 lr=1.84e-05 tok=23.725B dt=1300 wait=284 +15:50:59 step 180300 loss=3.3533 lr=1.84e-05 tok=23.728B dt=1364 wait=284 +15:51:11 pushed ckpt_w183.pt,log_w183.txt +15:54:12 waited 181s for global r=749 +15:54:22 global r=749 load 4.283 -> 4.282 +15:55:27 step 180320 loss=3.1766 lr=1.84e-05 tok=23.731B dt=1632 wait=284 +15:56:31 step 180340 loss=3.2318 lr=1.84e-05 tok=23.733B dt=1696 wait=284 +15:57:34 step 180360 loss=3.2157 lr=1.84e-05 tok=23.736B dt=1759 wait=284 +15:58:38 step 180380 loss=3.2234 lr=1.83e-05 tok=23.738B dt=1823 wait=284 +15:59:41 step 180400 loss=3.1171 lr=1.83e-05 tok=23.741B dt=1885 wait=284 +16:00:45 step 180420 loss=3.1557 lr=1.83e-05 tok=23.744B dt=1950 wait=285 +16:01:49 step 180440 loss=3.1320 lr=1.83e-05 tok=23.746B dt=2014 wait=285 +16:02:33 pushed ckpt_w183.pt,log_w183.txt +16:06:08 step 180460 loss=3.3714 lr=1.83e-05 tok=23.749B dt=2273 wait=285 +16:06:14 global r=750 load 4.281 -> 4.283 +16:06:46 step 180460 loss=3.2764 lr=1.83e-05 tok=23.750B dt=2311 wait=285 +16:07:50 step 180480 loss=3.0141 lr=1.83e-05 tok=23.753B dt=2375 wait=285 +16:08:54 step 180500 loss=2.9767 lr=1.83e-05 tok=23.755B dt=2439 wait=285 +16:09:58 step 180520 loss=3.1313 lr=1.83e-05 tok=23.758B dt=2503 wait=285 +16:11:02 step 180540 loss=3.0623 lr=1.83e-05 tok=23.761B dt=2567 wait=286 +16:12:06 step 180560 loss=3.0837 lr=1.83e-05 tok=23.763B dt=2630 wait=286 +16:13:09 step 180580 loss=3.1111 lr=1.83e-05 tok=23.766B dt=2693 wait=286 +16:14:12 step 180600 loss=3.2296 lr=1.83e-05 tok=23.769B dt=2757 wait=286 +16:14:24 pushed ckpt_w183.pt,log_w183.txt +16:15:54 waited 91s for global r=751 +16:16:00 global r=751 load 4.278 -> 4.282 +16:17:05 step 180620 loss=3.0367 lr=1.83e-05 tok=23.771B dt=2930 wait=286 +16:18:09 step 180640 loss=3.2322 lr=1.83e-05 tok=23.774B dt=2994 wait=286 +16:19:13 step 180660 loss=3.1544 lr=1.83e-05 tok=23.776B dt=3058 wait=287 +16:20:17 step 180680 loss=3.1311 lr=1.82e-05 tok=23.779B dt=3121 wait=287 +16:21:21 step 180700 loss=3.0656 lr=1.82e-05 tok=23.782B dt=3186 wait=287 +16:22:25 step 180720 loss=3.2775 lr=1.82e-05 tok=23.784B dt=3250 wait=287 +16:23:29 step 180740 loss=3.1155 lr=1.82e-05 tok=23.787B dt=3314 wait=287 +16:24:14 pushed ckpt_w183.pt,log_w183.txt +16:27:00 waited 166s for global r=752 +16:27:06 global r=752 load 4.279 -> 4.281 +16:27:40 step 180760 loss=3.2140 lr=1.82e-05 tok=23.790B dt=3565 wait=287 +16:28:44 step 180780 loss=2.9697 lr=1.82e-05 tok=23.792B dt=3629 wait=287 +16:29:48 step 180800 loss=3.0566 lr=1.82e-05 tok=23.795B dt=3693 wait=288 +16:30:53 step 180820 loss=3.1267 lr=1.82e-05 tok=23.797B dt=3757 wait=288 +16:31:56 step 180840 loss=3.0414 lr=1.82e-05 tok=23.800B dt=3821 wait=288 +16:33:01 step 180860 loss=3.1636 lr=1.82e-05 tok=23.803B dt=3886 wait=288 +16:34:05 step 180880 loss=3.0407 lr=1.82e-05 tok=23.805B dt=3950 wait=288 +16:35:10 step 180900 loss=3.1535 lr=1.82e-05 tok=23.808B dt=4015 wait=288 +16:35:24 pushed ckpt_w183.pt,log_w183.txt +launch +16:36:28 name=w183 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 130/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:36:33 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:36:33 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:36:42 params 1.08B +16:36:44 resumed step=180750 seen=23.82B round=752 +16:36:44 stream salt=180750 +16:38:33 blocks compiled +16:38:33 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:43:21 step 180751 loss=3.1427 lr=1.78e-05 tok=23.819B dt=288 wait=278 +16:43:28 step 180752 loss=3.2923 lr=1.78e-05 tok=23.819B dt=295 wait=278 +16:43:54 step 180760 loss=3.4568 lr=1.78e-05 tok=23.820B dt=321 wait=278 +16:44:09 global r=753 load 4.287 -> 4.284 +16:45:14 step 180920 loss=3.2401 lr=1.76e-05 tok=23.822B dt=401 wait=278 +16:46:20 step 180940 loss=3.2152 lr=1.76e-05 tok=23.825B dt=467 wait=278 +16:47:25 step 180960 loss=3.2772 lr=1.76e-05 tok=23.828B dt=532 wait=278 +16:48:30 step 180980 loss=3.3063 lr=1.76e-05 tok=23.830B dt=597 wait=278 +16:49:35 step 181000 loss=3.2166 lr=1.75e-05 tok=23.833B dt=662 wait=279 +16:50:41 step 181020 loss=3.1773 lr=1.75e-05 tok=23.835B dt=728 wait=279 +16:50:50 global r=754 load 4.286 -> 4.281 +16:51:23 step 181060 loss=3.1507 lr=1.75e-05 tok=23.837B dt=770 wait=279 +16:52:27 step 181080 loss=3.1284 lr=1.75e-05 tok=23.839B dt=834 wait=279 +16:53:32 step 181100 loss=3.0311 lr=1.75e-05 tok=23.842B dt=899 wait=279 +16:54:36 step 181120 loss=3.0300 lr=1.74e-05 tok=23.845B dt=963 wait=279 +16:55:41 step 181140 loss=2.9960 lr=1.74e-05 tok=23.847B dt=1028 wait=279 +16:56:46 step 181160 loss=3.0895 lr=1.74e-05 tok=23.850B dt=1093 wait=280 +16:57:50 step 181180 loss=2.9952 lr=1.74e-05 tok=23.852B dt=1157 wait=280 +16:58:55 step 181200 loss=3.0900 lr=1.74e-05 tok=23.855B dt=1222 wait=280 +16:59:10 pushed ckpt_w183.pt,log_w183.txt +17:00:56 waited 106s for global r=755 +17:01:05 global r=755 load 4.282 -> 4.280 +17:02:11 step 181220 loss=3.0783 lr=1.73e-05 tok=23.858B dt=1419 wait=280 +17:03:16 step 181240 loss=3.0125 lr=1.73e-05 tok=23.860B dt=1483 wait=280 +17:04:20 step 181260 loss=3.0555 lr=1.73e-05 tok=23.863B dt=1547 wait=280 +17:05:25 step 181280 loss=3.0414 lr=1.73e-05 tok=23.866B dt=1612 wait=281 +17:06:30 step 181300 loss=3.0313 lr=1.73e-05 tok=23.868B dt=1677 wait=281 +17:07:34 step 181320 loss=3.1169 lr=1.73e-05 tok=23.871B dt=1741 wait=281 +17:08:38 step 181340 loss=2.9438 lr=1.72e-05 tok=23.873B dt=1805 wait=281 +17:09:22 pushed ckpt_w183.pt,log_w183.txt +17:11:08 waited 106s for global r=756 +17:11:14 global r=756 load 4.280 -> 4.281 +17:11:48 step 181360 loss=3.0142 lr=1.72e-05 tok=23.876B dt=1995 wait=281 +17:12:52 step 181380 loss=2.9950 lr=1.72e-05 tok=23.879B dt=2059 wait=281 +17:13:56 step 181400 loss=2.9194 lr=1.72e-05 tok=23.881B dt=2123 wait=282 +17:15:01 step 181420 loss=2.9623 lr=1.72e-05 tok=23.884B dt=2188 wait=282 +17:16:06 step 181440 loss=2.8649 lr=1.71e-05 tok=23.887B dt=2253 wait=282 +17:17:10 step 181460 loss=2.9731 lr=1.71e-05 tok=23.889B dt=2317 wait=282 +17:18:15 step 181480 loss=2.9113 lr=1.71e-05 tok=23.892B dt=2382 wait=282 +17:19:19 step 181500 loss=2.9746 lr=1.71e-05 tok=23.894B dt=2446 wait=282 +17:19:31 pushed ckpt_w183.pt,log_w183.txt +17:22:32 waited 181s for global r=757 +17:22:47 global r=757 load 4.289 -> 4.282 +17:23:53 step 181520 loss=2.9975 lr=1.71e-05 tok=23.897B dt=2720 wait=283 +17:24:57 step 181540 loss=3.1086 lr=1.70e-05 tok=23.900B dt=2784 wait=283 +17:26:01 step 181560 loss=3.0480 lr=1.70e-05 tok=23.902B dt=2848 wait=283 +17:27:04 step 181580 loss=3.0287 lr=1.70e-05 tok=23.905B dt=2911 wait=283 +17:28:08 step 181600 loss=3.0129 lr=1.70e-05 tok=23.908B dt=2975 wait=283 +17:29:12 step 181620 loss=3.1374 lr=1.70e-05 tok=23.910B dt=3040 wait=283 +17:30:17 step 181640 loss=2.9888 lr=1.70e-05 tok=23.913B dt=3104 wait=283 +17:31:04 pushed ckpt_w183.pt,log_w183.txt +17:32:49 waited 106s for global r=758 +17:32:56 global r=758 load 4.282 -> 4.283 +17:33:29 step 181660 loss=3.0932 lr=1.69e-05 tok=23.915B dt=3296 wait=284 +17:34:33 step 181680 loss=2.9849 lr=1.69e-05 tok=23.918B dt=3360 wait=284 +17:35:36 step 181700 loss=2.9371 lr=1.69e-05 tok=23.921B dt=3423 wait=284 +17:36:40 step 181720 loss=3.0081 lr=1.69e-05 tok=23.923B dt=3487 wait=284 +17:37:44 step 181740 loss=2.9294 lr=1.69e-05 tok=23.926B dt=3551 wait=284 +17:38:47 step 181760 loss=2.9735 lr=1.68e-05 tok=23.929B dt=3614 wait=284 +17:39:50 step 181780 loss=3.0052 lr=1.68e-05 tok=23.931B dt=3677 wait=285 +17:40:54 step 181800 loss=2.9672 lr=1.68e-05 tok=23.934B dt=3741 wait=285 +17:41:08 pushed ckpt_w183.pt,log_w183.txt +17:42:39 waited 91s for global r=759 +17:43:04 global r=759 load 4.283 -> 4.283 +17:44:09 step 181820 loss=3.0524 lr=1.68e-05 tok=23.936B dt=3936 wait=285 +17:45:13 step 181840 loss=3.1331 lr=1.68e-05 tok=23.939B dt=4001 wait=285 +17:46:17 step 181860 loss=3.1404 lr=1.68e-05 tok=23.942B dt=4064 wait=285 +17:47:22 step 181880 loss=2.9684 lr=1.67e-05 tok=23.944B dt=4129 wait=285 +17:48:26 step 181900 loss=3.0749 lr=1.67e-05 tok=23.947B dt=4193 wait=286 +17:49:30 step 181920 loss=3.0272 lr=1.67e-05 tok=23.949B dt=4257 wait=286 +17:50:34 step 181940 loss=2.9805 lr=1.67e-05 tok=23.952B dt=4321 wait=286 +17:51:19 pushed ckpt_w183.pt,log_w183.txt +17:52:49 waited 91s for global r=760 +17:52:56 global r=760 load 4.279 -> 4.281 +17:53:29 step 181960 loss=2.9299 lr=1.67e-05 tok=23.955B dt=4497 wait=286 +17:54:34 step 181980 loss=2.9539 lr=1.66e-05 tok=23.957B dt=4561 wait=286 +17:55:38 step 182000 loss=2.9432 lr=1.66e-05 tok=23.960B dt=4625 wait=286 +17:56:41 step 182020 loss=3.0833 lr=1.66e-05 tok=23.963B dt=4688 wait=287 +17:57:46 step 182040 loss=2.9367 lr=1.66e-05 tok=23.965B dt=4753 wait=287 +17:58:50 step 182060 loss=3.0063 lr=1.66e-05 tok=23.968B dt=4817 wait=287 +17:59:54 step 182080 loss=2.9430 lr=1.65e-05 tok=23.970B dt=4881 wait=287 +18:00:58 step 182100 loss=2.9711 lr=1.65e-05 tok=23.973B dt=4945 wait=287 +18:01:12 pushed ckpt_w183.pt,log_w183.txt +18:02:43 waited 91s for global r=761 +18:02:53 global r=761 load 4.274 -> 4.281 +18:03:58 step 182120 loss=2.9949 lr=1.65e-05 tok=23.976B dt=5125 wait=287 +18:05:02 step 182140 loss=3.0081 lr=1.65e-05 tok=23.978B dt=5189 wait=288 +18:06:05 step 182160 loss=3.0435 lr=1.65e-05 tok=23.981B dt=5252 wait=288 +18:07:09 step 182180 loss=2.9270 lr=1.65e-05 tok=23.984B dt=5316 wait=288 +18:08:13 step 182200 loss=2.9201 lr=1.64e-05 tok=23.986B dt=5380 wait=288 +18:09:16 step 182220 loss=3.1032 lr=1.64e-05 tok=23.989B dt=5443 wait=288 +18:10:20 step 182240 loss=3.0330 lr=1.64e-05 tok=23.991B dt=5507 wait=288 diff --git a/log_w184.txt b/log_w184.txt new file mode 100644 index 0000000000000000000000000000000000000000..23425d8de7e5263dd4b431ddd81f4c978bbd6b6b --- /dev/null +++ b/log_w184.txt @@ -0,0 +1,253 @@ +launch +18:26:45 name=w184 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 131/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:26:52 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:26:52 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:27:01 params 1.08B +18:27:03 resumed step=182400 seen=24.03B round=763 +18:27:03 stream salt=182400 +18:28:51 blocks compiled +18:28:51 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +18:33:03 step 182401 loss=2.7636 lr=1.63e-05 tok=24.026B dt=252 wait=224 +18:33:19 step 182402 loss=2.8924 lr=1.63e-05 tok=24.026B dt=268 wait=224 +18:34:16 step 182420 loss=2.8260 lr=1.62e-05 tok=24.028B dt=326 wait=225 +18:35:20 step 182440 loss=3.0620 lr=1.62e-05 tok=24.031B dt=389 wait=225 +18:36:07 global r=764 load 4.281 -> 4.280 +18:36:39 step 182560 loss=3.1085 lr=1.61e-05 tok=24.033B dt=468 wait=225 +18:37:42 step 182580 loss=3.0843 lr=1.61e-05 tok=24.036B dt=531 wait=225 +18:38:45 step 182600 loss=3.0306 lr=1.61e-05 tok=24.039B dt=594 wait=225 +18:39:49 step 182620 loss=3.1768 lr=1.61e-05 tok=24.041B dt=658 wait=225 +18:40:53 step 182640 loss=3.0453 lr=1.60e-05 tok=24.044B dt=722 wait=226 +18:41:56 step 182660 loss=3.1708 lr=1.60e-05 tok=24.046B dt=785 wait=226 +18:43:01 step 182680 loss=2.7365 lr=1.60e-05 tok=24.049B dt=850 wait=226 +18:44:04 step 182700 loss=3.1356 lr=1.60e-05 tok=24.052B dt=914 wait=226 +18:44:18 pushed ckpt_w184.pt,log_w184.txt +18:45:18 waited 60s for global r=765 +18:45:26 global r=765 load 4.286 -> 4.280 +18:46:32 step 182720 loss=3.1324 lr=1.60e-05 tok=24.054B dt=1061 wait=226 +18:47:35 step 182740 loss=3.1037 lr=1.59e-05 tok=24.057B dt=1125 wait=226 +18:48:39 step 182760 loss=3.1576 lr=1.59e-05 tok=24.060B dt=1188 wait=227 +18:49:43 step 182780 loss=3.0753 lr=1.59e-05 tok=24.062B dt=1253 wait=227 +18:50:47 step 182800 loss=3.1469 lr=1.59e-05 tok=24.065B dt=1316 wait=227 +18:51:52 step 182820 loss=3.2261 lr=1.59e-05 tok=24.067B dt=1381 wait=227 +18:52:56 step 182840 loss=3.1497 lr=1.58e-05 tok=24.070B dt=1445 wait=227 +18:53:43 pushed ckpt_w184.pt,log_w184.txt +18:54:59 waited 76s for global r=766 +18:55:05 global r=766 load 4.277 -> 4.281 +18:55:38 step 182860 loss=3.2196 lr=1.58e-05 tok=24.073B dt=1607 wait=227 +18:56:43 step 182880 loss=3.2382 lr=1.58e-05 tok=24.075B dt=1672 wait=227 +18:57:47 step 182900 loss=3.1683 lr=1.58e-05 tok=24.078B dt=1737 wait=228 +18:58:52 step 182920 loss=3.2281 lr=1.58e-05 tok=24.081B dt=1801 wait=228 +18:59:57 step 182940 loss=3.1982 lr=1.58e-05 tok=24.083B dt=1866 wait=228 +19:01:01 step 182960 loss=3.3405 lr=1.57e-05 tok=24.086B dt=1930 wait=228 +19:02:05 step 182980 loss=3.1442 lr=1.57e-05 tok=24.088B dt=1994 wait=228 +19:03:10 step 183000 loss=3.1431 lr=1.57e-05 tok=24.091B dt=2059 wait=228 +19:03:21 pushed ckpt_w184.pt,log_w184.txt +19:04:37 waited 76s for global r=767 +19:04:46 global r=767 load 4.277 -> 4.283 +19:05:52 step 183020 loss=3.0840 lr=1.57e-05 tok=24.094B dt=2221 wait=228 +19:06:56 step 183040 loss=3.0820 lr=1.57e-05 tok=24.096B dt=2285 wait=229 +19:08:00 step 183060 loss=3.0081 lr=1.56e-05 tok=24.099B dt=2349 wait=229 +19:09:04 step 183080 loss=3.0450 lr=1.56e-05 tok=24.102B dt=2413 wait=229 +19:10:08 step 183100 loss=3.0726 lr=1.56e-05 tok=24.104B dt=2477 wait=229 +19:11:11 step 183120 loss=3.0112 lr=1.56e-05 tok=24.107B dt=2540 wait=229 +19:12:15 step 183140 loss=2.9115 lr=1.56e-05 tok=24.109B dt=2604 wait=229 +19:13:01 pushed ckpt_w184.pt,log_w184.txt +19:14:01 waited 61s for global r=768 +19:14:09 global r=768 load 4.290 -> 4.284 +19:14:43 step 183160 loss=3.0874 lr=1.56e-05 tok=24.112B dt=2752 wait=229 +19:15:46 step 183180 loss=3.0262 lr=1.55e-05 tok=24.115B dt=2816 wait=230 +19:16:50 step 183200 loss=3.0741 lr=1.55e-05 tok=24.117B dt=2879 wait=230 +19:17:55 step 183220 loss=3.0721 lr=1.55e-05 tok=24.120B dt=2944 wait=230 +19:18:59 step 183240 loss=3.1400 lr=1.55e-05 tok=24.122B dt=3008 wait=230 +19:20:04 step 183260 loss=3.1954 lr=1.55e-05 tok=24.125B dt=3074 wait=230 +19:21:09 step 183280 loss=3.1990 lr=1.54e-05 tok=24.128B dt=3138 wait=230 +19:22:14 step 183300 loss=3.1331 lr=1.54e-05 tok=24.130B dt=3203 wait=230 +19:22:26 pushed ckpt_w184.pt,log_w184.txt +19:23:27 waited 60s for global r=769 +19:23:37 global r=769 load 4.281 -> 4.282 +19:24:42 step 183320 loss=3.1796 lr=1.54e-05 tok=24.133B dt=3351 wait=231 +19:25:46 step 183340 loss=3.1521 lr=1.54e-05 tok=24.136B dt=3416 wait=231 +19:26:50 step 183360 loss=3.1027 lr=1.54e-05 tok=24.138B dt=3479 wait=231 +19:27:55 step 183380 loss=3.0996 lr=1.53e-05 tok=24.141B dt=3544 wait=231 +19:28:59 step 183400 loss=3.1182 lr=1.53e-05 tok=24.143B dt=3608 wait=231 +19:30:04 step 183420 loss=3.0967 lr=1.53e-05 tok=24.146B dt=3673 wait=231 +19:31:07 step 183440 loss=3.0656 lr=1.53e-05 tok=24.149B dt=3737 wait=231 +19:31:54 pushed ckpt_w184.pt,log_w184.txt +19:32:54 waited 61s for global r=770 +19:33:06 global r=770 load 4.281 -> 4.281 +19:33:40 step 183460 loss=3.1533 lr=1.53e-05 tok=24.151B dt=3889 wait=231 +19:34:44 step 183480 loss=3.0451 lr=1.53e-05 tok=24.154B dt=3953 wait=232 +19:35:49 step 183500 loss=2.7907 lr=1.52e-05 tok=24.157B dt=4018 wait=232 +19:36:53 step 183520 loss=3.2397 lr=1.52e-05 tok=24.159B dt=4082 wait=232 +19:37:57 step 183540 loss=3.0707 lr=1.52e-05 tok=24.162B dt=4147 wait=232 +19:39:02 step 183560 loss=3.2135 lr=1.52e-05 tok=24.164B dt=4212 wait=232 +19:40:07 step 183580 loss=3.2828 lr=1.52e-05 tok=24.167B dt=4276 wait=232 +19:41:12 step 183600 loss=3.2137 lr=1.51e-05 tok=24.170B dt=4341 wait=232 +19:41:26 pushed ckpt_w184.pt,log_w184.txt +19:44:27 waited 181s for global r=771 +19:44:35 global r=771 load 4.282 -> 4.282 +19:45:44 step 183620 loss=3.2385 lr=1.51e-05 tok=24.172B dt=4613 wait=233 +19:46:49 step 183640 loss=3.1211 lr=1.51e-05 tok=24.175B dt=4678 wait=233 +19:47:54 step 183660 loss=3.2287 lr=1.51e-05 tok=24.178B dt=4743 wait=233 +19:49:00 step 183680 loss=2.9960 lr=1.51e-05 tok=24.180B dt=4809 wait=233 +19:50:04 step 183700 loss=3.2014 lr=1.51e-05 tok=24.183B dt=4873 wait=233 +19:51:09 step 183720 loss=3.1403 lr=1.50e-05 tok=24.185B dt=4938 wait=233 +19:52:14 step 183740 loss=3.0104 lr=1.50e-05 tok=24.188B dt=5003 wait=234 +19:52:58 pushed ckpt_w184.pt,log_w184.txt +19:56:40 step 183760 loss=3.0786 lr=1.50e-05 tok=24.191B dt=5269 wait=234 +19:56:47 global r=772 load 4.279 -> 4.280 +19:57:20 step 183760 loss=3.0822 lr=1.50e-05 tok=24.192B dt=5309 wait=234 +19:58:25 step 183780 loss=2.8636 lr=1.50e-05 tok=24.195B dt=5375 wait=234 +19:59:30 step 183800 loss=2.9055 lr=1.50e-05 tok=24.197B dt=5439 wait=234 +20:00:36 step 183820 loss=2.9968 lr=1.49e-05 tok=24.200B dt=5505 wait=234 +20:01:41 step 183840 loss=2.8824 lr=1.49e-05 tok=24.202B dt=5570 wait=234 +20:02:47 step 183860 loss=2.9561 lr=1.49e-05 tok=24.205B dt=5636 wait=235 +20:03:52 step 183880 loss=2.8975 lr=1.49e-05 tok=24.208B dt=5701 wait=235 +20:04:58 step 183900 loss=2.8715 lr=1.49e-05 tok=24.210B dt=5767 wait=235 +20:05:10 pushed ckpt_w184.pt,log_w184.txt +20:07:56 waited 166s for global r=773 +20:08:02 global r=773 load 4.286 -> 4.282 +20:09:08 step 183920 loss=2.8639 lr=1.49e-05 tok=24.213B dt=6017 wait=235 +20:10:13 step 183940 loss=2.9967 lr=1.48e-05 tok=24.216B dt=6082 wait=235 +20:11:19 step 183960 loss=2.8685 lr=1.48e-05 tok=24.218B dt=6148 wait=235 +20:12:24 step 183980 loss=2.8717 lr=1.48e-05 tok=24.221B dt=6214 wait=235 +20:13:30 step 184000 loss=2.7478 lr=1.48e-05 tok=24.223B dt=6280 wait=236 +20:14:36 step 184020 loss=2.9001 lr=1.48e-05 tok=24.226B dt=6345 wait=236 +20:15:41 step 184040 loss=2.7754 lr=1.47e-05 tok=24.229B dt=6410 wait=236 +20:16:25 pushed ckpt_w184.pt,log_w184.txt +20:18:56 waited 151s for global r=774 +20:19:02 global r=774 load 4.288 -> 4.282 +20:19:36 step 184060 loss=2.9300 lr=1.47e-05 tok=24.231B dt=6646 wait=236 +20:20:42 step 184080 loss=2.8002 lr=1.47e-05 tok=24.234B dt=6711 wait=236 +20:21:47 step 184100 loss=2.8796 lr=1.47e-05 tok=24.237B dt=6776 wait=236 +20:22:53 step 184120 loss=2.8756 lr=1.47e-05 tok=24.239B dt=6842 wait=237 +20:23:58 step 184140 loss=2.8632 lr=1.46e-05 tok=24.242B dt=6907 wait=237 +20:25:03 step 184160 loss=2.9453 lr=1.46e-05 tok=24.244B dt=6972 wait=237 +20:26:08 step 184180 loss=2.8226 lr=1.46e-05 tok=24.247B dt=7038 wait=237 +20:27:14 step 184200 loss=2.9229 lr=1.46e-05 tok=24.250B dt=7103 wait=237 +20:27:27 pushed ckpt_w184.pt,log_w184.txt +20:28:27 waited 61s for global r=775 +20:28:35 global r=775 load 4.288 -> 4.281 +20:29:42 step 184220 loss=2.8653 lr=1.46e-05 tok=24.252B dt=7252 wait=237 +20:30:48 step 184240 loss=2.8993 lr=1.46e-05 tok=24.255B dt=7317 wait=237 +20:31:54 step 184260 loss=2.8923 lr=1.45e-05 tok=24.257B dt=7383 wait=237 +20:32:59 step 184280 loss=2.8887 lr=1.45e-05 tok=24.260B dt=7449 wait=238 +20:34:05 step 184300 loss=2.8386 lr=1.45e-05 tok=24.263B dt=7515 wait=238 +20:35:12 step 184320 loss=2.8951 lr=1.45e-05 tok=24.265B dt=7581 wait=238 +20:36:17 step 184340 loss=2.7377 lr=1.45e-05 tok=24.268B dt=7646 wait=238 +20:37:04 pushed ckpt_w184.pt,log_w184.txt +20:38:19 waited 76s for global r=776 +20:38:31 global r=776 load 4.290 -> 4.281 +20:39:05 step 184360 loss=2.9542 lr=1.44e-05 tok=24.271B dt=7815 wait=238 +20:40:11 step 184380 loss=2.8720 lr=1.44e-05 tok=24.273B dt=7880 wait=238 +20:41:16 step 184400 loss=2.8371 lr=1.44e-05 tok=24.276B dt=7946 wait=239 +20:42:22 step 184420 loss=2.9358 lr=1.44e-05 tok=24.278B dt=8011 wait=239 +20:43:27 step 184440 loss=2.8591 lr=1.44e-05 tok=24.281B dt=8076 wait=239 +20:44:31 step 184460 loss=3.0268 lr=1.44e-05 tok=24.284B dt=8140 wait=239 +20:45:35 step 184480 loss=2.9626 lr=1.43e-05 tok=24.286B dt=8204 wait=239 +20:46:40 step 184500 loss=2.7617 lr=1.43e-05 tok=24.289B dt=8269 wait=239 +20:46:53 pushed ckpt_w184.pt,log_w184.txt +20:47:53 waited 61s for global r=777 +20:47:59 global r=777 load 4.285 -> 4.280 +20:49:05 step 184520 loss=2.9514 lr=1.43e-05 tok=24.292B dt=8414 wait=239 +20:50:09 step 184540 loss=3.0254 lr=1.43e-05 tok=24.294B dt=8478 wait=240 +20:51:13 step 184560 loss=3.1445 lr=1.43e-05 tok=24.297B dt=8543 wait=240 +20:52:17 step 184580 loss=2.9852 lr=1.42e-05 tok=24.299B dt=8607 wait=240 +20:53:22 step 184600 loss=3.0010 lr=1.42e-05 tok=24.302B dt=8671 wait=240 +20:54:26 step 184620 loss=3.0788 lr=1.42e-05 tok=24.305B dt=8735 wait=240 +20:55:30 step 184640 loss=2.9813 lr=1.42e-05 tok=24.307B dt=8800 wait=240 +20:56:17 pushed ckpt_w184.pt,log_w184.txt +20:57:48 waited 91s for global r=778 +20:57:54 global r=778 load 4.277 -> 4.280 +20:58:27 step 184660 loss=3.0025 lr=1.42e-05 tok=24.310B dt=8976 wait=240 +20:59:31 step 184680 loss=2.9502 lr=1.41e-05 tok=24.313B dt=9041 wait=240 +21:00:35 step 184700 loss=3.0383 lr=1.41e-05 tok=24.315B dt=9104 wait=241 +21:01:40 step 184720 loss=3.1152 lr=1.41e-05 tok=24.318B dt=9169 wait=241 +21:02:45 step 184740 loss=2.9626 lr=1.41e-05 tok=24.320B dt=9234 wait=241 +21:03:49 step 184760 loss=3.0885 lr=1.41e-05 tok=24.323B dt=9298 wait=241 +21:04:53 step 184780 loss=3.0247 lr=1.41e-05 tok=24.326B dt=9363 wait=241 +21:05:58 step 184800 loss=3.0667 lr=1.40e-05 tok=24.328B dt=9428 wait=241 +21:06:13 pushed ckpt_w184.pt,log_w184.txt +21:07:43 waited 91s for global r=779 +21:07:48 global r=779 load 4.277 -> 4.280 +21:08:54 step 184820 loss=3.0003 lr=1.40e-05 tok=24.331B dt=9603 wait=241 +21:09:58 step 184840 loss=2.8543 lr=1.40e-05 tok=24.334B dt=9667 wait=242 +21:11:03 step 184860 loss=2.9532 lr=1.40e-05 tok=24.336B dt=9732 wait=242 +21:12:06 step 184880 loss=2.8400 lr=1.40e-05 tok=24.339B dt=9796 wait=242 +21:13:10 step 184900 loss=2.9163 lr=1.39e-05 tok=24.341B dt=9859 wait=242 +21:14:14 step 184920 loss=3.0101 lr=1.39e-05 tok=24.344B dt=9923 wait=242 +21:15:18 step 184940 loss=2.7476 lr=1.39e-05 tok=24.347B dt=9987 wait=242 +21:16:04 pushed ckpt_w184.pt,log_w184.txt +21:17:04 waited 61s for global r=780 +21:17:11 global r=780 load 4.282 -> 4.280 +21:17:45 step 184960 loss=2.6542 lr=1.39e-05 tok=24.349B dt=10134 wait=242 +21:18:48 step 184980 loss=2.8443 lr=1.39e-05 tok=24.352B dt=10197 wait=243 +21:19:53 step 185000 loss=2.7649 lr=1.39e-05 tok=24.354B dt=10263 wait=243 +21:20:58 step 185020 loss=2.9286 lr=1.38e-05 tok=24.357B dt=10327 wait=243 +21:22:04 step 185040 loss=2.3770 lr=1.38e-05 tok=24.360B dt=10393 wait=243 +21:23:09 step 185060 loss=2.9980 lr=1.38e-05 tok=24.362B dt=10458 wait=243 +21:24:13 step 185080 loss=2.9404 lr=1.38e-05 tok=24.365B dt=10522 wait=243 +21:25:17 step 185100 loss=3.0117 lr=1.38e-05 tok=24.368B dt=10586 wait=243 +21:25:31 pushed ckpt_w184.pt,log_w184.txt +21:26:46 waited 76s for global r=781 +21:26:54 global r=781 load 4.286 -> 4.282 +21:28:00 step 185120 loss=2.9656 lr=1.37e-05 tok=24.370B dt=10749 wait=244 +21:29:05 step 185140 loss=3.0342 lr=1.37e-05 tok=24.373B dt=10814 wait=244 +21:30:09 step 185160 loss=2.9921 lr=1.37e-05 tok=24.375B dt=10878 wait=244 +21:31:13 step 185180 loss=3.0140 lr=1.37e-05 tok=24.378B dt=10942 wait=244 +21:32:18 step 185200 loss=3.0363 lr=1.37e-05 tok=24.381B dt=11007 wait=244 +21:33:22 step 185220 loss=3.0570 lr=1.36e-05 tok=24.383B dt=11071 wait=244 +21:34:27 step 185240 loss=3.0040 lr=1.36e-05 tok=24.386B dt=11136 wait=244 +21:35:11 pushed ckpt_w184.pt,log_w184.txt +21:36:26 waited 76s for global r=782 +21:36:32 global r=782 load 4.277 -> 4.280 +21:37:05 step 185260 loss=2.9943 lr=1.36e-05 tok=24.389B dt=11294 wait=245 +21:38:10 step 185280 loss=2.9597 lr=1.36e-05 tok=24.391B dt=11359 wait=245 +21:39:15 step 185300 loss=2.9725 lr=1.36e-05 tok=24.394B dt=11424 wait=245 +21:40:19 step 185320 loss=3.0572 lr=1.36e-05 tok=24.396B dt=11488 wait=245 +21:41:24 step 185340 loss=3.0361 lr=1.35e-05 tok=24.399B dt=11553 wait=245 +21:42:29 step 185360 loss=3.1057 lr=1.35e-05 tok=24.402B dt=11618 wait=245 +21:43:33 step 185380 loss=3.0861 lr=1.35e-05 tok=24.404B dt=11682 wait=245 +21:44:38 step 185400 loss=3.0391 lr=1.35e-05 tok=24.407B dt=11747 wait=246 +21:44:50 pushed ckpt_w184.pt,log_w184.txt +21:46:06 waited 76s for global r=783 +21:46:12 global r=783 load 4.279 -> 4.279 +21:47:18 step 185420 loss=2.8100 lr=1.35e-05 tok=24.410B dt=11907 wait=246 +21:48:23 step 185440 loss=2.9336 lr=1.34e-05 tok=24.412B dt=11972 wait=246 +21:49:28 step 185460 loss=3.0521 lr=1.34e-05 tok=24.415B dt=12037 wait=246 +21:50:32 step 185480 loss=3.1072 lr=1.34e-05 tok=24.417B dt=12101 wait=246 +21:51:37 step 185500 loss=2.9693 lr=1.34e-05 tok=24.420B dt=12166 wait=246 +21:52:42 step 185520 loss=3.0395 lr=1.34e-05 tok=24.423B dt=12231 wait=246 +21:53:46 step 185540 loss=2.9856 lr=1.34e-05 tok=24.425B dt=12295 wait=246 +21:54:31 pushed ckpt_w184.pt,log_w184.txt +21:55:31 waited 61s for global r=784 +21:55:38 global r=784 load 4.279 -> 4.280 +21:56:12 step 185560 loss=3.0352 lr=1.33e-05 tok=24.428B dt=12441 wait=247 +21:57:17 step 185580 loss=2.9391 lr=1.33e-05 tok=24.431B dt=12506 wait=247 +21:58:21 step 185600 loss=2.9200 lr=1.33e-05 tok=24.433B dt=12570 wait=247 +21:59:25 step 185620 loss=2.9617 lr=1.33e-05 tok=24.436B dt=12634 wait=247 +22:00:29 step 185640 loss=2.8760 lr=1.33e-05 tok=24.438B dt=12698 wait=247 +22:01:33 step 185660 loss=2.9460 lr=1.32e-05 tok=24.441B dt=12762 wait=247 +22:02:37 step 185680 loss=3.0106 lr=1.32e-05 tok=24.444B dt=12827 wait=247 +22:03:41 step 185700 loss=2.9810 lr=1.32e-05 tok=24.446B dt=12890 wait=248 +22:03:55 pushed ckpt_w184.pt,log_w184.txt +22:05:11 waited 76s for global r=785 +22:05:19 global r=785 load 4.278 -> 4.279 +22:06:25 step 185720 loss=2.9118 lr=1.32e-05 tok=24.449B dt=13054 wait=248 +22:07:29 step 185740 loss=2.9845 lr=1.32e-05 tok=24.451B dt=13118 wait=248 +22:08:33 step 185760 loss=3.0410 lr=1.32e-05 tok=24.454B dt=13182 wait=248 +22:09:37 step 185780 loss=3.0327 lr=1.31e-05 tok=24.457B dt=13246 wait=248 +22:10:41 step 185800 loss=3.0300 lr=1.31e-05 tok=24.459B dt=13311 wait=248 +22:11:46 step 185820 loss=3.0573 lr=1.31e-05 tok=24.462B dt=13375 wait=248 +22:12:50 step 185840 loss=3.0854 lr=1.31e-05 tok=24.465B dt=13439 wait=249 +22:13:35 pushed ckpt_w184.pt,log_w184.txt +22:14:51 waited 76s for global r=786 +22:14:59 global r=786 load 4.278 -> 4.279 +22:15:33 step 185860 loss=3.2197 lr=1.31e-05 tok=24.467B dt=13602 wait=249 +22:16:38 step 185880 loss=2.9335 lr=1.30e-05 tok=24.470B dt=13668 wait=249 +22:17:43 step 185900 loss=3.0288 lr=1.30e-05 tok=24.472B dt=13732 wait=249 +22:18:48 step 185920 loss=3.1686 lr=1.30e-05 tok=24.475B dt=13797 wait=249 +22:19:52 step 185940 loss=3.0433 lr=1.30e-05 tok=24.478B dt=13861 wait=249 +22:20:57 step 185960 loss=3.1346 lr=1.30e-05 tok=24.480B dt=13926 wait=249 +22:22:01 step 185980 loss=2.9858 lr=1.29e-05 tok=24.483B dt=13991 wait=250 +22:23:06 step 186000 loss=3.0251 lr=1.29e-05 tok=24.486B dt=14055 wait=250 diff --git a/log_w185.txt b/log_w185.txt new file mode 100644 index 0000000000000000000000000000000000000000..52c543300260cf517999570496aa2354924484ed --- /dev/null +++ b/log_w185.txt @@ -0,0 +1,265 @@ +launch +launch +19:46:55 name=w185 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 132/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:47:02 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:47:02 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +19:47:12 params 1.08B +19:47:13 resumed step=183600 seen=24.18B round=771 +19:47:13 stream salt=183600 +19:48:59 blocks compiled +19:48:59 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +19:52:53 step 183601 loss=2.6275 lr=1.51e-05 tok=24.184B dt=234 wait=200 +19:53:08 step 183602 loss=2.7602 lr=1.51e-05 tok=24.184B dt=250 wait=200 +19:54:06 step 183620 loss=2.9341 lr=1.51e-05 tok=24.187B dt=308 wait=200 +19:55:11 step 183640 loss=2.9786 lr=1.51e-05 tok=24.189B dt=372 wait=200 +19:56:15 step 183660 loss=2.9411 lr=1.51e-05 tok=24.192B dt=437 wait=200 +19:57:01 global r=772 load 4.283 -> 4.280 +19:57:33 step 183760 loss=3.0069 lr=1.50e-05 tok=24.195B dt=515 wait=201 +19:58:37 step 183780 loss=2.9770 lr=1.50e-05 tok=24.197B dt=578 wait=201 +19:59:41 step 183800 loss=3.0813 lr=1.50e-05 tok=24.200B dt=643 wait=201 +20:00:47 step 183820 loss=2.8400 lr=1.49e-05 tok=24.202B dt=708 wait=201 +20:01:51 step 183840 loss=3.0227 lr=1.49e-05 tok=24.205B dt=772 wait=201 +20:02:56 step 183860 loss=3.0608 lr=1.49e-05 tok=24.208B dt=838 wait=201 +20:04:00 step 183880 loss=3.1558 lr=1.49e-05 tok=24.210B dt=902 wait=201 +20:05:06 step 183900 loss=2.9615 lr=1.49e-05 tok=24.213B dt=968 wait=202 +20:05:20 pushed ckpt_w185.pt,log_w185.txt +20:08:06 waited 166s for global r=773 +20:08:15 global r=773 load 4.285 -> 4.282 +20:09:22 step 183920 loss=3.1417 lr=1.49e-05 tok=24.216B dt=1224 wait=202 +20:10:28 step 183940 loss=3.1805 lr=1.48e-05 tok=24.218B dt=1289 wait=202 +20:11:34 step 183960 loss=3.0822 lr=1.48e-05 tok=24.221B dt=1355 wait=202 +20:12:38 step 183980 loss=3.1066 lr=1.48e-05 tok=24.223B dt=1419 wait=202 +20:13:43 step 184000 loss=3.2228 lr=1.48e-05 tok=24.226B dt=1484 wait=202 +20:14:48 step 184020 loss=3.1710 lr=1.48e-05 tok=24.229B dt=1550 wait=203 +20:15:53 step 184040 loss=3.1049 lr=1.47e-05 tok=24.231B dt=1615 wait=203 +20:16:38 pushed ckpt_w185.pt,log_w185.txt +20:18:53 waited 136s for global r=774 +20:18:59 global r=774 load 4.281 -> 4.282 +20:19:33 step 184060 loss=3.1848 lr=1.47e-05 tok=24.234B dt=1834 wait=203 +20:20:37 step 184080 loss=3.0691 lr=1.47e-05 tok=24.237B dt=1899 wait=203 +20:21:43 step 184100 loss=3.1240 lr=1.47e-05 tok=24.239B dt=1964 wait=203 +20:22:48 step 184120 loss=3.0416 lr=1.47e-05 tok=24.242B dt=2030 wait=203 +20:23:53 step 184140 loss=3.0293 lr=1.46e-05 tok=24.244B dt=2095 wait=204 +20:24:58 step 184160 loss=3.0935 lr=1.46e-05 tok=24.247B dt=2159 wait=204 +20:26:03 step 184180 loss=3.0459 lr=1.46e-05 tok=24.250B dt=2225 wait=204 +20:27:08 step 184200 loss=2.9747 lr=1.46e-05 tok=24.252B dt=2290 wait=204 +20:27:22 pushed ckpt_w185.pt,log_w185.txt +20:28:38 waited 76s for global r=775 +20:28:50 global r=775 load 4.282 -> 4.281 +20:29:57 step 184220 loss=3.0287 lr=1.46e-05 tok=24.255B dt=2458 wait=204 +20:31:02 step 184240 loss=3.1162 lr=1.46e-05 tok=24.257B dt=2523 wait=204 +20:32:06 step 184260 loss=3.0594 lr=1.45e-05 tok=24.260B dt=2587 wait=204 +20:33:10 step 184280 loss=3.0605 lr=1.45e-05 tok=24.263B dt=2651 wait=204 +20:34:14 step 184300 loss=3.0398 lr=1.45e-05 tok=24.265B dt=2715 wait=205 +20:35:18 step 184320 loss=3.0319 lr=1.45e-05 tok=24.268B dt=2779 wait=205 +20:36:22 step 184340 loss=2.9399 lr=1.45e-05 tok=24.271B dt=2844 wait=205 +20:37:06 pushed ckpt_w185.pt,log_w185.txt +20:38:21 waited 76s for global r=776 +20:38:31 global r=776 load 4.280 -> 4.281 +20:39:04 step 184360 loss=2.9300 lr=1.44e-05 tok=24.273B dt=3005 wait=205 +20:40:08 step 184380 loss=2.9114 lr=1.44e-05 tok=24.276B dt=3069 wait=205 +20:41:12 step 184400 loss=2.8707 lr=1.44e-05 tok=24.278B dt=3134 wait=205 +20:42:17 step 184420 loss=3.0220 lr=1.44e-05 tok=24.281B dt=3198 wait=205 +20:43:21 step 184440 loss=2.9896 lr=1.44e-05 tok=24.284B dt=3263 wait=206 +20:44:25 step 184460 loss=3.0327 lr=1.44e-05 tok=24.286B dt=3327 wait=206 +20:45:30 step 184480 loss=3.0483 lr=1.43e-05 tok=24.289B dt=3391 wait=206 +20:46:34 step 184500 loss=2.9697 lr=1.43e-05 tok=24.292B dt=3456 wait=206 +20:46:46 pushed ckpt_w185.pt,log_w185.txt +20:48:02 waited 76s for global r=777 +20:48:09 global r=777 load 4.279 -> 4.280 +20:49:14 step 184520 loss=3.0436 lr=1.43e-05 tok=24.294B dt=3616 wait=206 +20:50:19 step 184540 loss=3.0466 lr=1.43e-05 tok=24.297B dt=3680 wait=206 +20:51:24 step 184560 loss=3.1289 lr=1.43e-05 tok=24.299B dt=3745 wait=207 +20:52:29 step 184580 loss=3.1544 lr=1.42e-05 tok=24.302B dt=3810 wait=207 +20:53:34 step 184600 loss=2.9285 lr=1.42e-05 tok=24.305B dt=3875 wait=207 +20:54:38 step 184620 loss=3.0802 lr=1.42e-05 tok=24.307B dt=3939 wait=207 +20:55:43 step 184640 loss=2.9409 lr=1.42e-05 tok=24.310B dt=4005 wait=207 +20:56:28 pushed ckpt_w185.pt,log_w185.txt +20:57:43 waited 76s for global r=778 +20:57:51 global r=778 load 4.279 -> 4.280 +20:58:25 step 184660 loss=3.1107 lr=1.42e-05 tok=24.313B dt=4166 wait=207 +20:59:30 step 184680 loss=2.9984 lr=1.41e-05 tok=24.315B dt=4232 wait=207 +21:00:35 step 184700 loss=3.0182 lr=1.41e-05 tok=24.318B dt=4296 wait=208 +21:01:40 step 184720 loss=3.1413 lr=1.41e-05 tok=24.320B dt=4362 wait=208 +21:02:45 step 184740 loss=2.9789 lr=1.41e-05 tok=24.323B dt=4427 wait=208 +21:03:50 step 184760 loss=3.0615 lr=1.41e-05 tok=24.326B dt=4492 wait=208 +21:04:55 step 184780 loss=3.0659 lr=1.41e-05 tok=24.328B dt=4556 wait=208 +21:06:00 step 184800 loss=3.0133 lr=1.40e-05 tok=24.331B dt=4621 wait=208 +21:06:11 pushed ckpt_w185.pt,log_w185.txt +21:07:42 waited 91s for global r=779 +21:07:47 global r=779 load 4.278 -> 4.280 +21:08:54 step 184820 loss=2.9733 lr=1.40e-05 tok=24.334B dt=4795 wait=209 +21:09:58 step 184840 loss=3.0071 lr=1.40e-05 tok=24.336B dt=4860 wait=209 +21:11:03 step 184860 loss=2.9823 lr=1.40e-05 tok=24.339B dt=4925 wait=209 +21:12:07 step 184880 loss=3.0062 lr=1.40e-05 tok=24.341B dt=4989 wait=209 +21:13:12 step 184900 loss=2.8797 lr=1.39e-05 tok=24.344B dt=5054 wait=209 +21:14:16 step 184920 loss=3.0616 lr=1.39e-05 tok=24.347B dt=5118 wait=209 +21:15:21 step 184940 loss=2.8664 lr=1.39e-05 tok=24.349B dt=5182 wait=209 +21:16:05 pushed ckpt_w185.pt,log_w185.txt +21:17:06 waited 60s for global r=780 +21:17:11 global r=780 load 4.280 -> 4.280 +21:17:45 step 184960 loss=2.9716 lr=1.39e-05 tok=24.352B dt=5326 wait=209 +21:18:49 step 184980 loss=2.9946 lr=1.39e-05 tok=24.354B dt=5391 wait=210 +21:19:54 step 185000 loss=2.9963 lr=1.39e-05 tok=24.357B dt=5455 wait=210 +21:20:57 step 185020 loss=3.0615 lr=1.38e-05 tok=24.360B dt=5519 wait=210 +21:22:02 step 185040 loss=2.8674 lr=1.38e-05 tok=24.362B dt=5584 wait=210 +21:23:07 step 185060 loss=3.0383 lr=1.38e-05 tok=24.365B dt=5649 wait=210 +21:24:12 step 185080 loss=2.8543 lr=1.38e-05 tok=24.368B dt=5714 wait=210 +21:25:17 step 185100 loss=2.9267 lr=1.38e-05 tok=24.370B dt=5778 wait=210 +21:25:29 pushed ckpt_w185.pt,log_w185.txt +21:26:45 waited 76s for global r=781 +21:26:50 global r=781 load 4.284 -> 4.282 +21:27:55 step 185120 loss=2.9673 lr=1.37e-05 tok=24.373B dt=5937 wait=211 +21:29:00 step 185140 loss=2.9102 lr=1.37e-05 tok=24.375B dt=6001 wait=211 +21:30:04 step 185160 loss=2.9791 lr=1.37e-05 tok=24.378B dt=6066 wait=211 +21:31:09 step 185180 loss=3.0242 lr=1.37e-05 tok=24.381B dt=6130 wait=211 +21:32:13 step 185200 loss=2.9842 lr=1.37e-05 tok=24.383B dt=6194 wait=211 +21:33:17 step 185220 loss=3.0673 lr=1.36e-05 tok=24.386B dt=6258 wait=211 +21:34:20 step 185240 loss=2.8951 lr=1.36e-05 tok=24.389B dt=6321 wait=212 +21:35:07 pushed ckpt_w185.pt,log_w185.txt +21:36:23 waited 76s for global r=782 +21:36:28 global r=782 load 4.280 -> 4.280 +21:37:01 step 185260 loss=2.9835 lr=1.36e-05 tok=24.391B dt=6482 wait=212 +21:38:05 step 185280 loss=2.8346 lr=1.36e-05 tok=24.394B dt=6546 wait=212 +21:39:09 step 185300 loss=2.6853 lr=1.36e-05 tok=24.396B dt=6610 wait=212 +21:40:13 step 185320 loss=2.9848 lr=1.36e-05 tok=24.399B dt=6674 wait=212 +21:41:16 step 185340 loss=2.9313 lr=1.35e-05 tok=24.402B dt=6738 wait=212 +21:42:20 step 185360 loss=2.9495 lr=1.35e-05 tok=24.404B dt=6801 wait=212 +21:43:24 step 185380 loss=2.9148 lr=1.35e-05 tok=24.407B dt=6865 wait=213 +21:44:28 step 185400 loss=2.8508 lr=1.35e-05 tok=24.410B dt=6930 wait=213 +21:44:43 pushed ckpt_w185.pt,log_w185.txt +21:45:59 waited 76s for global r=783 +21:46:04 global r=783 load 4.278 -> 4.279 +21:47:10 step 185420 loss=2.8731 lr=1.35e-05 tok=24.412B dt=7091 wait=213 +21:48:14 step 185440 loss=2.9416 lr=1.34e-05 tok=24.415B dt=7156 wait=213 +21:49:18 step 185460 loss=2.9500 lr=1.34e-05 tok=24.417B dt=7220 wait=213 +21:50:23 step 185480 loss=2.8597 lr=1.34e-05 tok=24.420B dt=7284 wait=213 +21:51:27 step 185500 loss=2.9203 lr=1.34e-05 tok=24.423B dt=7348 wait=213 +21:52:32 step 185520 loss=2.6715 lr=1.34e-05 tok=24.425B dt=7413 wait=214 +21:53:36 step 185540 loss=2.9149 lr=1.34e-05 tok=24.428B dt=7477 wait=214 +21:54:21 pushed ckpt_w185.pt,log_w185.txt +21:55:37 waited 76s for global r=784 +21:55:42 global r=784 load 4.284 -> 4.280 +21:56:15 step 185560 loss=2.9147 lr=1.33e-05 tok=24.431B dt=7636 wait=214 +21:57:19 step 185580 loss=2.9061 lr=1.33e-05 tok=24.433B dt=7700 wait=214 +21:58:23 step 185600 loss=2.9321 lr=1.33e-05 tok=24.436B dt=7764 wait=214 +21:59:26 step 185620 loss=2.9436 lr=1.33e-05 tok=24.438B dt=7828 wait=214 +22:00:31 step 185640 loss=2.8278 lr=1.33e-05 tok=24.441B dt=7892 wait=214 +22:01:35 step 185660 loss=3.0098 lr=1.32e-05 tok=24.444B dt=7956 wait=215 +22:02:39 step 185680 loss=2.8870 lr=1.32e-05 tok=24.446B dt=8020 wait=215 +22:03:43 step 185700 loss=3.0334 lr=1.32e-05 tok=24.449B dt=8084 wait=215 +22:03:55 pushed ckpt_w185.pt,log_w185.txt +22:05:11 waited 76s for global r=785 +22:05:19 global r=785 load 4.279 -> 4.279 +22:06:24 step 185720 loss=2.8268 lr=1.32e-05 tok=24.451B dt=8245 wait=215 +22:07:28 step 185740 loss=2.9378 lr=1.32e-05 tok=24.454B dt=8309 wait=215 +22:08:31 step 185760 loss=2.9622 lr=1.32e-05 tok=24.457B dt=8372 wait=215 +22:09:35 step 185780 loss=2.8584 lr=1.31e-05 tok=24.459B dt=8436 wait=216 +22:10:39 step 185800 loss=2.8117 lr=1.31e-05 tok=24.462B dt=8500 wait=216 +22:11:42 step 185820 loss=2.9779 lr=1.31e-05 tok=24.465B dt=8564 wait=216 +22:12:46 step 185840 loss=2.8634 lr=1.31e-05 tok=24.467B dt=8628 wait=216 +22:13:33 pushed ckpt_w185.pt,log_w185.txt +22:14:48 waited 76s for global r=786 +22:14:56 global r=786 load 4.283 -> 4.279 +22:15:29 step 185860 loss=3.0934 lr=1.31e-05 tok=24.470B dt=8790 wait=216 +22:16:33 step 185880 loss=2.9301 lr=1.30e-05 tok=24.472B dt=8854 wait=216 +22:17:36 step 185900 loss=2.9508 lr=1.30e-05 tok=24.475B dt=8918 wait=216 +22:18:40 step 185920 loss=3.0255 lr=1.30e-05 tok=24.478B dt=8982 wait=217 +22:19:44 step 185940 loss=2.8766 lr=1.30e-05 tok=24.480B dt=9045 wait=217 +22:20:48 step 185960 loss=3.0619 lr=1.30e-05 tok=24.483B dt=9109 wait=217 +22:21:52 step 185980 loss=2.8719 lr=1.29e-05 tok=24.486B dt=9173 wait=217 +22:22:56 step 186000 loss=2.8108 lr=1.29e-05 tok=24.488B dt=9237 wait=217 +22:23:09 pushed ckpt_w185.pt,log_w185.txt +22:24:25 waited 76s for global r=787 +22:24:34 global r=787 load 4.280 -> 4.280 +22:25:39 step 186020 loss=3.0137 lr=1.29e-05 tok=24.491B dt=9400 wait=217 +22:26:43 step 186040 loss=2.9308 lr=1.29e-05 tok=24.493B dt=9464 wait=217 +22:27:47 step 186060 loss=2.9495 lr=1.29e-05 tok=24.496B dt=9529 wait=218 +22:28:51 step 186080 loss=2.8222 lr=1.29e-05 tok=24.499B dt=9593 wait=218 +22:29:55 step 186100 loss=2.8459 lr=1.28e-05 tok=24.501B dt=9657 wait=218 +22:30:59 step 186120 loss=2.9574 lr=1.28e-05 tok=24.504B dt=9721 wait=218 +22:32:03 step 186140 loss=2.7969 lr=1.28e-05 tok=24.507B dt=9785 wait=218 +22:32:47 pushed ckpt_w185.pt,log_w185.txt +22:36:21 step 186160 loss=2.9394 lr=1.28e-05 tok=24.509B dt=10043 wait=218 +22:37:25 step 186180 loss=2.9153 lr=1.28e-05 tok=24.512B dt=10107 wait=218 +22:38:29 step 186200 loss=2.8733 lr=1.27e-05 tok=24.514B dt=10170 wait=219 +22:39:33 step 186220 loss=2.7634 lr=1.27e-05 tok=24.517B dt=10235 wait=219 +22:40:12 global r=788 load 4.280 -> 4.280 +22:40:44 step 186160 loss=2.9577 lr=1.28e-05 tok=24.520B dt=10305 wait=219 +22:41:48 step 186180 loss=2.9326 lr=1.28e-05 tok=24.522B dt=10369 wait=219 +22:42:51 step 186200 loss=2.9558 lr=1.27e-05 tok=24.525B dt=10433 wait=219 +22:43:55 step 186220 loss=3.0589 lr=1.27e-05 tok=24.528B dt=10496 wait=219 +22:44:59 step 186240 loss=2.8432 lr=1.27e-05 tok=24.530B dt=10561 wait=219 +22:46:02 step 186260 loss=2.9993 lr=1.27e-05 tok=24.533B dt=10624 wait=220 +22:47:06 step 186280 loss=3.0016 lr=1.27e-05 tok=24.535B dt=10687 wait=220 +22:48:09 step 186300 loss=2.9710 lr=1.27e-05 tok=24.538B dt=10750 wait=220 +22:48:20 pushed ckpt_w185.pt,log_w185.txt +22:52:03 global r=789 load 4.284 -> 4.283 +22:53:07 step 186320 loss=2.7892 lr=1.26e-05 tok=24.542B dt=11049 wait=220 +22:54:11 step 186340 loss=3.0731 lr=1.26e-05 tok=24.545B dt=11113 wait=220 +22:55:15 step 186360 loss=3.0256 lr=1.26e-05 tok=24.547B dt=11176 wait=220 +22:56:18 step 186380 loss=2.9603 lr=1.26e-05 tok=24.550B dt=11239 wait=221 +22:57:22 step 186400 loss=3.0569 lr=1.26e-05 tok=24.552B dt=11303 wait=221 +22:58:26 step 186420 loss=2.9704 lr=1.25e-05 tok=24.555B dt=11367 wait=221 +22:59:30 step 186440 loss=2.9348 lr=1.25e-05 tok=24.558B dt=11432 wait=221 +23:00:14 pushed ckpt_w185.pt,log_w185.txt +23:03:49 step 186460 loss=3.0617 lr=1.25e-05 tok=24.560B dt=11690 wait=221 +23:03:55 global r=790 load 4.278 -> 4.280 +23:04:27 step 186460 loss=3.0810 lr=1.25e-05 tok=24.562B dt=11729 wait=221 +23:05:31 step 186480 loss=2.9380 lr=1.25e-05 tok=24.564B dt=11792 wait=221 +23:06:35 step 186500 loss=3.0427 lr=1.25e-05 tok=24.567B dt=11856 wait=222 +23:07:39 step 186520 loss=3.1499 lr=1.24e-05 tok=24.569B dt=11920 wait=222 +23:08:44 step 186540 loss=2.9294 lr=1.24e-05 tok=24.572B dt=11985 wait=222 +23:09:47 step 186560 loss=3.1469 lr=1.24e-05 tok=24.575B dt=12048 wait=222 +23:10:51 step 186580 loss=3.0511 lr=1.24e-05 tok=24.577B dt=12112 wait=222 +23:11:54 step 186600 loss=3.0272 lr=1.24e-05 tok=24.580B dt=12176 wait=222 +23:12:06 pushed ckpt_w185.pt,log_w185.txt +23:15:49 global r=791 load 4.276 -> 4.278 +23:16:52 step 186620 loss=2.9313 lr=1.24e-05 tok=24.584B dt=12473 wait=222 +23:17:56 step 186640 loss=2.9910 lr=1.23e-05 tok=24.586B dt=12537 wait=223 +23:18:59 step 186660 loss=3.0009 lr=1.23e-05 tok=24.589B dt=12601 wait=223 +23:20:02 step 186680 loss=3.0148 lr=1.23e-05 tok=24.592B dt=12664 wait=223 +23:21:06 step 186700 loss=2.9145 lr=1.23e-05 tok=24.594B dt=12728 wait=223 +23:22:10 step 186720 loss=3.0999 lr=1.23e-05 tok=24.597B dt=12791 wait=223 +23:23:14 step 186740 loss=2.9086 lr=1.22e-05 tok=24.600B dt=12855 wait=223 +23:23:57 pushed ckpt_w185.pt,log_w185.txt +23:27:32 step 186760 loss=3.1165 lr=1.22e-05 tok=24.602B dt=13114 wait=223 +23:27:39 global r=792 load 4.279 -> 4.279 +23:28:12 step 186760 loss=3.0405 lr=1.22e-05 tok=24.604B dt=13153 wait=223 +23:29:15 step 186780 loss=3.0455 lr=1.22e-05 tok=24.606B dt=13216 wait=224 +23:30:19 step 186800 loss=3.0352 lr=1.22e-05 tok=24.609B dt=13280 wait=224 +23:31:22 step 186820 loss=3.0281 lr=1.22e-05 tok=24.611B dt=13344 wait=224 +23:32:26 step 186840 loss=2.9119 lr=1.22e-05 tok=24.614B dt=13408 wait=224 +23:33:31 step 186860 loss=2.9957 lr=1.21e-05 tok=24.617B dt=13472 wait=224 +23:34:35 step 186880 loss=3.0276 lr=1.21e-05 tok=24.619B dt=13536 wait=224 +23:35:38 step 186900 loss=2.9488 lr=1.21e-05 tok=24.622B dt=13600 wait=224 +23:35:51 pushed ckpt_w185.pt,log_w185.txt +23:39:38 global r=793 load 4.283 -> 4.283 +23:40:42 step 186920 loss=2.9333 lr=1.21e-05 tok=24.626B dt=13903 wait=225 +23:41:45 step 186940 loss=2.9409 lr=1.21e-05 tok=24.628B dt=13967 wait=225 +23:42:49 step 186960 loss=2.9490 lr=1.20e-05 tok=24.631B dt=14030 wait=225 +23:43:53 step 186980 loss=2.7667 lr=1.20e-05 tok=24.634B dt=14095 wait=225 +23:44:57 step 187000 loss=2.9155 lr=1.20e-05 tok=24.636B dt=14159 wait=225 +23:46:01 step 187020 loss=2.9186 lr=1.20e-05 tok=24.639B dt=14223 wait=225 +23:47:05 step 187040 loss=2.8679 lr=1.20e-05 tok=24.642B dt=14287 wait=226 +23:47:49 pushed ckpt_w185.pt,log_w185.txt +23:51:24 step 187060 loss=2.9056 lr=1.20e-05 tok=24.644B dt=14546 wait=226 +23:51:29 global r=794 load 4.279 -> 4.279 +23:52:01 step 187060 loss=2.9113 lr=1.20e-05 tok=24.645B dt=14583 wait=226 +23:53:05 step 187080 loss=2.8353 lr=1.19e-05 tok=24.648B dt=14647 wait=226 +23:54:09 step 187100 loss=2.8872 lr=1.19e-05 tok=24.651B dt=14711 wait=226 +23:55:14 step 187120 loss=3.0776 lr=1.19e-05 tok=24.653B dt=14775 wait=226 +23:56:18 step 187140 loss=3.0067 lr=1.19e-05 tok=24.656B dt=14839 wait=226 +23:57:23 step 187160 loss=3.0032 lr=1.19e-05 tok=24.659B dt=14904 wait=226 +23:58:27 step 187180 loss=2.9583 lr=1.18e-05 tok=24.661B dt=14968 wait=226 +23:59:31 step 187200 loss=3.0967 lr=1.18e-05 tok=24.664B dt=15033 wait=227 +23:59:47 pushed ckpt_w185.pt,log_w185.txt +00:02:49 waited 181s for global r=795 +00:02:53 global r=795 load 4.282 -> 4.279 +00:03:59 step 187220 loss=3.0282 lr=1.18e-05 tok=24.666B dt=15301 wait=227 +00:05:04 step 187240 loss=3.0247 lr=1.18e-05 tok=24.669B dt=15365 wait=227 +00:06:08 step 187260 loss=2.9471 lr=1.18e-05 tok=24.672B dt=15430 wait=227 +00:07:12 step 187280 loss=3.0347 lr=1.17e-05 tok=24.674B dt=15494 wait=227 +00:08:17 step 187300 loss=3.0519 lr=1.17e-05 tok=24.677B dt=15558 wait=227 +00:09:22 step 187320 loss=3.0819 lr=1.17e-05 tok=24.680B dt=15623 wait=227 +00:10:26 step 187340 loss=2.9632 lr=1.17e-05 tok=24.682B dt=15687 wait=228 diff --git a/log_w186.txt b/log_w186.txt new file mode 100644 index 0000000000000000000000000000000000000000..84e04a75764971534b0e4679c6db7866a2563ef0 --- /dev/null +++ b/log_w186.txt @@ -0,0 +1,168 @@ +launch +launch +20:03:00 name=w186 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 133/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:03:07 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:03:07 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +20:03:17 params 1.08B +20:03:18 resumed step=183750 seen=24.21B round=772 +20:03:18 stream salt=183750 +20:05:02 blocks compiled +20:05:02 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +20:09:00 step 183751 loss=2.7663 lr=1.50e-05 tok=24.205B dt=238 wait=208 +20:09:16 step 183752 loss=2.9371 lr=1.50e-05 tok=24.205B dt=254 wait=208 +20:09:42 step 183760 loss=3.2548 lr=1.50e-05 tok=24.206B dt=280 wait=208 +20:09:56 global r=773 load 4.274 -> 4.282 +20:11:01 step 183920 loss=3.0835 lr=1.49e-05 tok=24.209B dt=359 wait=209 +20:12:06 step 183940 loss=3.0851 lr=1.48e-05 tok=24.212B dt=424 wait=209 +20:13:11 step 183960 loss=3.1577 lr=1.48e-05 tok=24.214B dt=489 wait=209 +20:14:15 step 183980 loss=3.1272 lr=1.48e-05 tok=24.217B dt=554 wait=209 +20:15:21 step 184000 loss=3.1550 lr=1.48e-05 tok=24.219B dt=619 wait=209 +20:16:26 step 184020 loss=3.2014 lr=1.48e-05 tok=24.222B dt=684 wait=209 +20:17:31 step 184040 loss=3.0348 lr=1.47e-05 tok=24.225B dt=749 wait=209 +20:18:18 pushed ckpt_w186.pt,log_w186.txt +20:18:49 waited 30s for global r=774 +20:18:54 global r=774 load 4.280 -> 4.282 +20:19:29 step 184060 loss=3.1260 lr=1.47e-05 tok=24.227B dt=867 wait=210 +20:20:33 step 184080 loss=3.0220 lr=1.47e-05 tok=24.230B dt=932 wait=210 +20:21:38 step 184100 loss=3.0222 lr=1.47e-05 tok=24.233B dt=996 wait=210 +20:22:43 step 184120 loss=3.0503 lr=1.47e-05 tok=24.235B dt=1061 wait=210 +20:23:48 step 184140 loss=2.9472 lr=1.46e-05 tok=24.238B dt=1126 wait=210 +20:24:53 step 184160 loss=2.9725 lr=1.46e-05 tok=24.240B dt=1191 wait=210 +20:25:57 step 184180 loss=2.9406 lr=1.46e-05 tok=24.243B dt=1255 wait=210 +20:27:01 step 184200 loss=3.0193 lr=1.46e-05 tok=24.246B dt=1319 wait=211 +20:27:14 pushed ckpt_w186.pt,log_w186.txt +20:28:30 waited 76s for global r=775 +20:28:37 global r=775 load 4.281 -> 4.281 +20:29:43 step 184220 loss=2.9243 lr=1.46e-05 tok=24.248B dt=1481 wait=211 +20:30:47 step 184240 loss=3.0289 lr=1.46e-05 tok=24.251B dt=1545 wait=211 +20:31:51 step 184260 loss=3.0484 lr=1.45e-05 tok=24.254B dt=1609 wait=211 +20:32:55 step 184280 loss=2.8900 lr=1.45e-05 tok=24.256B dt=1673 wait=211 +20:33:59 step 184300 loss=3.0864 lr=1.45e-05 tok=24.259B dt=1737 wait=211 +20:35:03 step 184320 loss=2.9814 lr=1.45e-05 tok=24.261B dt=1801 wait=211 +20:36:08 step 184340 loss=2.9811 lr=1.45e-05 tok=24.264B dt=1866 wait=212 +20:36:53 pushed ckpt_w186.pt,log_w186.txt +20:38:23 waited 91s for global r=776 +20:38:30 global r=776 load 4.277 -> 4.281 +20:39:03 step 184360 loss=3.1618 lr=1.44e-05 tok=24.267B dt=2042 wait=212 +20:40:08 step 184380 loss=2.9288 lr=1.44e-05 tok=24.269B dt=2106 wait=212 +20:41:13 step 184400 loss=2.9393 lr=1.44e-05 tok=24.272B dt=2171 wait=212 +20:42:18 step 184420 loss=3.0863 lr=1.44e-05 tok=24.275B dt=2236 wait=212 +20:43:23 step 184440 loss=3.0259 lr=1.44e-05 tok=24.277B dt=2302 wait=212 +20:44:28 step 184460 loss=3.2000 lr=1.44e-05 tok=24.280B dt=2367 wait=212 +20:45:33 step 184480 loss=2.9385 lr=1.43e-05 tok=24.282B dt=2431 wait=213 +20:46:38 step 184500 loss=3.0752 lr=1.43e-05 tok=24.285B dt=2496 wait=213 +20:46:52 pushed ckpt_w186.pt,log_w186.txt +20:48:07 waited 76s for global r=777 +20:48:12 global r=777 load 4.283 -> 4.280 +20:49:18 step 184520 loss=2.9925 lr=1.43e-05 tok=24.288B dt=2656 wait=213 +20:50:22 step 184540 loss=3.1037 lr=1.43e-05 tok=24.290B dt=2721 wait=213 +20:51:27 step 184560 loss=3.0436 lr=1.43e-05 tok=24.293B dt=2785 wait=213 +20:52:32 step 184580 loss=3.1062 lr=1.42e-05 tok=24.296B dt=2850 wait=213 +20:53:36 step 184600 loss=3.0968 lr=1.42e-05 tok=24.298B dt=2915 wait=213 +20:54:42 step 184620 loss=3.0771 lr=1.42e-05 tok=24.301B dt=2980 wait=214 +20:55:46 step 184640 loss=3.0655 lr=1.42e-05 tok=24.303B dt=3044 wait=214 +20:56:30 pushed ckpt_w186.pt,log_w186.txt +20:57:46 waited 76s for global r=778 +20:57:53 global r=778 load 4.290 -> 4.280 +20:58:26 step 184660 loss=3.1798 lr=1.42e-05 tok=24.306B dt=3205 wait=214 +20:59:31 step 184680 loss=3.0410 lr=1.41e-05 tok=24.309B dt=3269 wait=214 +21:00:36 step 184700 loss=2.9863 lr=1.41e-05 tok=24.311B dt=3334 wait=214 +21:01:40 step 184720 loss=3.0432 lr=1.41e-05 tok=24.314B dt=3398 wait=214 +21:02:44 step 184740 loss=3.0693 lr=1.41e-05 tok=24.316B dt=3462 wait=214 +21:03:48 step 184760 loss=3.0287 lr=1.41e-05 tok=24.319B dt=3526 wait=215 +21:04:52 step 184780 loss=3.0286 lr=1.41e-05 tok=24.322B dt=3591 wait=215 +21:05:57 step 184800 loss=2.9599 lr=1.40e-05 tok=24.324B dt=3655 wait=215 +21:06:09 pushed ckpt_w186.pt,log_w186.txt +21:07:39 waited 91s for global r=779 +21:07:46 global r=779 load 4.285 -> 4.280 +21:08:51 step 184820 loss=2.8454 lr=1.40e-05 tok=24.327B dt=3829 wait=215 +21:09:55 step 184840 loss=2.9151 lr=1.40e-05 tok=24.330B dt=3893 wait=215 +21:11:01 step 184860 loss=2.5228 lr=1.40e-05 tok=24.332B dt=3959 wait=215 +21:12:05 step 184880 loss=2.9027 lr=1.40e-05 tok=24.335B dt=4023 wait=216 +21:13:08 step 184900 loss=2.9487 lr=1.39e-05 tok=24.337B dt=4086 wait=216 +21:14:13 step 184920 loss=3.0191 lr=1.39e-05 tok=24.340B dt=4151 wait=216 +21:15:17 step 184940 loss=2.9410 lr=1.39e-05 tok=24.343B dt=4215 wait=216 +21:16:02 pushed ckpt_w186.pt,log_w186.txt +21:17:18 waited 76s for global r=780 +21:17:22 global r=780 load 4.282 -> 4.280 +21:17:56 step 184960 loss=3.0193 lr=1.39e-05 tok=24.345B dt=4374 wait=216 +21:19:00 step 184980 loss=2.9832 lr=1.39e-05 tok=24.348B dt=4439 wait=216 +21:20:04 step 185000 loss=2.9733 lr=1.39e-05 tok=24.351B dt=4503 wait=216 +21:21:09 step 185020 loss=2.9822 lr=1.38e-05 tok=24.353B dt=4567 wait=217 +21:22:14 step 185040 loss=2.9017 lr=1.38e-05 tok=24.356B dt=4632 wait=217 +21:23:19 step 185060 loss=2.9480 lr=1.38e-05 tok=24.358B dt=4697 wait=217 +21:24:23 step 185080 loss=2.8965 lr=1.38e-05 tok=24.361B dt=4761 wait=217 +21:25:27 step 185100 loss=3.0124 lr=1.38e-05 tok=24.364B dt=4825 wait=217 +21:25:45 pushed ckpt_w186.pt,log_w186.txt +21:26:46 waited 61s for global r=781 +21:26:50 global r=781 load 4.285 -> 4.282 +21:27:56 step 185120 loss=2.9328 lr=1.37e-05 tok=24.366B dt=4974 wait=217 +21:29:01 step 185140 loss=2.9836 lr=1.37e-05 tok=24.369B dt=5039 wait=217 +21:30:05 step 185160 loss=2.9296 lr=1.37e-05 tok=24.372B dt=5103 wait=218 +21:31:10 step 185180 loss=2.8469 lr=1.37e-05 tok=24.374B dt=5168 wait=218 +21:32:14 step 185200 loss=2.9479 lr=1.37e-05 tok=24.377B dt=5232 wait=218 +21:33:19 step 185220 loss=2.9374 lr=1.36e-05 tok=24.379B dt=5297 wait=218 +21:34:22 step 185240 loss=2.9485 lr=1.36e-05 tok=24.382B dt=5360 wait=218 +21:35:08 pushed ckpt_w186.pt,log_w186.txt +21:36:24 waited 76s for global r=782 +21:36:29 global r=782 load 4.286 -> 4.280 +21:37:03 step 185260 loss=2.9481 lr=1.36e-05 tok=24.385B dt=5521 wait=218 +21:38:07 step 185280 loss=2.8849 lr=1.36e-05 tok=24.387B dt=5585 wait=219 +21:39:11 step 185300 loss=2.9521 lr=1.36e-05 tok=24.390B dt=5650 wait=219 +21:40:16 step 185320 loss=3.0254 lr=1.36e-05 tok=24.392B dt=5714 wait=219 +21:41:20 step 185340 loss=2.9530 lr=1.35e-05 tok=24.395B dt=5778 wait=219 +21:42:25 step 185360 loss=2.9633 lr=1.35e-05 tok=24.398B dt=5843 wait=219 +21:43:28 step 185380 loss=2.9898 lr=1.35e-05 tok=24.400B dt=5906 wait=219 +21:44:33 step 185400 loss=2.9837 lr=1.35e-05 tok=24.403B dt=5971 wait=219 +21:44:47 pushed ckpt_w186.pt,log_w186.txt +21:46:03 waited 76s for global r=783 +21:46:13 global r=783 load 4.283 -> 4.279 +21:47:18 step 185420 loss=2.9929 lr=1.35e-05 tok=24.406B dt=6136 wait=220 +21:48:22 step 185440 loss=3.0431 lr=1.34e-05 tok=24.408B dt=6200 wait=220 +21:49:25 step 185460 loss=3.0219 lr=1.34e-05 tok=24.411B dt=6264 wait=220 +21:50:30 step 185480 loss=2.9492 lr=1.34e-05 tok=24.413B dt=6328 wait=220 +21:51:33 step 185500 loss=2.9678 lr=1.34e-05 tok=24.416B dt=6391 wait=220 +21:52:38 step 185520 loss=3.0492 lr=1.34e-05 tok=24.419B dt=6456 wait=220 +21:53:41 step 185540 loss=2.9391 lr=1.34e-05 tok=24.421B dt=6519 wait=220 +21:54:27 pushed ckpt_w186.pt,log_w186.txt +21:55:42 waited 76s for global r=784 +21:55:49 global r=784 load 4.281 -> 4.280 +21:56:23 step 185560 loss=2.9051 lr=1.33e-05 tok=24.424B dt=6681 wait=221 +21:57:27 step 185580 loss=2.9470 lr=1.33e-05 tok=24.427B dt=6745 wait=221 +21:58:32 step 185600 loss=2.9355 lr=1.33e-05 tok=24.429B dt=6810 wait=221 +21:59:37 step 185620 loss=3.0568 lr=1.33e-05 tok=24.432B dt=6875 wait=221 +22:00:42 step 185640 loss=3.0313 lr=1.33e-05 tok=24.434B dt=6941 wait=221 +22:01:48 step 185660 loss=3.1758 lr=1.32e-05 tok=24.437B dt=7006 wait=221 +22:02:53 step 185680 loss=2.9657 lr=1.32e-05 tok=24.440B dt=7071 wait=221 +22:03:58 step 185700 loss=2.9828 lr=1.32e-05 tok=24.442B dt=7136 wait=222 +22:04:10 pushed ckpt_w186.pt,log_w186.txt +22:05:10 waited 61s for global r=785 +22:05:16 global r=785 load 4.280 -> 4.279 +22:06:22 step 185720 loss=3.0030 lr=1.32e-05 tok=24.445B dt=7280 wait=222 +22:07:27 step 185740 loss=2.9771 lr=1.32e-05 tok=24.448B dt=7345 wait=222 +22:08:31 step 185760 loss=2.9901 lr=1.32e-05 tok=24.450B dt=7409 wait=222 +22:09:35 step 185780 loss=2.8910 lr=1.31e-05 tok=24.453B dt=7473 wait=222 +22:10:40 step 185800 loss=2.9514 lr=1.31e-05 tok=24.455B dt=7538 wait=222 +22:11:45 step 185820 loss=3.0065 lr=1.31e-05 tok=24.458B dt=7603 wait=222 +22:12:49 step 185840 loss=2.8894 lr=1.31e-05 tok=24.461B dt=7667 wait=222 +22:13:33 pushed ckpt_w186.pt,log_w186.txt +22:14:49 waited 76s for global r=786 +22:14:57 global r=786 load 4.281 -> 4.279 +22:15:31 step 185860 loss=2.9615 lr=1.31e-05 tok=24.463B dt=7829 wait=223 +22:16:35 step 185880 loss=2.9777 lr=1.30e-05 tok=24.466B dt=7893 wait=223 +22:17:39 step 185900 loss=2.9435 lr=1.30e-05 tok=24.469B dt=7957 wait=223 +22:18:43 step 185920 loss=3.0516 lr=1.30e-05 tok=24.471B dt=8022 wait=223 +22:19:48 step 185940 loss=2.9455 lr=1.30e-05 tok=24.474B dt=8086 wait=223 +22:20:53 step 185960 loss=2.9857 lr=1.30e-05 tok=24.476B dt=8151 wait=223 +22:21:57 step 185980 loss=2.8598 lr=1.29e-05 tok=24.479B dt=8215 wait=223 +22:23:01 step 186000 loss=2.9001 lr=1.29e-05 tok=24.482B dt=8280 wait=224 +22:23:16 pushed ckpt_w186.pt,log_w186.txt +22:24:16 waited 61s for global r=787 +22:24:22 global r=787 load 4.282 -> 4.280 +22:25:27 step 186020 loss=2.9685 lr=1.29e-05 tok=24.484B dt=8426 wait=224 +22:26:32 step 186040 loss=2.9997 lr=1.29e-05 tok=24.487B dt=8490 wait=224 +22:27:36 step 186060 loss=2.9137 lr=1.29e-05 tok=24.489B dt=8554 wait=224 +22:28:40 step 186080 loss=2.8574 lr=1.29e-05 tok=24.492B dt=8619 wait=224 +22:29:44 step 186100 loss=2.9079 lr=1.28e-05 tok=24.495B dt=8682 wait=224 +22:30:49 step 186120 loss=2.9193 lr=1.28e-05 tok=24.497B dt=8747 wait=224 +22:31:53 step 186140 loss=2.8914 lr=1.28e-05 tok=24.500B dt=8811 wait=225 diff --git a/log_w187.txt b/log_w187.txt new file mode 100644 index 0000000000000000000000000000000000000000..2510989568f8b3db6d8d78a93aaff7ee9083c763 --- /dev/null +++ b/log_w187.txt @@ -0,0 +1,186 @@ +launch +04:51:43 name=w187 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 134/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +04:51:50 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +04:51:50 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +04:51:59 params 1.08B +04:52:01 resumed step=189600 seen=24.99B round=811 +04:52:01 stream salt=189600 +04:53:46 blocks compiled +04:53:46 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +04:57:17 step 189601 loss=2.9264 lr=9.60e-06 tok=24.986B dt=211 wait=181 +04:57:33 step 189602 loss=2.9368 lr=9.60e-06 tok=24.987B dt=227 wait=181 +04:58:31 step 189620 loss=3.2833 lr=9.59e-06 tok=24.989B dt=285 wait=182 +04:59:35 step 189640 loss=3.2347 lr=9.57e-06 tok=24.991B dt=349 wait=182 +05:00:39 step 189660 loss=3.1415 lr=9.55e-06 tok=24.994B dt=413 wait=182 +05:01:42 step 189680 loss=3.1632 lr=9.53e-06 tok=24.997B dt=476 wait=182 +05:02:46 step 189700 loss=3.1239 lr=9.51e-06 tok=24.999B dt=540 wait=182 +05:03:49 step 189720 loss=3.0164 lr=9.49e-06 tok=25.002B dt=604 wait=182 +05:04:53 step 189740 loss=3.0193 lr=9.48e-06 tok=25.005B dt=667 wait=182 +05:05:38 pushed ckpt_w187.pt,log_w187.txt +05:09:15 step 189760 loss=3.1331 lr=9.46e-06 tok=25.007B dt=929 wait=183 +05:09:29 global r=812 load 4.277 -> 4.278 +05:10:01 step 189760 loss=3.0672 lr=9.46e-06 tok=25.009B dt=975 wait=183 +05:11:04 step 189780 loss=2.9886 lr=9.44e-06 tok=25.011B dt=1039 wait=183 +05:12:09 step 189800 loss=2.9450 lr=9.42e-06 tok=25.014B dt=1103 wait=183 +05:13:14 step 189820 loss=3.1723 lr=9.40e-06 tok=25.016B dt=1168 wait=183 +05:14:18 step 189840 loss=3.0472 lr=9.38e-06 tok=25.019B dt=1232 wait=183 +05:15:23 step 189860 loss=3.0869 lr=9.36e-06 tok=25.022B dt=1297 wait=183 +05:16:27 step 189880 loss=3.1049 lr=9.35e-06 tok=25.024B dt=1361 wait=184 +05:17:30 step 189900 loss=3.1253 lr=9.33e-06 tok=25.027B dt=1424 wait=184 +05:17:53 pushed ckpt_w187.pt,log_w187.txt +05:18:53 waited 61s for global r=813 +05:19:01 global r=813 load 4.278 -> 4.279 +05:20:08 step 189920 loss=2.9556 lr=9.31e-06 tok=25.030B dt=1582 wait=184 +05:21:12 step 189940 loss=3.0475 lr=9.29e-06 tok=25.032B dt=1646 wait=184 +05:22:17 step 189960 loss=2.9173 lr=9.27e-06 tok=25.035B dt=1711 wait=184 +05:23:21 step 189980 loss=2.9426 lr=9.25e-06 tok=25.037B dt=1775 wait=184 +05:24:25 step 190000 loss=2.8734 lr=9.24e-06 tok=25.040B dt=1839 wait=185 +05:25:30 step 190020 loss=2.9150 lr=9.22e-06 tok=25.043B dt=1904 wait=185 +05:26:34 step 190040 loss=2.8308 lr=9.20e-06 tok=25.045B dt=1968 wait=185 +05:27:30 pushed ckpt_w187.pt,log_w187.txt +05:28:15 waited 45s for global r=814 +05:28:24 global r=814 load 4.283 -> 4.279 +05:29:00 step 190060 loss=2.9466 lr=9.18e-06 tok=25.048B dt=2114 wait=185 +05:30:05 step 190080 loss=2.7395 lr=9.16e-06 tok=25.050B dt=2179 wait=185 +05:31:11 step 190100 loss=2.7542 lr=9.14e-06 tok=25.053B dt=2245 wait=185 +05:32:16 step 190120 loss=2.8672 lr=9.12e-06 tok=25.056B dt=2310 wait=186 +05:33:21 step 190140 loss=2.7763 lr=9.11e-06 tok=25.058B dt=2375 wait=186 +05:34:25 step 190160 loss=2.9724 lr=9.09e-06 tok=25.061B dt=2440 wait=186 +05:35:30 step 190180 loss=2.9396 lr=9.07e-06 tok=25.064B dt=2504 wait=186 +05:36:35 step 190200 loss=2.8784 lr=9.05e-06 tok=25.066B dt=2569 wait=186 +05:36:48 pushed ckpt_w187.pt,log_w187.txt +05:37:49 waited 61s for global r=815 +05:37:58 global r=815 load 4.284 -> 4.279 +05:39:05 step 190220 loss=2.9225 lr=9.03e-06 tok=25.069B dt=2719 wait=186 +05:40:10 step 190240 loss=2.9205 lr=9.01e-06 tok=25.071B dt=2784 wait=186 +05:41:15 step 190260 loss=2.9385 lr=9.00e-06 tok=25.074B dt=2849 wait=187 +05:42:20 step 190280 loss=2.9122 lr=8.98e-06 tok=25.077B dt=2914 wait=187 +05:43:25 step 190300 loss=2.9944 lr=8.96e-06 tok=25.079B dt=2979 wait=187 +05:44:31 step 190320 loss=3.0805 lr=8.94e-06 tok=25.082B dt=3045 wait=187 +05:45:35 step 190340 loss=2.8794 lr=8.92e-06 tok=25.085B dt=3109 wait=187 +05:46:28 pushed ckpt_w187.pt,log_w187.txt +05:47:28 waited 61s for global r=816 +05:47:34 global r=816 load 4.283 -> 4.280 +05:48:08 step 190360 loss=3.1011 lr=8.90e-06 tok=25.087B dt=3262 wait=187 +05:49:13 step 190380 loss=2.9565 lr=8.88e-06 tok=25.090B dt=3327 wait=187 +05:50:19 step 190400 loss=3.0147 lr=8.87e-06 tok=25.092B dt=3393 wait=188 +05:51:24 step 190420 loss=3.0661 lr=8.85e-06 tok=25.095B dt=3458 wait=188 +05:52:29 step 190440 loss=2.9402 lr=8.83e-06 tok=25.098B dt=3524 wait=188 +05:53:35 step 190460 loss=3.0247 lr=8.81e-06 tok=25.100B dt=3589 wait=188 +05:54:39 step 190480 loss=2.9716 lr=8.79e-06 tok=25.103B dt=3654 wait=188 +05:55:45 step 190500 loss=2.9593 lr=8.77e-06 tok=25.106B dt=3719 wait=188 +05:55:59 pushed ckpt_w187.pt,log_w187.txt +05:57:00 waited 61s for global r=817 +05:57:11 global r=817 load 4.280 -> 4.280 +05:58:18 step 190520 loss=2.9682 lr=8.76e-06 tok=25.108B dt=3872 wait=189 +05:59:23 step 190540 loss=2.9788 lr=8.74e-06 tok=25.111B dt=3937 wait=189 +06:00:27 step 190560 loss=2.9333 lr=8.72e-06 tok=25.113B dt=4002 wait=189 +06:01:31 step 190580 loss=2.9578 lr=8.70e-06 tok=25.116B dt=4066 wait=189 +06:02:36 step 190600 loss=3.0671 lr=8.68e-06 tok=25.119B dt=4130 wait=191 +06:03:44 step 190620 loss=3.0219 lr=8.66e-06 tok=25.121B dt=4198 wait=195 +06:04:48 step 190640 loss=2.9062 lr=8.64e-06 tok=25.124B dt=4262 wait=195 +06:05:36 pushed ckpt_w187.pt,log_w187.txt +06:06:37 waited 61s for global r=818 +06:06:47 global r=818 load 4.281 -> 4.279 +06:07:20 step 190660 loss=3.0560 lr=8.63e-06 tok=25.127B dt=4414 wait=195 +06:08:23 step 190680 loss=3.0363 lr=8.61e-06 tok=25.129B dt=4478 wait=195 +06:09:26 step 190700 loss=2.9927 lr=8.59e-06 tok=25.132B dt=4541 wait=196 +06:10:30 step 190720 loss=3.1096 lr=8.57e-06 tok=25.134B dt=4604 wait=196 +06:11:34 step 190740 loss=2.9621 lr=8.55e-06 tok=25.137B dt=4668 wait=196 +06:12:37 step 190760 loss=3.0688 lr=8.53e-06 tok=25.140B dt=4731 wait=196 +06:13:40 step 190780 loss=3.0483 lr=8.52e-06 tok=25.142B dt=4795 wait=196 +06:14:44 step 190800 loss=3.0617 lr=8.50e-06 tok=25.145B dt=4858 wait=196 +06:14:59 pushed ckpt_w187.pt,log_w187.txt +06:16:16 waited 76s for global r=819 +06:16:22 global r=819 load 4.282 -> 4.280 +06:17:26 step 190820 loss=3.0766 lr=8.48e-06 tok=25.147B dt=5021 wait=197 +06:18:30 step 190840 loss=3.1062 lr=8.46e-06 tok=25.150B dt=5084 wait=197 +06:19:34 step 190860 loss=3.1429 lr=8.44e-06 tok=25.153B dt=5148 wait=197 +06:20:38 step 190880 loss=3.0502 lr=8.42e-06 tok=25.155B dt=5212 wait=197 +06:21:41 step 190900 loss=3.0867 lr=8.40e-06 tok=25.158B dt=5275 wait=197 +06:22:45 step 190920 loss=3.1669 lr=8.39e-06 tok=25.161B dt=5339 wait=197 +06:23:48 step 190940 loss=2.9483 lr=8.37e-06 tok=25.163B dt=5403 wait=197 +06:24:35 pushed ckpt_w187.pt,log_w187.txt +06:25:36 waited 61s for global r=820 +06:25:47 global r=820 load 4.280 -> 4.279 +06:26:20 step 190960 loss=3.1131 lr=8.35e-06 tok=25.166B dt=5554 wait=198 +06:27:25 step 190980 loss=2.9817 lr=8.33e-06 tok=25.168B dt=5619 wait=198 +06:28:29 step 191000 loss=2.9497 lr=8.31e-06 tok=25.171B dt=5683 wait=198 +06:29:33 step 191020 loss=3.0266 lr=8.29e-06 tok=25.174B dt=5747 wait=198 +06:30:37 step 191040 loss=2.9167 lr=8.28e-06 tok=25.176B dt=5811 wait=198 +06:31:41 step 191060 loss=3.0093 lr=8.26e-06 tok=25.179B dt=5875 wait=198 +06:32:45 step 191080 loss=2.9502 lr=8.24e-06 tok=25.182B dt=5939 wait=199 +06:33:49 step 191100 loss=2.8597 lr=8.22e-06 tok=25.184B dt=6004 wait=199 +06:34:00 pushed ckpt_w187.pt,log_w187.txt +06:35:19 waited 79s for global r=821 +06:35:28 global r=821 load 4.279 -> 4.279 +06:36:33 step 191120 loss=2.8784 lr=8.20e-06 tok=25.187B dt=6167 wait=199 +06:37:36 step 191140 loss=2.9140 lr=8.18e-06 tok=25.189B dt=6231 wait=199 +06:38:40 step 191160 loss=2.9756 lr=8.16e-06 tok=25.192B dt=6294 wait=199 +06:39:44 step 191180 loss=2.9099 lr=8.15e-06 tok=25.195B dt=6358 wait=199 +06:40:48 step 191200 loss=2.9618 lr=8.13e-06 tok=25.197B dt=6422 wait=199 +06:41:52 step 191220 loss=2.9438 lr=8.11e-06 tok=25.200B dt=6486 wait=200 +06:42:57 step 191240 loss=2.9764 lr=8.09e-06 tok=25.203B dt=6551 wait=200 +06:43:49 pushed ckpt_w187.pt,log_w187.txt +06:44:50 waited 61s for global r=822 +06:44:57 global r=822 load 4.281 -> 4.278 +06:45:31 step 191260 loss=3.0136 lr=8.07e-06 tok=25.205B dt=6705 wait=200 +06:46:35 step 191280 loss=2.9817 lr=8.05e-06 tok=25.208B dt=6769 wait=200 +06:47:40 step 191300 loss=2.9392 lr=8.04e-06 tok=25.210B dt=6834 wait=200 +06:48:45 step 191320 loss=3.0431 lr=8.02e-06 tok=25.213B dt=6899 wait=200 +06:49:50 step 191340 loss=2.8764 lr=8.00e-06 tok=25.216B dt=6964 wait=201 +06:50:54 step 191360 loss=2.9639 lr=7.98e-06 tok=25.218B dt=7028 wait=201 +06:51:58 step 191380 loss=2.8337 lr=7.96e-06 tok=25.221B dt=7092 wait=201 +06:53:03 step 191400 loss=2.8547 lr=7.94e-06 tok=25.223B dt=7157 wait=201 +06:53:23 pushed ckpt_w187.pt,log_w187.txt +06:54:13 waited 51s for global r=823 +06:54:22 global r=823 load 4.278 -> 4.279 +06:55:28 step 191420 loss=2.8756 lr=7.92e-06 tok=25.226B dt=7302 wait=201 +06:56:32 step 191440 loss=2.8846 lr=7.91e-06 tok=25.229B dt=7366 wait=201 +06:57:36 step 191460 loss=2.8624 lr=7.89e-06 tok=25.231B dt=7430 wait=201 +06:58:39 step 191480 loss=2.8237 lr=7.87e-06 tok=25.234B dt=7494 wait=202 +06:59:43 step 191500 loss=2.8156 lr=7.85e-06 tok=25.237B dt=7557 wait=202 +07:00:47 step 191520 loss=2.9731 lr=7.83e-06 tok=25.239B dt=7621 wait=202 +07:01:51 step 191540 loss=2.8181 lr=7.81e-06 tok=25.242B dt=7685 wait=202 +07:02:34 pushed ckpt_w187.pt,log_w187.txt +07:03:45 waited 71s for global r=824 +07:03:52 global r=824 load 4.280 -> 4.279 +07:04:26 step 191560 loss=3.0545 lr=7.80e-06 tok=25.244B dt=7840 wait=202 +07:05:29 step 191580 loss=2.9252 lr=7.78e-06 tok=25.247B dt=7903 wait=202 +07:06:32 step 191600 loss=2.9698 lr=7.76e-06 tok=25.250B dt=7966 wait=203 +07:07:36 step 191620 loss=3.0298 lr=7.74e-06 tok=25.252B dt=8030 wait=203 +07:08:39 step 191640 loss=3.0290 lr=7.72e-06 tok=25.255B dt=8093 wait=203 +07:09:42 step 191660 loss=3.0103 lr=7.70e-06 tok=25.258B dt=8156 wait=203 +07:10:46 step 191680 loss=2.9249 lr=7.68e-06 tok=25.260B dt=8220 wait=203 +07:11:49 step 191700 loss=3.0170 lr=7.67e-06 tok=25.263B dt=8284 wait=203 +07:12:00 pushed ckpt_w187.pt,log_w187.txt +07:13:16 waited 76s for global r=825 +07:13:24 global r=825 load 4.277 -> 4.279 +07:14:29 step 191720 loss=2.9759 lr=7.65e-06 tok=25.265B dt=8443 wait=203 +07:15:33 step 191740 loss=3.0274 lr=7.63e-06 tok=25.268B dt=8507 wait=203 +07:16:36 step 191760 loss=3.0516 lr=7.61e-06 tok=25.271B dt=8570 wait=204 +07:17:39 step 191780 loss=2.8369 lr=7.59e-06 tok=25.273B dt=8633 wait=204 +07:18:42 step 191800 loss=2.9283 lr=7.57e-06 tok=25.276B dt=8696 wait=204 +07:19:46 step 191820 loss=2.9804 lr=7.55e-06 tok=25.279B dt=8760 wait=204 +07:20:50 step 191840 loss=2.8133 lr=7.54e-06 tok=25.281B dt=8824 wait=204 +07:21:34 pushed ckpt_w187.pt,log_w187.txt +07:22:51 waited 77s for global r=826 +07:23:01 global r=826 load 4.281 -> 4.280 +07:23:34 step 191860 loss=3.0374 lr=7.52e-06 tok=25.284B dt=8988 wait=204 +07:24:38 step 191880 loss=2.9222 lr=7.50e-06 tok=25.286B dt=9052 wait=205 +07:25:41 step 191900 loss=2.9080 lr=7.48e-06 tok=25.289B dt=9115 wait=205 +07:26:44 step 191920 loss=2.9862 lr=7.46e-06 tok=25.292B dt=9179 wait=205 +07:27:48 step 191940 loss=2.8730 lr=7.44e-06 tok=25.294B dt=9242 wait=205 +07:28:52 step 191960 loss=2.9440 lr=7.43e-06 tok=25.297B dt=9306 wait=205 +07:29:55 step 191980 loss=2.7803 lr=7.41e-06 tok=25.300B dt=9369 wait=205 +07:30:58 step 192000 loss=2.8940 lr=7.39e-06 tok=25.302B dt=9432 wait=205 +07:31:11 pushed ckpt_w187.pt,log_w187.txt +07:32:29 waited 78s for global r=827 +07:32:36 global r=827 load 4.280 -> 4.280 +07:33:40 step 192020 loss=2.9097 lr=7.37e-06 tok=25.305B dt=9594 wait=206 +07:34:44 step 192040 loss=2.9297 lr=7.35e-06 tok=25.307B dt=9658 wait=206 +07:35:48 step 192060 loss=2.9183 lr=7.33e-06 tok=25.310B dt=9722 wait=206 +07:36:52 step 192080 loss=2.8210 lr=7.31e-06 tok=25.313B dt=9786 wait=206 +07:37:56 step 192100 loss=2.8643 lr=7.30e-06 tok=25.315B dt=9850 wait=206 +07:39:00 step 192120 loss=2.9159 lr=7.28e-06 tok=25.318B dt=9914 wait=206 +07:40:04 step 192140 loss=2.7812 lr=7.26e-06 tok=25.320B dt=9978 wait=207 diff --git a/log_w188.txt b/log_w188.txt new file mode 100644 index 0000000000000000000000000000000000000000..06f83f0ade8df034be4049632cfa6a5b886b77bd --- /dev/null +++ b/log_w188.txt @@ -0,0 +1,701 @@ +launch +04:53:21 name=w188 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 135/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +04:53:27 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +04:53:27 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +04:53:37 params 1.08B +04:53:38 resumed step=189600 seen=24.99B round=811 +04:53:38 stream salt=189600 +04:55:23 blocks compiled +04:55:23 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +04:59:10 step 189601 loss=3.1122 lr=9.60e-06 tok=24.986B dt=227 wait=193 +04:59:25 step 189602 loss=3.3001 lr=9.60e-06 tok=24.987B dt=243 wait=193 +05:00:24 step 189620 loss=3.3134 lr=9.59e-06 tok=24.989B dt=301 wait=193 +05:01:30 step 189640 loss=3.2603 lr=9.57e-06 tok=24.991B dt=367 wait=193 +05:02:35 step 189660 loss=3.1691 lr=9.55e-06 tok=24.994B dt=432 wait=193 +05:03:39 step 189680 loss=3.3131 lr=9.53e-06 tok=24.997B dt=496 wait=193 +05:04:44 step 189700 loss=3.2792 lr=9.51e-06 tok=24.999B dt=561 wait=193 +05:05:49 step 189720 loss=3.2406 lr=9.49e-06 tok=25.002B dt=626 wait=194 +05:06:55 step 189740 loss=3.1764 lr=9.48e-06 tok=25.005B dt=692 wait=194 +05:07:43 pushed ckpt_w188.pt,log_w188.txt +05:08:59 waited 76s for global r=812 +05:09:09 global r=812 load 4.280 -> 4.278 +05:09:45 step 189760 loss=3.2216 lr=9.46e-06 tok=25.007B dt=862 wait=194 +05:10:50 step 189780 loss=3.0549 lr=9.44e-06 tok=25.010B dt=927 wait=194 +05:11:54 step 189800 loss=3.1146 lr=9.42e-06 tok=25.012B dt=991 wait=194 +05:12:59 step 189820 loss=3.1460 lr=9.40e-06 tok=25.015B dt=1056 wait=194 +05:14:04 step 189840 loss=3.0929 lr=9.38e-06 tok=25.018B dt=1121 wait=194 +05:15:09 step 189860 loss=3.0480 lr=9.36e-06 tok=25.020B dt=1186 wait=195 +05:16:14 step 189880 loss=2.9551 lr=9.35e-06 tok=25.023B dt=1251 wait=195 +05:17:18 step 189900 loss=3.0196 lr=9.33e-06 tok=25.026B dt=1315 wait=195 +05:17:30 pushed ckpt_w188.pt,log_w188.txt +05:18:46 waited 76s for global r=813 +05:18:53 global r=813 load 4.280 -> 4.279 +05:19:59 step 189920 loss=2.7417 lr=9.31e-06 tok=25.028B dt=1476 wait=195 +05:21:04 step 189940 loss=3.0496 lr=9.29e-06 tok=25.031B dt=1541 wait=195 +05:22:08 step 189960 loss=2.9860 lr=9.27e-06 tok=25.033B dt=1605 wait=195 +05:23:12 step 189980 loss=2.9531 lr=9.25e-06 tok=25.036B dt=1669 wait=195 +05:24:16 step 190000 loss=3.0169 lr=9.24e-06 tok=25.039B dt=1733 wait=196 +05:25:21 step 190020 loss=3.0066 lr=9.22e-06 tok=25.041B dt=1798 wait=196 +05:26:25 step 190040 loss=2.9148 lr=9.20e-06 tok=25.044B dt=1862 wait=196 +05:27:11 pushed ckpt_w188.pt,log_w188.txt +05:28:27 waited 76s for global r=814 +05:28:33 global r=814 load 4.279 -> 4.279 +05:29:09 step 190060 loss=2.9693 lr=9.18e-06 tok=25.047B dt=2026 wait=196 +05:30:13 step 190080 loss=2.8219 lr=9.16e-06 tok=25.049B dt=2090 wait=196 +05:31:18 step 190100 loss=2.7679 lr=9.14e-06 tok=25.052B dt=2155 wait=196 +05:32:24 step 190120 loss=2.9677 lr=9.12e-06 tok=25.054B dt=2221 wait=196 +05:33:29 step 190140 loss=2.8877 lr=9.11e-06 tok=25.057B dt=2286 wait=197 +05:34:34 step 190160 loss=2.8922 lr=9.09e-06 tok=25.060B dt=2351 wait=197 +05:35:38 step 190180 loss=2.8907 lr=9.07e-06 tok=25.062B dt=2416 wait=197 +05:36:43 step 190200 loss=2.9472 lr=9.05e-06 tok=25.065B dt=2480 wait=197 +05:36:56 pushed ckpt_w188.pt,log_w188.txt +05:37:56 waited 60s for global r=815 +05:38:01 global r=815 load 4.280 -> 4.279 +05:39:06 step 190220 loss=2.8849 lr=9.03e-06 tok=25.068B dt=2623 wait=197 +05:40:11 step 190240 loss=2.9477 lr=9.01e-06 tok=25.070B dt=2688 wait=197 +05:41:15 step 190260 loss=2.9903 lr=9.00e-06 tok=25.073B dt=2752 wait=197 +05:42:19 step 190280 loss=2.8341 lr=8.98e-06 tok=25.075B dt=2816 wait=198 +05:43:24 step 190300 loss=2.9401 lr=8.96e-06 tok=25.078B dt=2881 wait=198 +05:44:29 step 190320 loss=3.0372 lr=8.94e-06 tok=25.081B dt=2946 wait=198 +05:45:34 step 190340 loss=2.8629 lr=8.92e-06 tok=25.083B dt=3011 wait=198 +05:46:20 pushed ckpt_w188.pt,log_w188.txt +05:47:20 waited 61s for global r=816 +05:47:28 global r=816 load 4.279 -> 4.280 +05:48:02 step 190360 loss=3.1214 lr=8.90e-06 tok=25.086B dt=3159 wait=198 +05:49:06 step 190380 loss=2.8408 lr=8.88e-06 tok=25.088B dt=3223 wait=198 +05:50:11 step 190400 loss=3.0021 lr=8.87e-06 tok=25.091B dt=3288 wait=198 +05:51:16 step 190420 loss=3.0368 lr=8.85e-06 tok=25.094B dt=3353 wait=199 +05:52:21 step 190440 loss=3.0238 lr=8.83e-06 tok=25.096B dt=3418 wait=199 +05:53:25 step 190460 loss=3.0402 lr=8.81e-06 tok=25.099B dt=3482 wait=199 +05:54:31 step 190480 loss=2.9148 lr=8.79e-06 tok=25.102B dt=3548 wait=199 +05:55:36 step 190500 loss=2.7494 lr=8.77e-06 tok=25.104B dt=3613 wait=199 +05:55:49 pushed ckpt_w188.pt,log_w188.txt +05:57:04 waited 76s for global r=817 +05:57:09 global r=817 load 4.280 -> 4.280 +05:58:15 step 190520 loss=3.0807 lr=8.76e-06 tok=25.107B dt=3772 wait=199 +05:59:20 step 190540 loss=3.2901 lr=8.74e-06 tok=25.109B dt=3837 wait=199 +06:00:25 step 190560 loss=3.1266 lr=8.72e-06 tok=25.112B dt=3902 wait=199 +06:01:30 step 190580 loss=3.0063 lr=8.70e-06 tok=25.115B dt=3967 wait=200 +06:02:35 step 190600 loss=3.0482 lr=8.68e-06 tok=25.117B dt=4032 wait=200 +06:03:42 step 190620 loss=3.1072 lr=8.66e-06 tok=25.120B dt=4099 wait=202 +06:04:47 step 190640 loss=3.1002 lr=8.64e-06 tok=25.123B dt=4164 wait=203 +06:05:33 pushed ckpt_w188.pt,log_w188.txt +06:06:34 waited 61s for global r=818 +06:06:40 global r=818 load 4.282 -> 4.279 +06:07:14 step 190660 loss=3.0762 lr=8.63e-06 tok=25.125B dt=4311 wait=203 +06:08:18 step 190680 loss=3.1055 lr=8.61e-06 tok=25.128B dt=4375 wait=203 +06:09:24 step 190700 loss=3.0588 lr=8.59e-06 tok=25.130B dt=4441 wait=203 +06:10:29 step 190720 loss=3.1590 lr=8.57e-06 tok=25.133B dt=4506 wait=203 +06:11:35 step 190740 loss=2.9845 lr=8.55e-06 tok=25.136B dt=4572 wait=203 +06:12:40 step 190760 loss=3.0216 lr=8.53e-06 tok=25.138B dt=4637 wait=203 +06:13:46 step 190780 loss=2.9058 lr=8.52e-06 tok=25.141B dt=4703 wait=204 +06:14:51 step 190800 loss=3.0424 lr=8.50e-06 tok=25.144B dt=4768 wait=204 +06:15:04 pushed ckpt_w188.pt,log_w188.txt +06:16:05 waited 61s for global r=819 +06:16:15 global r=819 load 4.280 -> 4.280 +06:17:22 step 190820 loss=2.9879 lr=8.48e-06 tok=25.146B dt=4919 wait=204 +06:18:25 step 190840 loss=3.0153 lr=8.46e-06 tok=25.149B dt=4982 wait=204 +06:19:31 step 190860 loss=2.9717 lr=8.44e-06 tok=25.151B dt=5048 wait=204 +06:20:36 step 190880 loss=2.9099 lr=8.42e-06 tok=25.154B dt=5113 wait=204 +06:21:41 step 190900 loss=2.9512 lr=8.40e-06 tok=25.157B dt=5178 wait=204 +06:22:46 step 190920 loss=2.9807 lr=8.39e-06 tok=25.159B dt=5243 wait=205 +06:23:50 step 190940 loss=2.7711 lr=8.37e-06 tok=25.162B dt=5307 wait=205 +06:24:35 pushed ckpt_w188.pt,log_w188.txt +06:25:36 waited 60s for global r=820 +06:25:42 global r=820 load 4.281 -> 4.279 +06:26:16 step 190960 loss=3.0161 lr=8.35e-06 tok=25.165B dt=5453 wait=205 +06:27:20 step 190980 loss=2.8363 lr=8.33e-06 tok=25.167B dt=5517 wait=205 +06:28:25 step 191000 loss=2.8648 lr=8.31e-06 tok=25.170B dt=5582 wait=205 +06:29:29 step 191020 loss=2.9409 lr=8.29e-06 tok=25.172B dt=5646 wait=205 +06:30:34 step 191040 loss=2.9269 lr=8.28e-06 tok=25.175B dt=5711 wait=205 +06:31:38 step 191060 loss=2.9606 lr=8.26e-06 tok=25.178B dt=5775 wait=206 +06:32:43 step 191080 loss=2.8869 lr=8.24e-06 tok=25.180B dt=5840 wait=206 +06:33:48 step 191100 loss=2.8408 lr=8.22e-06 tok=25.183B dt=5905 wait=206 +06:33:59 pushed ckpt_w188.pt,log_w188.txt +06:35:15 waited 76s for global r=821 +06:35:20 global r=821 load 4.279 -> 4.279 +06:36:26 step 191120 loss=2.8246 lr=8.20e-06 tok=25.185B dt=6063 wait=206 +06:37:30 step 191140 loss=2.9684 lr=8.18e-06 tok=25.188B dt=6127 wait=206 +06:38:34 step 191160 loss=3.0042 lr=8.16e-06 tok=25.191B dt=6191 wait=206 +06:39:37 step 191180 loss=2.8347 lr=8.15e-06 tok=25.193B dt=6254 wait=207 +06:40:42 step 191200 loss=2.8401 lr=8.13e-06 tok=25.196B dt=6319 wait=207 +06:41:46 step 191220 loss=3.0109 lr=8.11e-06 tok=25.199B dt=6383 wait=207 +06:42:50 step 191240 loss=2.8200 lr=8.09e-06 tok=25.201B dt=6447 wait=207 +06:43:34 pushed ckpt_w188.pt,log_w188.txt +06:44:50 waited 76s for global r=822 +06:44:55 global r=822 load 4.279 -> 4.278 +06:45:28 step 191260 loss=2.9270 lr=8.07e-06 tok=25.204B dt=6605 wait=207 +06:46:32 step 191280 loss=2.9365 lr=8.05e-06 tok=25.206B dt=6669 wait=207 +06:47:37 step 191300 loss=2.9303 lr=8.04e-06 tok=25.209B dt=6734 wait=207 +06:48:42 step 191320 loss=3.0322 lr=8.02e-06 tok=25.212B dt=6799 wait=208 +06:49:47 step 191340 loss=2.8324 lr=8.00e-06 tok=25.214B dt=6864 wait=208 +06:50:51 step 191360 loss=3.0229 lr=7.98e-06 tok=25.217B dt=6928 wait=208 +06:51:56 step 191380 loss=2.8690 lr=7.96e-06 tok=25.220B dt=6993 wait=208 +06:53:02 step 191400 loss=3.0393 lr=7.94e-06 tok=25.222B dt=7059 wait=208 +06:53:14 pushed ckpt_w188.pt,log_w188.txt +06:54:15 waited 60s for global r=823 +06:54:21 global r=823 load 4.279 -> 4.279 +06:55:27 step 191420 loss=2.8686 lr=7.92e-06 tok=25.225B dt=7205 wait=208 +06:56:32 step 191440 loss=2.9737 lr=7.91e-06 tok=25.227B dt=7269 wait=209 +06:57:37 step 191460 loss=2.7371 lr=7.89e-06 tok=25.230B dt=7334 wait=209 +06:58:42 step 191480 loss=2.8618 lr=7.87e-06 tok=25.233B dt=7399 wait=209 +06:59:46 step 191500 loss=2.9155 lr=7.85e-06 tok=25.235B dt=7463 wait=209 +07:00:50 step 191520 loss=2.8117 lr=7.83e-06 tok=25.238B dt=7527 wait=209 +07:01:55 step 191540 loss=2.7432 lr=7.81e-06 tok=25.241B dt=7592 wait=209 +07:02:41 pushed ckpt_w188.pt,log_w188.txt +07:03:42 waited 60s for global r=824 +07:03:47 global r=824 load 4.280 -> 4.279 +07:04:21 step 191560 loss=2.8566 lr=7.80e-06 tok=25.243B dt=7738 wait=209 +07:05:26 step 191580 loss=2.6357 lr=7.78e-06 tok=25.246B dt=7803 wait=210 +07:06:30 step 191600 loss=2.7824 lr=7.76e-06 tok=25.248B dt=7867 wait=210 +07:07:35 step 191620 loss=2.8701 lr=7.74e-06 tok=25.251B dt=7932 wait=210 +07:08:40 step 191640 loss=2.6481 lr=7.72e-06 tok=25.254B dt=7997 wait=210 +07:09:45 step 191660 loss=2.8432 lr=7.70e-06 tok=25.256B dt=8062 wait=210 +07:10:49 step 191680 loss=2.7739 lr=7.68e-06 tok=25.259B dt=8126 wait=210 +07:11:53 step 191700 loss=2.7932 lr=7.67e-06 tok=25.262B dt=8190 wait=211 +07:12:07 pushed ckpt_w188.pt,log_w188.txt +07:13:22 waited 76s for global r=825 +07:13:27 global r=825 load 4.280 -> 4.279 +07:14:32 step 191720 loss=2.6627 lr=7.65e-06 tok=25.264B dt=8349 wait=211 +07:15:37 step 191740 loss=2.6435 lr=7.63e-06 tok=25.267B dt=8414 wait=211 +07:16:42 step 191760 loss=2.8248 lr=7.61e-06 tok=25.269B dt=8479 wait=211 +07:17:47 step 191780 loss=2.8031 lr=7.59e-06 tok=25.272B dt=8544 wait=211 +07:18:51 step 191800 loss=2.8346 lr=7.57e-06 tok=25.275B dt=8608 wait=211 +07:19:56 step 191820 loss=2.9973 lr=7.55e-06 tok=25.277B dt=8673 wait=211 +07:21:01 step 191840 loss=2.9199 lr=7.54e-06 tok=25.280B dt=8738 wait=211 +07:21:50 pushed ckpt_w188.pt,log_w188.txt +07:22:51 waited 61s for global r=826 +07:22:58 global r=826 load 4.282 -> 4.280 +07:23:31 step 191860 loss=2.9379 lr=7.52e-06 tok=25.282B dt=8888 wait=212 +07:24:36 step 191880 loss=2.8014 lr=7.50e-06 tok=25.285B dt=8953 wait=212 +07:25:40 step 191900 loss=2.7801 lr=7.48e-06 tok=25.288B dt=9017 wait=212 +07:26:45 step 191920 loss=2.7982 lr=7.46e-06 tok=25.290B dt=9082 wait=212 +07:27:49 step 191940 loss=2.7485 lr=7.44e-06 tok=25.293B dt=9146 wait=212 +07:28:53 step 191960 loss=2.8574 lr=7.43e-06 tok=25.296B dt=9210 wait=212 +07:29:58 step 191980 loss=2.7585 lr=7.41e-06 tok=25.298B dt=9275 wait=213 +07:31:03 step 192000 loss=2.6665 lr=7.39e-06 tok=25.301B dt=9340 wait=213 +07:31:17 pushed ckpt_w188.pt,log_w188.txt +07:32:17 waited 60s for global r=827 +07:32:23 global r=827 load 4.280 -> 4.280 +07:33:29 step 192020 loss=2.7195 lr=7.37e-06 tok=25.303B dt=9486 wait=213 +07:34:33 step 192040 loss=2.8472 lr=7.35e-06 tok=25.306B dt=9550 wait=213 +07:35:38 step 192060 loss=2.6925 lr=7.33e-06 tok=25.309B dt=9615 wait=213 +07:36:42 step 192080 loss=2.7970 lr=7.31e-06 tok=25.311B dt=9679 wait=213 +07:37:47 step 192100 loss=2.8011 lr=7.30e-06 tok=25.314B dt=9744 wait=213 +07:38:51 step 192120 loss=2.8638 lr=7.28e-06 tok=25.317B dt=9808 wait=214 +07:39:56 step 192140 loss=2.7353 lr=7.26e-06 tok=25.319B dt=9873 wait=214 +07:40:44 pushed ckpt_w188.pt,log_w188.txt +07:41:44 waited 60s for global r=828 +07:41:50 global r=828 load 4.281 -> 4.279 +07:42:24 step 192160 loss=2.8773 lr=7.24e-06 tok=25.322B dt=10021 wait=214 +07:43:28 step 192180 loss=2.6900 lr=7.22e-06 tok=25.324B dt=10085 wait=214 +07:44:32 step 192200 loss=2.7217 lr=7.20e-06 tok=25.327B dt=10149 wait=214 +07:45:36 step 192220 loss=2.8897 lr=7.19e-06 tok=25.330B dt=10213 wait=214 +07:46:42 step 192240 loss=2.7748 lr=7.17e-06 tok=25.332B dt=10279 wait=214 +07:47:46 step 192260 loss=2.8705 lr=7.15e-06 tok=25.335B dt=10343 wait=215 +07:48:50 step 192280 loss=2.8864 lr=7.13e-06 tok=25.338B dt=10408 wait=215 +07:49:55 step 192300 loss=2.8780 lr=7.11e-06 tok=25.340B dt=10472 wait=215 +07:50:07 pushed ckpt_w188.pt,log_w188.txt +07:54:16 step 192320 loss=2.8568 lr=7.09e-06 tok=25.343B dt=10733 wait=215 +07:55:21 step 192340 loss=2.7689 lr=7.07e-06 tok=25.345B dt=10798 wait=215 +07:56:26 step 192360 loss=2.7618 lr=7.06e-06 tok=25.348B dt=10863 wait=215 +07:56:37 global r=829 load 4.282 -> 4.281 +07:57:42 step 192320 loss=2.8094 lr=7.09e-06 tok=25.351B dt=10939 wait=215 +07:58:47 step 192340 loss=2.8766 lr=7.07e-06 tok=25.353B dt=11004 wait=216 +07:59:52 step 192360 loss=2.9325 lr=7.06e-06 tok=25.356B dt=11069 wait=216 +08:00:57 step 192380 loss=2.8186 lr=7.04e-06 tok=25.358B dt=11134 wait=216 +08:02:02 step 192400 loss=2.7379 lr=7.02e-06 tok=25.361B dt=11199 wait=216 +08:03:06 step 192420 loss=2.9491 lr=7.00e-06 tok=25.364B dt=11263 wait=216 +08:04:10 step 192440 loss=2.7910 lr=6.98e-06 tok=25.366B dt=11327 wait=216 +08:04:57 pushed ckpt_w188.pt,log_w188.txt +08:08:33 step 192460 loss=2.8626 lr=6.96e-06 tok=25.369B dt=11590 wait=217 +08:08:41 global r=830 load 4.283 -> 4.282 +08:09:13 step 192460 loss=2.8279 lr=6.96e-06 tok=25.370B dt=11630 wait=217 +08:10:18 step 192480 loss=2.7631 lr=6.95e-06 tok=25.373B dt=11695 wait=217 +08:11:22 step 192500 loss=2.7607 lr=6.93e-06 tok=25.376B dt=11759 wait=217 +08:12:26 step 192520 loss=2.9144 lr=6.91e-06 tok=25.378B dt=11823 wait=217 +08:13:32 step 192540 loss=2.7780 lr=6.89e-06 tok=25.381B dt=11889 wait=217 +08:14:37 step 192560 loss=2.8421 lr=6.87e-06 tok=25.383B dt=11954 wait=217 +08:15:41 step 192580 loss=2.7600 lr=6.85e-06 tok=25.386B dt=12018 wait=217 +08:16:45 step 192600 loss=2.8264 lr=6.83e-06 tok=25.389B dt=12082 wait=218 +08:16:57 pushed ckpt_w188.pt,log_w188.txt +08:19:59 waited 181s for global r=831 +08:20:11 global r=831 load 4.280 -> 4.280 +08:21:18 step 192620 loss=2.8502 lr=6.82e-06 tok=25.391B dt=12355 wait=218 +08:22:21 step 192640 loss=2.7935 lr=6.80e-06 tok=25.394B dt=12418 wait=218 +08:23:26 step 192660 loss=2.7723 lr=6.78e-06 tok=25.397B dt=12483 wait=218 +08:24:30 step 192680 loss=2.7894 lr=6.76e-06 tok=25.399B dt=12547 wait=218 +08:25:35 step 192700 loss=2.8141 lr=6.74e-06 tok=25.402B dt=12612 wait=219 +08:26:39 step 192720 loss=2.8659 lr=6.72e-06 tok=25.404B dt=12676 wait=219 +08:27:43 step 192740 loss=2.8330 lr=6.71e-06 tok=25.407B dt=12740 wait=219 +08:28:27 pushed ckpt_w188.pt,log_w188.txt +08:32:03 step 192760 loss=2.8515 lr=6.69e-06 tok=25.410B dt=13000 wait=219 +08:32:09 global r=832 load 4.282 -> 4.282 +08:32:41 step 192760 loss=2.8753 lr=6.69e-06 tok=25.411B dt=13038 wait=219 +08:33:45 step 192780 loss=2.8531 lr=6.67e-06 tok=25.414B dt=13102 wait=219 +08:34:50 step 192800 loss=2.7302 lr=6.65e-06 tok=25.416B dt=13167 wait=219 +08:35:54 step 192820 loss=2.7913 lr=6.63e-06 tok=25.419B dt=13231 wait=220 +08:36:58 step 192840 loss=2.7403 lr=6.61e-06 tok=25.421B dt=13295 wait=220 +08:38:03 step 192860 loss=2.8522 lr=6.59e-06 tok=25.424B dt=13360 wait=220 +08:39:06 step 192880 loss=2.7934 lr=6.58e-06 tok=25.427B dt=13423 wait=220 +08:40:11 step 192900 loss=2.7977 lr=6.56e-06 tok=25.429B dt=13488 wait=220 +08:40:22 pushed ckpt_w188.pt,log_w188.txt +08:43:24 waited 181s for global r=833 +08:43:36 global r=833 load 4.282 -> 4.282 +08:44:42 step 192920 loss=2.8374 lr=6.54e-06 tok=25.432B dt=13759 wait=220 +08:45:46 step 192940 loss=2.9080 lr=6.52e-06 tok=25.435B dt=13823 wait=220 +08:46:50 step 192960 loss=2.9304 lr=6.50e-06 tok=25.437B dt=13887 wait=221 +08:47:54 step 192980 loss=2.8132 lr=6.48e-06 tok=25.440B dt=13951 wait=221 +08:48:58 step 193000 loss=2.9265 lr=6.47e-06 tok=25.442B dt=14015 wait=221 +08:50:03 step 193020 loss=3.0517 lr=6.45e-06 tok=25.445B dt=14080 wait=221 +08:51:07 step 193040 loss=2.8717 lr=6.43e-06 tok=25.448B dt=14144 wait=221 +08:51:51 pushed ckpt_w188.pt,log_w188.txt +08:54:52 waited 181s for global r=834 +08:55:01 global r=834 load 4.283 -> 4.283 +08:55:35 step 193060 loss=2.9625 lr=6.41e-06 tok=25.450B dt=14412 wait=221 +08:56:39 step 193080 loss=2.9536 lr=6.39e-06 tok=25.453B dt=14476 wait=222 +08:57:43 step 193100 loss=2.8805 lr=6.37e-06 tok=25.455B dt=14540 wait=222 +08:58:47 step 193120 loss=3.0857 lr=6.35e-06 tok=25.458B dt=14604 wait=222 +08:59:52 step 193140 loss=3.0628 lr=6.34e-06 tok=25.461B dt=14669 wait=222 +09:00:56 step 193160 loss=3.1749 lr=6.32e-06 tok=25.463B dt=14733 wait=222 +09:02:00 step 193180 loss=3.0917 lr=6.30e-06 tok=25.466B dt=14797 wait=222 +09:03:04 step 193200 loss=2.9050 lr=6.28e-06 tok=25.469B dt=14861 wait=223 +09:03:16 pushed ckpt_w188.pt,log_w188.txt +09:06:18 waited 181s for global r=835 +09:06:34 global r=835 load 4.283 -> 4.283 +09:07:40 step 193220 loss=2.9660 lr=6.26e-06 tok=25.471B dt=15137 wait=223 +09:08:44 step 193240 loss=3.0042 lr=6.24e-06 tok=25.474B dt=15201 wait=223 +09:09:48 step 193260 loss=2.9297 lr=6.23e-06 tok=25.476B dt=15265 wait=223 +09:10:53 step 193280 loss=2.8735 lr=6.21e-06 tok=25.479B dt=15330 wait=223 +09:11:57 step 193300 loss=2.9700 lr=6.19e-06 tok=25.482B dt=15394 wait=223 +09:13:01 step 193320 loss=3.0485 lr=6.17e-06 tok=25.484B dt=15458 wait=223 +09:14:05 step 193340 loss=2.7448 lr=6.15e-06 tok=25.487B dt=15522 wait=223 +09:14:49 pushed ckpt_w188.pt,log_w188.txt +09:17:52 waited 183s for global r=836 +09:18:02 global r=836 load 4.282 -> 4.282 +09:18:35 step 193360 loss=2.9710 lr=6.13e-06 tok=25.490B dt=15792 wait=224 +09:19:40 step 193380 loss=2.8655 lr=6.11e-06 tok=25.492B dt=15857 wait=224 +09:20:45 step 193400 loss=2.8963 lr=6.10e-06 tok=25.495B dt=15922 wait=224 +09:21:49 step 193420 loss=3.0347 lr=6.08e-06 tok=25.497B dt=15986 wait=224 +09:22:52 step 193440 loss=2.8985 lr=6.06e-06 tok=25.500B dt=16049 wait=224 +09:23:57 step 193460 loss=2.8942 lr=6.04e-06 tok=25.503B dt=16114 wait=224 +09:25:00 step 193480 loss=3.0089 lr=6.02e-06 tok=25.505B dt=16177 wait=225 +09:26:03 step 193500 loss=2.9148 lr=6.00e-06 tok=25.508B dt=16240 wait=225 +09:26:15 pushed ckpt_w188.pt,log_w188.txt +09:29:59 global r=837 load 4.281 -> 4.281 +09:31:04 step 193520 loss=2.7489 lr=5.99e-06 tok=25.512B dt=16541 wait=225 +09:32:08 step 193540 loss=2.9860 lr=5.97e-06 tok=25.514B dt=16605 wait=225 +09:33:11 step 193560 loss=2.9439 lr=5.95e-06 tok=25.517B dt=16668 wait=225 +09:34:15 step 193580 loss=2.9072 lr=5.93e-06 tok=25.520B dt=16732 wait=226 +09:35:19 step 193600 loss=2.9176 lr=5.91e-06 tok=25.522B dt=16796 wait=226 +09:36:23 step 193620 loss=2.9326 lr=5.89e-06 tok=25.525B dt=16860 wait=226 +09:37:27 step 193640 loss=2.7919 lr=5.87e-06 tok=25.528B dt=16924 wait=226 +09:38:13 pushed ckpt_w188.pt,log_w188.txt +09:41:14 waited 181s for global r=838 +09:41:28 global r=838 load 4.281 -> 4.281 +09:42:03 step 193660 loss=3.0231 lr=5.86e-06 tok=25.530B dt=17200 wait=226 +09:43:07 step 193680 loss=2.7939 lr=5.84e-06 tok=25.533B dt=17264 wait=226 +09:44:12 step 193700 loss=2.8831 lr=5.82e-06 tok=25.535B dt=17329 wait=227 +09:45:17 step 193720 loss=2.9545 lr=5.80e-06 tok=25.538B dt=17394 wait=227 +09:46:22 step 193740 loss=2.9015 lr=5.78e-06 tok=25.541B dt=17459 wait=227 +09:47:26 step 193760 loss=2.9861 lr=5.76e-06 tok=25.543B dt=17523 wait=227 +09:48:31 step 193780 loss=2.8653 lr=5.74e-06 tok=25.546B dt=17588 wait=227 +09:49:35 step 193800 loss=2.9796 lr=5.73e-06 tok=25.549B dt=17652 wait=227 +09:49:47 pushed ckpt_w188.pt,log_w188.txt +09:52:49 waited 182s for global r=839 +09:52:58 global r=839 load 4.281 -> 4.281 +09:54:04 step 193820 loss=2.9463 lr=5.71e-06 tok=25.551B dt=17921 wait=227 +09:55:08 step 193840 loss=2.9493 lr=5.69e-06 tok=25.554B dt=17985 wait=228 +09:56:12 step 193860 loss=2.8970 lr=5.67e-06 tok=25.556B dt=18049 wait=228 +09:57:16 step 193880 loss=2.9479 lr=5.65e-06 tok=25.559B dt=18113 wait=228 +09:58:21 step 193900 loss=2.8912 lr=5.63e-06 tok=25.562B dt=18178 wait=228 +09:59:24 step 193920 loss=2.9914 lr=5.62e-06 tok=25.564B dt=18241 wait=228 +10:00:29 step 193940 loss=2.8431 lr=5.60e-06 tok=25.567B dt=18306 wait=228 +10:01:13 pushed ckpt_w188.pt,log_w188.txt +10:04:14 waited 181s for global r=840 +10:04:24 global r=840 load 4.283 -> 4.283 +10:04:57 step 193960 loss=2.9428 lr=5.58e-06 tok=25.570B dt=18574 wait=229 +10:06:01 step 193980 loss=2.9754 lr=5.56e-06 tok=25.572B dt=18638 wait=229 +10:07:05 step 194000 loss=2.9980 lr=5.54e-06 tok=25.575B dt=18702 wait=229 +10:08:09 step 194020 loss=2.8814 lr=5.52e-06 tok=25.577B dt=18766 wait=229 +10:09:14 step 194040 loss=2.9541 lr=5.50e-06 tok=25.580B dt=18831 wait=229 +10:10:18 step 194060 loss=2.9183 lr=5.49e-06 tok=25.583B dt=18895 wait=229 +10:11:23 step 194080 loss=2.7194 lr=5.47e-06 tok=25.585B dt=18960 wait=229 +10:12:26 step 194100 loss=2.9593 lr=5.45e-06 tok=25.588B dt=19023 wait=230 +10:12:40 pushed ckpt_w188.pt,log_w188.txt +10:16:24 global r=841 load 4.282 -> 4.282 +10:17:28 step 194120 loss=2.8847 lr=5.43e-06 tok=25.592B dt=19325 wait=230 +10:18:32 step 194140 loss=2.8924 lr=5.41e-06 tok=25.594B dt=19389 wait=230 +10:19:35 step 194160 loss=2.9369 lr=5.39e-06 tok=25.597B dt=19452 wait=230 +10:20:39 step 194180 loss=2.8214 lr=5.38e-06 tok=25.600B dt=19516 wait=230 +10:21:42 step 194200 loss=2.9095 lr=5.36e-06 tok=25.602B dt=19579 wait=231 +10:22:46 step 194220 loss=2.9110 lr=5.34e-06 tok=25.605B dt=19643 wait=231 +10:23:50 step 194240 loss=2.7509 lr=5.32e-06 tok=25.608B dt=19707 wait=231 +10:24:35 pushed ckpt_w188.pt,log_w188.txt +10:27:36 waited 181s for global r=842 +10:27:50 global r=842 load 4.282 -> 4.282 +10:28:23 step 194260 loss=2.9366 lr=5.30e-06 tok=25.610B dt=19980 wait=231 +10:29:27 step 194280 loss=2.8764 lr=5.28e-06 tok=25.613B dt=20044 wait=231 +10:30:30 step 194300 loss=2.7169 lr=5.26e-06 tok=25.615B dt=20107 wait=231 +10:31:34 step 194320 loss=2.8750 lr=5.25e-06 tok=25.618B dt=20171 wait=231 +10:32:38 step 194340 loss=2.8751 lr=5.23e-06 tok=25.621B dt=20235 wait=232 +10:33:42 step 194360 loss=2.9673 lr=5.21e-06 tok=25.623B dt=20299 wait=232 +10:34:45 step 194380 loss=2.8699 lr=5.19e-06 tok=25.626B dt=20362 wait=232 +10:35:48 step 194400 loss=2.7187 lr=5.17e-06 tok=25.629B dt=20425 wait=232 +10:36:00 pushed ckpt_w188.pt,log_w188.txt +10:39:01 waited 181s for global r=843 +10:39:11 global r=843 load 4.282 -> 4.282 +10:40:17 step 194420 loss=2.7692 lr=5.15e-06 tok=25.631B dt=20694 wait=232 +10:41:21 step 194440 loss=2.8703 lr=5.14e-06 tok=25.634B dt=20758 wait=232 +10:42:27 step 194460 loss=2.5308 lr=5.12e-06 tok=25.636B dt=20824 wait=233 +10:43:31 step 194480 loss=2.8924 lr=5.10e-06 tok=25.639B dt=20888 wait=233 +10:44:34 step 194500 loss=2.8675 lr=5.08e-06 tok=25.642B dt=20951 wait=233 +10:45:39 step 194520 loss=2.9214 lr=5.06e-06 tok=25.644B dt=21016 wait=233 +10:46:43 step 194540 loss=2.7861 lr=5.04e-06 tok=25.647B dt=21080 wait=233 +10:47:28 pushed ckpt_w188.pt,log_w188.txt +10:51:03 step 194560 loss=2.8940 lr=5.02e-06 tok=25.649B dt=21340 wait=233 +10:51:13 global r=844 load 4.283 -> 4.283 +10:51:44 step 194560 loss=2.8635 lr=5.02e-06 tok=25.651B dt=21381 wait=233 +10:52:48 step 194580 loss=2.8309 lr=5.01e-06 tok=25.653B dt=21445 wait=234 +10:53:52 step 194600 loss=2.7634 lr=4.99e-06 tok=25.656B dt=21509 wait=234 +10:54:55 step 194620 loss=2.9344 lr=4.97e-06 tok=25.659B dt=21572 wait=234 +10:55:59 step 194640 loss=2.6883 lr=4.95e-06 tok=25.661B dt=21636 wait=234 +10:57:02 step 194660 loss=2.8543 lr=4.93e-06 tok=25.664B dt=21699 wait=234 +10:58:06 step 194680 loss=2.8086 lr=4.91e-06 tok=25.667B dt=21763 wait=234 +10:59:11 step 194700 loss=2.7407 lr=4.90e-06 tok=25.669B dt=21828 wait=235 +10:59:23 pushed ckpt_w188.pt,log_w188.txt +11:02:25 waited 182s for global r=845 +11:02:34 global r=845 load 4.282 -> 4.282 +11:03:40 step 194720 loss=2.7659 lr=4.88e-06 tok=25.672B dt=22097 wait=235 +11:04:45 step 194740 loss=2.7136 lr=4.86e-06 tok=25.674B dt=22162 wait=235 +11:05:49 step 194760 loss=2.7305 lr=4.84e-06 tok=25.677B dt=22226 wait=235 +11:06:53 step 194780 loss=2.8056 lr=4.82e-06 tok=25.680B dt=22290 wait=235 +11:07:57 step 194800 loss=2.7701 lr=4.80e-06 tok=25.682B dt=22354 wait=235 +11:09:01 step 194820 loss=2.8643 lr=4.78e-06 tok=25.685B dt=22418 wait=235 +11:10:05 step 194840 loss=2.7732 lr=4.77e-06 tok=25.687B dt=22482 wait=235 +11:10:51 pushed ckpt_w188.pt,log_w188.txt +11:13:52 waited 181s for global r=846 +11:14:03 global r=846 load 4.282 -> 4.282 +11:14:36 step 194860 loss=2.9326 lr=4.75e-06 tok=25.690B dt=22753 wait=236 +11:15:42 step 194880 loss=2.8155 lr=4.73e-06 tok=25.693B dt=22819 wait=236 +11:16:45 step 194900 loss=2.8037 lr=4.71e-06 tok=25.695B dt=22882 wait=236 +11:17:50 step 194920 loss=2.7263 lr=4.69e-06 tok=25.698B dt=22947 wait=236 +11:18:53 step 194940 loss=2.7544 lr=4.67e-06 tok=25.701B dt=23010 wait=236 +11:19:56 step 194960 loss=2.9035 lr=4.66e-06 tok=25.703B dt=23073 wait=236 +11:20:59 step 194980 loss=2.7625 lr=4.64e-06 tok=25.706B dt=23136 wait=237 +11:22:03 step 195000 loss=2.7862 lr=4.62e-06 tok=25.708B dt=23200 wait=237 +11:22:16 pushed ckpt_w188.pt,log_w188.txt +11:25:17 waited 181s for global r=847 +11:25:27 global r=847 load 4.282 -> 4.282 +11:26:31 step 195020 loss=2.7542 lr=4.60e-06 tok=25.711B dt=23468 wait=237 +11:27:34 step 195040 loss=2.8696 lr=4.58e-06 tok=25.714B dt=23531 wait=237 +11:28:38 step 195060 loss=2.8295 lr=4.56e-06 tok=25.716B dt=23595 wait=237 +11:29:41 step 195080 loss=2.7574 lr=4.54e-06 tok=25.719B dt=23658 wait=237 +11:30:44 step 195100 loss=2.6518 lr=4.53e-06 tok=25.722B dt=23721 wait=237 +11:31:47 step 195120 loss=2.8290 lr=4.51e-06 tok=25.724B dt=23785 wait=238 +11:32:51 step 195140 loss=2.7243 lr=4.49e-06 tok=25.727B dt=23848 wait=238 +11:33:38 pushed ckpt_w188.pt,log_w188.txt +11:36:40 waited 182s for global r=848 +11:36:52 global r=848 load 4.283 -> 4.283 +11:37:25 step 195160 loss=2.9269 lr=4.47e-06 tok=25.729B dt=24122 wait=238 +11:38:30 step 195180 loss=2.6885 lr=4.45e-06 tok=25.732B dt=24187 wait=238 +11:39:34 step 195200 loss=2.7653 lr=4.43e-06 tok=25.735B dt=24251 wait=238 +11:40:37 step 195220 loss=2.8999 lr=4.42e-06 tok=25.737B dt=24314 wait=238 +11:41:41 step 195240 loss=2.7672 lr=4.40e-06 tok=25.740B dt=24379 wait=239 +11:42:46 step 195260 loss=2.5038 lr=4.38e-06 tok=25.743B dt=24443 wait=239 +11:43:50 step 195280 loss=2.7250 lr=4.36e-06 tok=25.745B dt=24507 wait=239 +11:44:54 step 195300 loss=2.9150 lr=4.34e-06 tok=25.748B dt=24571 wait=239 +11:45:06 pushed ckpt_w188.pt,log_w188.txt +11:48:08 waited 182s for global r=849 +11:48:18 global r=849 load 4.282 -> 4.282 +11:49:24 step 195320 loss=2.7723 lr=4.32e-06 tok=25.750B dt=24841 wait=239 +11:50:28 step 195340 loss=2.8040 lr=4.30e-06 tok=25.753B dt=24905 wait=239 +11:51:31 step 195360 loss=2.7700 lr=4.29e-06 tok=25.756B dt=24968 wait=240 +11:52:34 step 195380 loss=2.7969 lr=4.27e-06 tok=25.758B dt=25031 wait=240 +11:53:37 step 195400 loss=2.7900 lr=4.25e-06 tok=25.761B dt=25094 wait=240 +11:54:41 step 195420 loss=2.8321 lr=4.23e-06 tok=25.764B dt=25158 wait=240 +11:55:44 step 195440 loss=2.7726 lr=4.21e-06 tok=25.766B dt=25221 wait=240 +11:56:28 pushed ckpt_w188.pt,log_w188.txt +11:59:29 waited 181s for global r=850 +11:59:40 global r=850 load 4.281 -> 4.281 +12:00:13 step 195460 loss=2.8647 lr=4.19e-06 tok=25.769B dt=25490 wait=240 +12:01:17 step 195480 loss=2.8340 lr=4.18e-06 tok=25.771B dt=25554 wait=240 +12:02:20 step 195500 loss=2.8354 lr=4.16e-06 tok=25.774B dt=25617 wait=241 +12:03:23 step 195520 loss=2.8555 lr=4.14e-06 tok=25.777B dt=25680 wait=241 +12:04:26 step 195540 loss=2.7465 lr=4.12e-06 tok=25.779B dt=25743 wait=241 +12:05:29 step 195560 loss=2.8483 lr=4.10e-06 tok=25.782B dt=25806 wait=241 +12:06:32 step 195580 loss=2.7786 lr=4.08e-06 tok=25.784B dt=25869 wait=241 +12:07:36 step 195600 loss=2.6436 lr=4.06e-06 tok=25.787B dt=25933 wait=241 +12:07:50 pushed ckpt_w188.pt,log_w188.txt +12:10:52 waited 181s for global r=851 +12:11:04 global r=851 load 4.281 -> 4.281 +12:12:09 step 195620 loss=2.7447 lr=4.05e-06 tok=25.790B dt=26206 wait=242 +12:13:12 step 195640 loss=2.8213 lr=4.03e-06 tok=25.792B dt=26269 wait=242 +12:14:15 step 195660 loss=2.7697 lr=4.01e-06 tok=25.795B dt=26332 wait=242 +12:15:18 step 195680 loss=2.6663 lr=3.99e-06 tok=25.798B dt=26395 wait=242 +12:16:21 step 195700 loss=2.7931 lr=3.97e-06 tok=25.800B dt=26458 wait=242 +12:17:25 step 195720 loss=2.8132 lr=3.95e-06 tok=25.803B dt=26522 wait=242 +12:18:28 step 195740 loss=2.7216 lr=3.93e-06 tok=25.805B dt=26585 wait=242 +12:19:15 pushed ckpt_w188.pt,log_w188.txt +12:22:16 waited 182s for global r=852 +12:22:27 global r=852 load 4.281 -> 4.281 +12:23:00 step 195760 loss=2.7798 lr=3.92e-06 tok=25.808B dt=26857 wait=242 +12:24:04 step 195780 loss=2.6937 lr=3.90e-06 tok=25.811B dt=26921 wait=243 +12:25:07 step 195800 loss=2.7582 lr=3.88e-06 tok=25.813B dt=26984 wait=243 +12:26:11 step 195820 loss=2.8173 lr=3.86e-06 tok=25.816B dt=27048 wait=243 +12:27:15 step 195840 loss=2.7790 lr=3.84e-06 tok=25.819B dt=27112 wait=243 +12:28:19 step 195860 loss=2.8642 lr=3.82e-06 tok=25.821B dt=27176 wait=243 +12:29:23 step 195880 loss=2.8492 lr=3.81e-06 tok=25.824B dt=27240 wait=243 +12:30:28 step 195900 loss=2.8251 lr=3.79e-06 tok=25.826B dt=27305 wait=244 +12:30:42 pushed ckpt_w188.pt,log_w188.txt +12:33:44 waited 181s for global r=853 +12:33:54 global r=853 load 4.282 -> 4.282 +12:35:00 step 195920 loss=2.8614 lr=3.77e-06 tok=25.829B dt=27577 wait=244 +12:36:04 step 195940 loss=2.8097 lr=3.75e-06 tok=25.832B dt=27641 wait=244 +12:37:09 step 195960 loss=2.8060 lr=3.73e-06 tok=25.834B dt=27706 wait=244 +12:38:13 step 195980 loss=2.7574 lr=3.71e-06 tok=25.837B dt=27770 wait=244 +12:39:17 step 196000 loss=2.8411 lr=3.69e-06 tok=25.840B dt=27834 wait=244 +12:40:21 step 196020 loss=2.8127 lr=3.68e-06 tok=25.842B dt=27898 wait=244 +12:41:24 step 196040 loss=2.7828 lr=3.66e-06 tok=25.845B dt=27961 wait=245 +12:42:11 pushed ckpt_w188.pt,log_w188.txt +12:45:47 step 196060 loss=2.9076 lr=3.64e-06 tok=25.847B dt=28224 wait=245 +12:45:55 global r=854 load 4.282 -> 4.282 +12:46:27 step 196060 loss=2.8500 lr=3.64e-06 tok=25.849B dt=28264 wait=245 +12:47:30 step 196080 loss=2.7782 lr=3.62e-06 tok=25.851B dt=28327 wait=245 +12:48:34 step 196100 loss=2.8166 lr=3.60e-06 tok=25.854B dt=28391 wait=245 +12:49:38 step 196120 loss=2.8894 lr=3.58e-06 tok=25.857B dt=28455 wait=245 +12:50:42 step 196140 loss=2.7604 lr=3.57e-06 tok=25.859B dt=28519 wait=245 +12:51:45 step 196160 loss=2.8257 lr=3.55e-06 tok=25.862B dt=28582 wait=246 +12:52:48 step 196180 loss=2.7898 lr=3.53e-06 tok=25.864B dt=28645 wait=246 +12:53:53 step 196200 loss=2.7156 lr=3.51e-06 tok=25.867B dt=28710 wait=246 +12:54:05 pushed ckpt_w188.pt,log_w188.txt +12:58:12 step 196220 loss=2.8809 lr=3.49e-06 tok=25.870B dt=28969 wait=246 +12:59:16 step 196240 loss=2.7211 lr=3.47e-06 tok=25.872B dt=29033 wait=246 +13:00:20 step 196260 loss=2.8698 lr=3.45e-06 tok=25.875B dt=29097 wait=246 +13:01:24 step 196280 loss=2.7502 lr=3.44e-06 tok=25.878B dt=29161 wait=247 +13:02:28 step 196300 loss=2.8237 lr=3.42e-06 tok=25.880B dt=29225 wait=247 +13:03:32 step 196320 loss=2.8196 lr=3.40e-06 tok=25.883B dt=29289 wait=247 +13:04:36 step 196340 loss=2.7913 lr=3.38e-06 tok=25.885B dt=29353 wait=247 +13:05:21 pushed ckpt_w188.pt,log_w188.txt +13:08:56 step 196360 loss=2.8988 lr=3.36e-06 tok=25.888B dt=29613 wait=247 +13:10:01 step 196380 loss=2.9355 lr=3.34e-06 tok=25.891B dt=29678 wait=247 +13:11:05 step 196400 loss=2.9301 lr=3.33e-06 tok=25.893B dt=29742 wait=247 +13:12:10 step 196420 loss=2.8982 lr=3.31e-06 tok=25.896B dt=29807 wait=248 +13:12:18 global r=855 load 4.281 -> 4.281 +13:13:23 step 196220 loss=2.8922 lr=3.49e-06 tok=25.899B dt=29880 wait=248 +13:14:27 step 196240 loss=2.9440 lr=3.47e-06 tok=25.901B dt=29944 wait=248 +13:15:31 step 196260 loss=2.9390 lr=3.45e-06 tok=25.904B dt=30008 wait=248 +13:16:35 step 196280 loss=2.8998 lr=3.44e-06 tok=25.906B dt=30072 wait=248 +13:17:40 step 196300 loss=2.9752 lr=3.42e-06 tok=25.909B dt=30137 wait=248 +13:18:44 step 196320 loss=2.8199 lr=3.40e-06 tok=25.912B dt=30201 wait=249 +13:19:48 step 196340 loss=2.8943 lr=3.38e-06 tok=25.914B dt=30265 wait=249 +13:20:45 pushed ckpt_w188.pt,log_w188.txt +13:22:03 waited 76s for global r=856 +13:22:11 global r=856 load 4.281 -> 4.281 +13:22:45 step 196360 loss=3.0058 lr=3.36e-06 tok=25.917B dt=30442 wait=249 +13:23:48 step 196380 loss=2.9489 lr=3.34e-06 tok=25.919B dt=30505 wait=249 +13:24:53 step 196400 loss=2.7556 lr=3.33e-06 tok=25.922B dt=30570 wait=249 +13:25:56 step 196420 loss=3.2199 lr=3.31e-06 tok=25.925B dt=30633 wait=249 +13:27:01 step 196440 loss=2.8605 lr=3.29e-06 tok=25.927B dt=30698 wait=250 +13:28:04 step 196460 loss=3.0118 lr=3.27e-06 tok=25.930B dt=30761 wait=250 +13:29:08 step 196480 loss=2.9083 lr=3.25e-06 tok=25.933B dt=30825 wait=250 +13:30:12 step 196500 loss=2.8919 lr=3.23e-06 tok=25.935B dt=30889 wait=250 +13:30:26 pushed ckpt_w188.pt,log_w188.txt +13:31:42 waited 76s for global r=857 +13:31:50 global r=857 load 4.281 -> 4.281 +13:32:56 step 196520 loss=2.7784 lr=3.21e-06 tok=25.938B dt=31053 wait=250 +13:33:59 step 196540 loss=2.8278 lr=3.20e-06 tok=25.940B dt=31116 wait=250 +13:35:03 step 196560 loss=2.8430 lr=3.18e-06 tok=25.943B dt=31180 wait=250 +13:36:06 step 196580 loss=2.8860 lr=3.16e-06 tok=25.946B dt=31243 wait=251 +13:37:10 step 196600 loss=2.8629 lr=3.14e-06 tok=25.948B dt=31307 wait=251 +13:38:15 step 196620 loss=2.9849 lr=3.12e-06 tok=25.951B dt=31372 wait=251 +13:39:19 step 196640 loss=2.8145 lr=3.10e-06 tok=25.954B dt=31436 wait=251 +13:40:05 pushed ckpt_w188.pt,log_w188.txt +13:41:27 waited 83s for global r=858 +13:41:37 global r=858 load 4.281 -> 4.280 +13:42:10 step 196660 loss=2.7994 lr=3.09e-06 tok=25.956B dt=31607 wait=251 +13:43:14 step 196680 loss=2.8266 lr=3.07e-06 tok=25.959B dt=31671 wait=251 +13:44:18 step 196700 loss=2.8292 lr=3.05e-06 tok=25.961B dt=31735 wait=252 +13:45:21 step 196720 loss=3.0069 lr=3.03e-06 tok=25.964B dt=31798 wait=252 +13:46:25 step 196740 loss=2.7797 lr=3.01e-06 tok=25.967B dt=31862 wait=252 +13:47:29 step 196760 loss=2.9706 lr=2.99e-06 tok=25.969B dt=31926 wait=252 +13:48:33 step 196780 loss=2.8328 lr=2.97e-06 tok=25.972B dt=31990 wait=252 +13:49:37 step 196800 loss=2.8662 lr=2.96e-06 tok=25.975B dt=32054 wait=252 +13:49:49 pushed ckpt_w188.pt,log_w188.txt +13:51:21 waited 92s for global r=859 +13:51:31 global r=859 load 4.280 -> 4.281 +13:52:36 step 196820 loss=2.8837 lr=2.94e-06 tok=25.977B dt=32233 wait=252 +13:53:41 step 196840 loss=2.8504 lr=2.92e-06 tok=25.980B dt=32298 wait=253 +13:54:44 step 196860 loss=2.9108 lr=2.90e-06 tok=25.982B dt=32361 wait=253 +13:55:48 step 196880 loss=2.8289 lr=2.88e-06 tok=25.985B dt=32425 wait=253 +13:56:54 step 196900 loss=2.8902 lr=2.86e-06 tok=25.988B dt=32491 wait=253 +13:57:58 step 196920 loss=3.0350 lr=2.85e-06 tok=25.990B dt=32555 wait=253 +13:59:03 step 196940 loss=2.9712 lr=2.83e-06 tok=25.993B dt=32620 wait=253 +13:59:51 pushed ckpt_w188.pt,log_w188.txt +14:01:17 waited 86s for global r=860 +14:01:25 global r=860 load 4.280 -> 4.281 +14:01:59 step 196960 loss=3.0559 lr=2.81e-06 tok=25.996B dt=32796 wait=254 +14:03:03 step 196980 loss=2.9376 lr=2.79e-06 tok=25.998B dt=32860 wait=254 +14:04:07 step 197000 loss=2.9572 lr=2.77e-06 tok=26.001B dt=32924 wait=254 +14:05:11 step 197020 loss=3.1116 lr=2.75e-06 tok=26.003B dt=32988 wait=254 +14:06:17 step 197040 loss=2.9082 lr=2.73e-06 tok=26.006B dt=33054 wait=254 +14:07:22 step 197060 loss=2.9902 lr=2.72e-06 tok=26.009B dt=33119 wait=254 +14:08:26 step 197080 loss=3.0827 lr=2.70e-06 tok=26.011B dt=33183 wait=254 +14:09:32 step 197100 loss=3.0575 lr=2.68e-06 tok=26.014B dt=33249 wait=255 +14:09:47 pushed ckpt_w188.pt,log_w188.txt +14:11:03 waited 76s for global r=861 +14:11:11 global r=861 load 4.280 -> 4.280 +14:12:17 step 197120 loss=2.9296 lr=2.66e-06 tok=26.016B dt=33414 wait=255 +14:13:21 step 197140 loss=3.0260 lr=2.64e-06 tok=26.019B dt=33478 wait=255 +14:14:26 step 197160 loss=3.0539 lr=2.62e-06 tok=26.022B dt=33543 wait=255 +14:15:31 step 197180 loss=2.9560 lr=2.61e-06 tok=26.024B dt=33608 wait=255 +14:16:35 step 197200 loss=3.0659 lr=2.59e-06 tok=26.027B dt=33672 wait=255 +14:17:39 step 197220 loss=3.1188 lr=2.57e-06 tok=26.030B dt=33736 wait=256 +14:18:44 step 197240 loss=2.9793 lr=2.55e-06 tok=26.032B dt=33801 wait=256 +14:19:38 pushed ckpt_w188.pt,log_w188.txt +14:20:44 waited 66s for global r=862 +14:20:52 global r=862 load 4.280 -> 4.281 +14:21:27 step 197260 loss=3.0527 lr=2.53e-06 tok=26.035B dt=33964 wait=256 +14:22:31 step 197280 loss=2.9003 lr=2.51e-06 tok=26.037B dt=34028 wait=256 +14:23:36 step 197300 loss=3.0476 lr=2.49e-06 tok=26.040B dt=34093 wait=256 +14:24:40 step 197320 loss=3.0509 lr=2.48e-06 tok=26.043B dt=34157 wait=256 +14:25:44 step 197340 loss=2.9833 lr=2.46e-06 tok=26.045B dt=34221 wait=256 +14:26:48 step 197360 loss=3.0018 lr=2.44e-06 tok=26.048B dt=34285 wait=257 +14:27:51 step 197380 loss=2.9573 lr=2.42e-06 tok=26.051B dt=34348 wait=257 +14:28:56 step 197400 loss=2.9044 lr=2.40e-06 tok=26.053B dt=34413 wait=257 +14:29:09 pushed ckpt_w188.pt,log_w188.txt +14:30:25 waited 76s for global r=863 +14:30:34 global r=863 load 4.280 -> 4.280 +14:31:41 step 197420 loss=2.8496 lr=2.38e-06 tok=26.056B dt=34578 wait=257 +14:32:45 step 197440 loss=2.9026 lr=2.37e-06 tok=26.058B dt=34642 wait=257 +14:33:49 step 197460 loss=2.9224 lr=2.35e-06 tok=26.061B dt=34706 wait=257 +14:34:52 step 197480 loss=2.8670 lr=2.33e-06 tok=26.064B dt=34769 wait=257 +14:35:56 step 197500 loss=2.8344 lr=2.31e-06 tok=26.066B dt=34833 wait=258 +14:37:00 step 197520 loss=2.9760 lr=2.29e-06 tok=26.069B dt=34897 wait=258 +14:38:04 step 197540 loss=2.9086 lr=2.27e-06 tok=26.072B dt=34961 wait=258 +14:38:50 pushed ckpt_w188.pt,log_w188.txt +14:40:24 waited 94s for global r=864 +14:40:33 global r=864 load 4.281 -> 4.280 +14:41:07 step 197560 loss=3.0033 lr=2.25e-06 tok=26.074B dt=35144 wait=258 +14:42:12 step 197580 loss=2.8216 lr=2.24e-06 tok=26.077B dt=35209 wait=258 +14:43:17 step 197600 loss=2.8694 lr=2.22e-06 tok=26.079B dt=35274 wait=258 +14:44:21 step 197620 loss=2.9368 lr=2.20e-06 tok=26.082B dt=35338 wait=259 +14:45:27 step 197640 loss=2.8743 lr=2.18e-06 tok=26.085B dt=35404 wait=259 +14:46:31 step 197660 loss=2.9953 lr=2.16e-06 tok=26.087B dt=35468 wait=259 +14:47:36 step 197680 loss=2.9437 lr=2.14e-06 tok=26.090B dt=35533 wait=259 +14:48:41 step 197700 loss=2.8385 lr=2.12e-06 tok=26.093B dt=35598 wait=259 +14:49:03 pushed ckpt_w188.pt,log_w188.txt +14:50:04 waited 61s for global r=865 +14:50:13 global r=865 load 4.281 -> 4.281 +14:51:20 step 197720 loss=2.8397 lr=2.11e-06 tok=26.095B dt=35757 wait=259 +14:52:24 step 197740 loss=2.9786 lr=2.09e-06 tok=26.098B dt=35821 wait=260 +14:53:29 step 197760 loss=2.9407 lr=2.07e-06 tok=26.100B dt=35886 wait=260 +14:54:33 step 197780 loss=2.7893 lr=2.05e-06 tok=26.103B dt=35950 wait=260 +14:55:37 step 197800 loss=2.9512 lr=2.03e-06 tok=26.106B dt=36014 wait=260 +14:56:42 step 197820 loss=3.0495 lr=2.01e-06 tok=26.108B dt=36079 wait=260 +14:57:46 step 197840 loss=2.8661 lr=2.00e-06 tok=26.111B dt=36143 wait=260 +14:58:33 pushed ckpt_w188.pt,log_w188.txt +14:59:49 waited 76s for global r=866 +14:59:58 global r=866 load 4.281 -> 4.280 +15:00:32 step 197860 loss=2.8288 lr=1.98e-06 tok=26.113B dt=36309 wait=261 +15:01:37 step 197880 loss=2.9118 lr=1.96e-06 tok=26.116B dt=36374 wait=261 +15:02:41 step 197900 loss=2.8844 lr=1.94e-06 tok=26.119B dt=36438 wait=261 +15:03:45 step 197920 loss=2.9183 lr=1.92e-06 tok=26.121B dt=36502 wait=261 +15:04:50 step 197940 loss=2.9640 lr=1.90e-06 tok=26.124B dt=36567 wait=261 +15:05:54 step 197960 loss=2.9554 lr=1.88e-06 tok=26.127B dt=36631 wait=261 +15:06:59 step 197980 loss=2.9605 lr=1.87e-06 tok=26.129B dt=36696 wait=261 +15:08:03 step 198000 loss=3.0741 lr=1.85e-06 tok=26.132B dt=36760 wait=262 +15:08:21 pushed ckpt_w188.pt,log_w188.txt +15:09:23 waited 62s for global r=867 +15:09:32 global r=867 load 4.281 -> 4.280 +15:10:38 step 198020 loss=2.9500 lr=1.83e-06 tok=26.134B dt=36916 wait=262 +15:11:43 step 198040 loss=2.9853 lr=1.81e-06 tok=26.137B dt=36980 wait=262 +15:12:47 step 198060 loss=3.0603 lr=1.79e-06 tok=26.140B dt=37044 wait=262 +15:13:51 step 198080 loss=3.0053 lr=1.77e-06 tok=26.142B dt=37108 wait=262 +15:14:55 step 198100 loss=2.9739 lr=1.76e-06 tok=26.145B dt=37172 wait=262 +15:15:59 step 198120 loss=2.9670 lr=1.74e-06 tok=26.148B dt=37236 wait=262 +15:17:03 step 198140 loss=2.9700 lr=1.72e-06 tok=26.150B dt=37300 wait=263 +15:17:51 pushed ckpt_w188.pt,log_w188.txt +15:19:07 waited 76s for global r=868 +15:19:18 global r=868 load 4.281 -> 4.280 +15:19:52 step 198160 loss=3.0143 lr=1.70e-06 tok=26.153B dt=37469 wait=263 +15:20:55 step 198180 loss=2.9671 lr=1.68e-06 tok=26.155B dt=37532 wait=263 +15:21:59 step 198200 loss=2.8724 lr=1.66e-06 tok=26.158B dt=37596 wait=263 +15:23:03 step 198220 loss=2.9196 lr=1.64e-06 tok=26.161B dt=37660 wait=263 +15:24:07 step 198240 loss=2.9635 lr=1.63e-06 tok=26.163B dt=37724 wait=263 +15:25:11 step 198260 loss=3.0810 lr=1.61e-06 tok=26.166B dt=37788 wait=263 +15:26:14 step 198280 loss=3.0459 lr=1.59e-06 tok=26.169B dt=37851 wait=264 +15:27:18 step 198300 loss=3.1100 lr=1.57e-06 tok=26.171B dt=37915 wait=264 +15:27:35 pushed ckpt_w188.pt,log_w188.txt +15:28:36 waited 61s for global r=869 +15:28:43 global r=869 load 4.280 -> 4.281 +15:29:48 step 198320 loss=3.0323 lr=1.55e-06 tok=26.174B dt=38065 wait=264 +15:30:52 step 198340 loss=3.0095 lr=1.53e-06 tok=26.176B dt=38129 wait=264 +15:31:56 step 198360 loss=2.9668 lr=1.52e-06 tok=26.179B dt=38193 wait=264 +15:32:59 step 198380 loss=2.9380 lr=1.50e-06 tok=26.182B dt=38256 wait=265 +15:34:03 step 198400 loss=2.9129 lr=1.48e-06 tok=26.184B dt=38320 wait=265 +15:35:07 step 198420 loss=3.0479 lr=1.46e-06 tok=26.187B dt=38384 wait=265 +15:36:11 step 198440 loss=2.8482 lr=1.44e-06 tok=26.189B dt=38448 wait=265 +15:37:03 pushed ckpt_w188.pt,log_w188.txt +15:38:19 waited 76s for global r=870 +15:38:28 global r=870 load 4.280 -> 4.281 +15:39:02 step 198460 loss=3.0126 lr=1.42e-06 tok=26.192B dt=38619 wait=265 +15:40:05 step 198480 loss=2.9657 lr=1.40e-06 tok=26.195B dt=38682 wait=265 +15:41:08 step 198500 loss=2.8878 lr=1.39e-06 tok=26.197B dt=38745 wait=265 +15:42:13 step 198520 loss=2.9203 lr=1.37e-06 tok=26.200B dt=38810 wait=266 +15:43:17 step 198540 loss=2.9298 lr=1.35e-06 tok=26.203B dt=38874 wait=266 +15:44:20 step 198560 loss=2.9738 lr=1.33e-06 tok=26.205B dt=38937 wait=266 +15:45:23 step 198580 loss=2.9496 lr=1.31e-06 tok=26.208B dt=39000 wait=266 +15:46:27 step 198600 loss=2.8481 lr=1.29e-06 tok=26.210B dt=39064 wait=266 +15:46:45 pushed ckpt_w188.pt,log_w188.txt +15:48:02 waited 78s for global r=871 +15:48:15 global r=871 load 4.280 -> 4.281 +15:49:20 step 198620 loss=2.8770 lr=1.28e-06 tok=26.213B dt=39237 wait=266 +15:50:23 step 198640 loss=2.8846 lr=1.26e-06 tok=26.216B dt=39300 wait=266 +15:51:27 step 198660 loss=2.9465 lr=1.24e-06 tok=26.218B dt=39364 wait=267 +15:52:31 step 198680 loss=2.9161 lr=1.22e-06 tok=26.221B dt=39428 wait=267 +15:53:34 step 198700 loss=2.9831 lr=1.20e-06 tok=26.224B dt=39491 wait=267 +15:54:37 step 198720 loss=2.9875 lr=1.18e-06 tok=26.226B dt=39554 wait=267 +15:55:39 step 198740 loss=2.8966 lr=1.16e-06 tok=26.229B dt=39616 wait=267 +15:56:30 pushed ckpt_w188.pt,log_w188.txt +15:59:33 waited 183s for global r=872 +15:59:44 global r=872 load 4.280 -> 4.281 +16:00:17 step 198760 loss=3.0712 lr=1.15e-06 tok=26.231B dt=39894 wait=267 +16:01:20 step 198780 loss=2.9011 lr=1.13e-06 tok=26.234B dt=39957 wait=267 +16:02:25 step 198800 loss=2.8451 lr=1.11e-06 tok=26.237B dt=40022 wait=268 +16:03:28 step 198820 loss=2.9652 lr=1.09e-06 tok=26.239B dt=40085 wait=268 +16:04:31 step 198840 loss=2.9198 lr=1.07e-06 tok=26.242B dt=40148 wait=268 +16:05:35 step 198860 loss=3.0149 lr=1.05e-06 tok=26.245B dt=40212 wait=268 +16:06:39 step 198880 loss=2.8216 lr=1.04e-06 tok=26.247B dt=40276 wait=268 +16:07:42 step 198900 loss=2.9573 lr=1.02e-06 tok=26.250B dt=40339 wait=268 +16:07:56 pushed ckpt_w188.pt,log_w188.txt +16:11:51 global r=873 load 4.280 -> 4.281 +16:12:55 step 198920 loss=3.0305 lr=9.98e-07 tok=26.254B dt=40652 wait=269 +16:13:59 step 198940 loss=3.0243 lr=9.80e-07 tok=26.256B dt=40716 wait=269 +16:15:08 step 198960 loss=2.9690 lr=9.61e-07 tok=26.259B dt=40785 wait=269 +16:16:12 step 198980 loss=2.9881 lr=9.43e-07 tok=26.262B dt=40849 wait=269 +16:17:17 step 199000 loss=2.9149 lr=9.24e-07 tok=26.264B dt=40914 wait=269 +16:18:21 step 199020 loss=3.1345 lr=9.06e-07 tok=26.267B dt=40978 wait=269 +16:19:25 step 199040 loss=2.9656 lr=8.87e-07 tok=26.269B dt=41042 wait=270 +16:20:09 pushed ckpt_w188.pt,log_w188.txt +16:23:45 step 199060 loss=3.0710 lr=8.69e-07 tok=26.272B dt=41302 wait=270 +16:23:56 global r=874 load 4.280 -> 4.280 +16:24:28 step 199060 loss=3.0614 lr=8.69e-07 tok=26.273B dt=41345 wait=270 +16:25:32 step 199080 loss=2.9104 lr=8.51e-07 tok=26.276B dt=41409 wait=270 +16:26:37 step 199100 loss=2.9968 lr=8.32e-07 tok=26.279B dt=41474 wait=270 +16:27:41 step 199120 loss=3.0401 lr=8.14e-07 tok=26.281B dt=41538 wait=270 +16:28:45 step 199140 loss=2.9392 lr=7.95e-07 tok=26.284B dt=41602 wait=270 +16:29:50 step 199160 loss=2.9898 lr=7.77e-07 tok=26.286B dt=41667 wait=270 +16:29:54 mix source 5 lap 1 +16:30:55 step 199180 loss=2.9104 lr=7.58e-07 tok=26.289B dt=41732 wait=271 +16:31:59 step 199200 loss=3.0094 lr=7.40e-07 tok=26.292B dt=41796 wait=271 +16:32:25 pushed ckpt_w188.pt,log_w188.txt +16:33:41 waited 76s for global r=875 +16:33:51 global r=875 load 4.280 -> 4.281 +16:34:57 step 199220 loss=2.9461 lr=7.21e-07 tok=26.294B dt=41974 wait=271 +16:36:01 step 199240 loss=2.9776 lr=7.03e-07 tok=26.297B dt=42038 wait=271 +16:37:06 step 199260 loss=2.9471 lr=6.84e-07 tok=26.300B dt=42103 wait=271 +16:38:10 step 199280 loss=2.8489 lr=6.66e-07 tok=26.302B dt=42167 wait=271 +16:39:15 step 199300 loss=2.7272 lr=6.47e-07 tok=26.305B dt=42232 wait=272 +16:40:19 step 199320 loss=3.0152 lr=6.29e-07 tok=26.307B dt=42296 wait=272 +16:41:24 step 199340 loss=2.7752 lr=6.10e-07 tok=26.310B dt=42361 wait=272 diff --git a/log_w189.txt b/log_w189.txt new file mode 100644 index 0000000000000000000000000000000000000000..e5a1e6e0fb41b3a1240dbda80e00578669126b11 --- /dev/null +++ b/log_w189.txt @@ -0,0 +1,186 @@ +launch +04:55:01 name=w189 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 136/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +04:55:08 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +04:55:08 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +04:55:17 params 1.08B +04:55:19 resumed step=189600 seen=24.99B round=811 +04:55:19 stream salt=189600 +04:57:03 blocks compiled +04:57:03 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:00:26 step 189601 loss=2.7306 lr=9.60e-06 tok=24.986B dt=203 wait=175 +05:00:42 step 189602 loss=2.8935 lr=9.60e-06 tok=24.987B dt=219 wait=175 +05:01:39 step 189620 loss=3.1064 lr=9.59e-06 tok=24.989B dt=276 wait=175 +05:02:44 step 189640 loss=2.9980 lr=9.57e-06 tok=24.991B dt=340 wait=175 +05:03:47 step 189660 loss=3.0040 lr=9.55e-06 tok=24.994B dt=404 wait=175 +05:04:51 step 189680 loss=3.0649 lr=9.53e-06 tok=24.997B dt=468 wait=175 +05:05:55 step 189700 loss=3.1782 lr=9.51e-06 tok=24.999B dt=532 wait=175 +05:06:59 step 189720 loss=3.0130 lr=9.49e-06 tok=25.002B dt=596 wait=176 +05:08:04 step 189740 loss=3.0132 lr=9.48e-06 tok=25.005B dt=661 wait=176 +05:08:52 pushed ckpt_w189.pt,log_w189.txt +05:08:52 waited 0s for global r=812 +05:09:07 global r=812 load 4.280 -> 4.278 +05:09:43 step 189760 loss=3.1713 lr=9.46e-06 tok=25.007B dt=759 wait=176 +05:10:47 step 189780 loss=3.0453 lr=9.44e-06 tok=25.010B dt=824 wait=176 +05:11:51 step 189800 loss=3.0706 lr=9.42e-06 tok=25.012B dt=887 wait=176 +05:12:55 step 189820 loss=3.0730 lr=9.40e-06 tok=25.015B dt=952 wait=176 +05:13:59 step 189840 loss=3.1094 lr=9.38e-06 tok=25.018B dt=1016 wait=176 +05:15:04 step 189860 loss=3.2108 lr=9.36e-06 tok=25.020B dt=1080 wait=177 +05:16:08 step 189880 loss=3.0493 lr=9.35e-06 tok=25.023B dt=1145 wait=177 +05:17:12 step 189900 loss=3.2332 lr=9.33e-06 tok=25.026B dt=1208 wait=177 +05:17:26 pushed ckpt_w189.pt,log_w189.txt +05:18:42 waited 76s for global r=813 +05:18:52 global r=813 load 4.279 -> 4.279 +05:19:57 step 189920 loss=3.0979 lr=9.31e-06 tok=25.028B dt=1374 wait=177 +05:21:01 step 189940 loss=3.2474 lr=9.29e-06 tok=25.031B dt=1438 wait=177 +05:22:05 step 189960 loss=3.0916 lr=9.27e-06 tok=25.033B dt=1502 wait=177 +05:23:09 step 189980 loss=3.1096 lr=9.25e-06 tok=25.036B dt=1566 wait=177 +05:24:12 step 190000 loss=3.0479 lr=9.24e-06 tok=25.039B dt=1629 wait=178 +05:25:16 step 190020 loss=3.1357 lr=9.22e-06 tok=25.041B dt=1692 wait=178 +05:26:19 step 190040 loss=2.9771 lr=9.20e-06 tok=25.044B dt=1756 wait=178 +05:27:03 pushed ckpt_w189.pt,log_w189.txt +05:28:19 waited 76s for global r=814 +05:28:24 global r=814 load 4.279 -> 4.279 +05:28:58 step 190060 loss=3.1139 lr=9.18e-06 tok=25.047B dt=1914 wait=178 +05:30:01 step 190080 loss=3.0194 lr=9.16e-06 tok=25.049B dt=1978 wait=178 +05:31:05 step 190100 loss=2.9548 lr=9.14e-06 tok=25.052B dt=2042 wait=178 +05:32:09 step 190120 loss=3.0935 lr=9.12e-06 tok=25.054B dt=2105 wait=178 +05:33:13 step 190140 loss=2.9468 lr=9.11e-06 tok=25.057B dt=2170 wait=179 +05:34:17 step 190160 loss=3.1340 lr=9.09e-06 tok=25.060B dt=2234 wait=179 +05:35:21 step 190180 loss=3.0231 lr=9.07e-06 tok=25.062B dt=2298 wait=179 +05:36:25 step 190200 loss=3.0688 lr=9.05e-06 tok=25.065B dt=2361 wait=179 +05:36:38 pushed ckpt_w189.pt,log_w189.txt +05:37:54 waited 76s for global r=815 +05:37:59 global r=815 load 4.279 -> 4.279 +05:39:05 step 190220 loss=3.2024 lr=9.03e-06 tok=25.068B dt=2522 wait=179 +05:40:09 step 190240 loss=3.1190 lr=9.01e-06 tok=25.070B dt=2586 wait=179 +05:41:13 step 190260 loss=3.0686 lr=9.00e-06 tok=25.073B dt=2649 wait=179 +05:42:16 step 190280 loss=3.0526 lr=8.98e-06 tok=25.075B dt=2713 wait=180 +05:43:20 step 190300 loss=3.1636 lr=8.96e-06 tok=25.078B dt=2777 wait=180 +05:44:25 step 190320 loss=3.0732 lr=8.94e-06 tok=25.081B dt=2841 wait=180 +05:45:29 step 190340 loss=2.6719 lr=8.92e-06 tok=25.083B dt=2906 wait=180 +05:46:13 pushed ckpt_w189.pt,log_w189.txt +05:47:28 waited 76s for global r=816 +05:47:35 global r=816 load 4.281 -> 4.280 +05:48:08 step 190360 loss=3.1475 lr=8.90e-06 tok=25.086B dt=3065 wait=180 +05:49:13 step 190380 loss=3.0455 lr=8.88e-06 tok=25.088B dt=3129 wait=180 +05:50:17 step 190400 loss=3.0409 lr=8.87e-06 tok=25.091B dt=3194 wait=180 +05:51:22 step 190420 loss=3.1253 lr=8.85e-06 tok=25.094B dt=3259 wait=181 +05:52:26 step 190440 loss=3.1097 lr=8.83e-06 tok=25.096B dt=3323 wait=181 +05:53:30 step 190460 loss=3.1312 lr=8.81e-06 tok=25.099B dt=3386 wait=181 +05:54:34 step 190480 loss=3.0234 lr=8.79e-06 tok=25.102B dt=3450 wait=181 +05:55:37 step 190500 loss=3.1191 lr=8.77e-06 tok=25.104B dt=3514 wait=181 +05:55:49 pushed ckpt_w189.pt,log_w189.txt +05:57:05 waited 76s for global r=817 +05:57:12 global r=817 load 4.279 -> 4.280 +05:58:17 step 190520 loss=2.9728 lr=8.76e-06 tok=25.107B dt=3673 wait=181 +05:59:20 step 190540 loss=3.0820 lr=8.74e-06 tok=25.109B dt=3736 wait=181 +06:00:24 step 190560 loss=2.9457 lr=8.72e-06 tok=25.112B dt=3800 wait=182 +06:01:27 step 190580 loss=2.9662 lr=8.70e-06 tok=25.115B dt=3863 wait=182 +06:02:30 step 190600 loss=2.9550 lr=8.68e-06 tok=25.117B dt=3926 wait=182 +06:03:33 step 190620 loss=3.1012 lr=8.66e-06 tok=25.120B dt=3989 wait=182 +06:04:36 step 190640 loss=2.9095 lr=8.64e-06 tok=25.123B dt=4052 wait=182 +06:05:19 pushed ckpt_w189.pt,log_w189.txt +06:06:35 waited 76s for global r=818 +06:06:42 global r=818 load 4.282 -> 4.279 +06:07:15 step 190660 loss=3.0221 lr=8.63e-06 tok=25.125B dt=4212 wait=182 +06:08:18 step 190680 loss=3.0823 lr=8.61e-06 tok=25.128B dt=4275 wait=182 +06:09:21 step 190700 loss=3.0616 lr=8.59e-06 tok=25.130B dt=4337 wait=183 +06:10:24 step 190720 loss=3.0743 lr=8.57e-06 tok=25.133B dt=4401 wait=183 +06:11:28 step 190740 loss=3.0282 lr=8.55e-06 tok=25.136B dt=4465 wait=183 +06:12:32 step 190760 loss=3.1493 lr=8.53e-06 tok=25.138B dt=4528 wait=183 +06:13:35 step 190780 loss=3.0549 lr=8.52e-06 tok=25.141B dt=4592 wait=183 +06:14:38 step 190800 loss=3.0844 lr=8.50e-06 tok=25.144B dt=4655 wait=183 +06:14:52 pushed ckpt_w189.pt,log_w189.txt +06:16:07 waited 76s for global r=819 +06:16:13 global r=819 load 4.278 -> 4.280 +06:17:18 step 190820 loss=3.0393 lr=8.48e-06 tok=25.146B dt=4815 wait=183 +06:18:21 step 190840 loss=3.0677 lr=8.46e-06 tok=25.149B dt=4878 wait=183 +06:19:24 step 190860 loss=3.0137 lr=8.44e-06 tok=25.151B dt=4941 wait=184 +06:20:28 step 190880 loss=3.0059 lr=8.42e-06 tok=25.154B dt=5005 wait=184 +06:21:32 step 190900 loss=3.0407 lr=8.40e-06 tok=25.157B dt=5068 wait=184 +06:22:36 step 190920 loss=3.1330 lr=8.39e-06 tok=25.159B dt=5132 wait=184 +06:23:39 step 190940 loss=2.9836 lr=8.37e-06 tok=25.162B dt=5196 wait=184 +06:24:24 pushed ckpt_w189.pt,log_w189.txt +06:25:39 waited 76s for global r=820 +06:25:45 global r=820 load 4.279 -> 4.279 +06:26:19 step 190960 loss=3.0493 lr=8.35e-06 tok=25.165B dt=5355 wait=184 +06:27:23 step 190980 loss=3.0163 lr=8.33e-06 tok=25.167B dt=5419 wait=185 +06:28:27 step 191000 loss=2.9723 lr=8.31e-06 tok=25.170B dt=5484 wait=185 +06:29:31 step 191020 loss=3.0744 lr=8.29e-06 tok=25.172B dt=5548 wait=185 +06:30:36 step 191040 loss=2.9333 lr=8.28e-06 tok=25.175B dt=5612 wait=185 +06:31:40 step 191060 loss=2.9923 lr=8.26e-06 tok=25.178B dt=5676 wait=185 +06:32:44 step 191080 loss=2.9170 lr=8.24e-06 tok=25.180B dt=5741 wait=185 +06:33:49 step 191100 loss=2.8496 lr=8.22e-06 tok=25.183B dt=5805 wait=185 +06:34:03 pushed ckpt_w189.pt,log_w189.txt +06:35:19 waited 75s for global r=821 +06:35:26 global r=821 load 4.278 -> 4.279 +06:36:32 step 191120 loss=2.8941 lr=8.20e-06 tok=25.185B dt=5968 wait=186 +06:37:35 step 191140 loss=3.0256 lr=8.18e-06 tok=25.188B dt=6032 wait=186 +06:38:39 step 191160 loss=3.0487 lr=8.16e-06 tok=25.191B dt=6095 wait=186 +06:39:43 step 191180 loss=2.9628 lr=8.15e-06 tok=25.193B dt=6159 wait=186 +06:40:48 step 191200 loss=2.9676 lr=8.13e-06 tok=25.196B dt=6224 wait=186 +06:41:51 step 191220 loss=3.0366 lr=8.11e-06 tok=25.199B dt=6288 wait=186 +06:42:56 step 191240 loss=2.8320 lr=8.09e-06 tok=25.201B dt=6353 wait=186 +06:43:42 pushed ckpt_w189.pt,log_w189.txt +06:44:43 waited 60s for global r=822 +06:44:48 global r=822 load 4.278 -> 4.278 +06:45:21 step 191260 loss=3.0186 lr=8.07e-06 tok=25.204B dt=6498 wait=187 +06:46:25 step 191280 loss=2.9932 lr=8.05e-06 tok=25.206B dt=6562 wait=187 +06:47:29 step 191300 loss=3.0082 lr=8.04e-06 tok=25.209B dt=6625 wait=187 +06:48:33 step 191320 loss=3.1005 lr=8.02e-06 tok=25.212B dt=6689 wait=187 +06:49:37 step 191340 loss=2.9634 lr=8.00e-06 tok=25.214B dt=6754 wait=187 +06:50:40 step 191360 loss=3.0114 lr=7.98e-06 tok=25.217B dt=6817 wait=187 +06:51:44 step 191380 loss=2.9373 lr=7.96e-06 tok=25.220B dt=6881 wait=187 +06:52:48 step 191400 loss=2.9456 lr=7.94e-06 tok=25.222B dt=6945 wait=188 +06:53:05 pushed ckpt_w189.pt,log_w189.txt +06:54:20 waited 76s for global r=823 +06:54:26 global r=823 load 4.277 -> 4.279 +06:55:31 step 191420 loss=2.9425 lr=7.92e-06 tok=25.225B dt=7107 wait=188 +06:56:34 step 191440 loss=2.9765 lr=7.91e-06 tok=25.227B dt=7171 wait=188 +06:57:38 step 191460 loss=2.8748 lr=7.89e-06 tok=25.230B dt=7235 wait=188 +06:58:41 step 191480 loss=3.0080 lr=7.87e-06 tok=25.233B dt=7298 wait=188 +06:59:46 step 191500 loss=2.8850 lr=7.85e-06 tok=25.235B dt=7362 wait=188 +07:00:49 step 191520 loss=3.0305 lr=7.83e-06 tok=25.238B dt=7426 wait=188 +07:01:53 step 191540 loss=2.9189 lr=7.81e-06 tok=25.241B dt=7489 wait=189 +07:02:38 pushed ckpt_w189.pt,log_w189.txt +07:03:53 waited 76s for global r=824 +07:04:00 global r=824 load 4.278 -> 4.279 +07:04:33 step 191560 loss=2.9893 lr=7.80e-06 tok=25.243B dt=7650 wait=189 +07:05:37 step 191580 loss=2.9227 lr=7.78e-06 tok=25.246B dt=7713 wait=189 +07:06:40 step 191600 loss=3.0143 lr=7.76e-06 tok=25.248B dt=7777 wait=189 +07:07:44 step 191620 loss=3.0590 lr=7.74e-06 tok=25.251B dt=7840 wait=189 +07:08:48 step 191640 loss=2.9822 lr=7.72e-06 tok=25.254B dt=7905 wait=189 +07:09:52 step 191660 loss=3.1197 lr=7.70e-06 tok=25.256B dt=7969 wait=189 +07:10:56 step 191680 loss=2.8860 lr=7.68e-06 tok=25.259B dt=8033 wait=190 +07:12:00 step 191700 loss=2.9850 lr=7.67e-06 tok=25.262B dt=8097 wait=190 +07:12:14 pushed ckpt_w189.pt,log_w189.txt +07:13:14 waited 60s for global r=825 +07:13:23 global r=825 load 4.281 -> 4.279 +07:14:29 step 191720 loss=2.8980 lr=7.65e-06 tok=25.264B dt=8245 wait=190 +07:15:33 step 191740 loss=2.9308 lr=7.63e-06 tok=25.267B dt=8309 wait=190 +07:16:37 step 191760 loss=2.9790 lr=7.61e-06 tok=25.269B dt=8374 wait=190 +07:17:41 step 191780 loss=2.8817 lr=7.59e-06 tok=25.272B dt=8438 wait=190 +07:18:45 step 191800 loss=2.9046 lr=7.57e-06 tok=25.275B dt=8501 wait=190 +07:19:48 step 191820 loss=3.0487 lr=7.55e-06 tok=25.277B dt=8565 wait=191 +07:20:52 step 191840 loss=2.8913 lr=7.54e-06 tok=25.280B dt=8629 wait=191 +07:21:37 pushed ckpt_w189.pt,log_w189.txt +07:22:52 waited 76s for global r=826 +07:22:57 global r=826 load 4.279 -> 4.280 +07:23:31 step 191860 loss=2.9679 lr=7.52e-06 tok=25.282B dt=8787 wait=191 +07:24:35 step 191880 loss=2.9998 lr=7.50e-06 tok=25.285B dt=8852 wait=191 +07:25:39 step 191900 loss=2.9110 lr=7.48e-06 tok=25.288B dt=8916 wait=191 +07:26:43 step 191920 loss=2.9251 lr=7.46e-06 tok=25.290B dt=8980 wait=191 +07:27:47 step 191940 loss=2.9027 lr=7.44e-06 tok=25.293B dt=9044 wait=191 +07:28:51 step 191960 loss=3.0496 lr=7.43e-06 tok=25.296B dt=9108 wait=191 +07:29:56 step 191980 loss=2.9848 lr=7.41e-06 tok=25.298B dt=9172 wait=192 +07:31:00 step 192000 loss=3.0867 lr=7.39e-06 tok=25.301B dt=9237 wait=192 +07:31:12 pushed ckpt_w189.pt,log_w189.txt +07:32:28 waited 76s for global r=827 +07:32:33 global r=827 load 4.280 -> 4.280 +07:33:39 step 192020 loss=2.9463 lr=7.37e-06 tok=25.303B dt=9395 wait=192 +07:34:43 step 192040 loss=3.1153 lr=7.35e-06 tok=25.306B dt=9459 wait=192 +07:35:48 step 192060 loss=3.0211 lr=7.33e-06 tok=25.309B dt=9525 wait=192 +07:36:53 step 192080 loss=2.9360 lr=7.31e-06 tok=25.311B dt=9589 wait=192 +07:37:57 step 192100 loss=3.0049 lr=7.30e-06 tok=25.314B dt=9654 wait=193 +07:39:02 step 192120 loss=2.9981 lr=7.28e-06 tok=25.317B dt=9719 wait=193 +07:40:06 step 192140 loss=2.9040 lr=7.26e-06 tok=25.319B dt=9783 wait=193 diff --git a/log_w190.txt b/log_w190.txt new file mode 100644 index 0000000000000000000000000000000000000000..6c2d826c8d8c1b6a21052c63a245c29ee021601e --- /dev/null +++ b/log_w190.txt @@ -0,0 +1,183 @@ +launch +04:56:36 name=w190 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 137/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +04:56:42 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +04:56:42 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +04:56:52 params 1.08B +04:56:53 resumed step=189600 seen=24.99B round=811 +04:56:53 stream salt=189600 +04:58:34 blocks compiled +04:58:34 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:02:00 step 189601 loss=2.7217 lr=9.60e-06 tok=24.986B dt=206 wait=179 +05:02:16 step 189602 loss=2.9357 lr=9.60e-06 tok=24.987B dt=221 wait=179 +05:03:13 step 189620 loss=3.0487 lr=9.59e-06 tok=24.989B dt=279 wait=179 +05:04:17 step 189640 loss=2.7143 lr=9.57e-06 tok=24.991B dt=343 wait=179 +05:05:20 step 189660 loss=3.0163 lr=9.55e-06 tok=24.994B dt=406 wait=180 +05:06:24 step 189680 loss=3.0423 lr=9.53e-06 tok=24.997B dt=470 wait=180 +05:07:27 step 189700 loss=3.1413 lr=9.51e-06 tok=24.999B dt=533 wait=180 +05:08:31 step 189720 loss=3.0909 lr=9.49e-06 tok=25.002B dt=597 wait=180 +05:09:16 global r=812 load 4.276 -> 4.278 +05:09:47 step 189760 loss=3.0601 lr=9.46e-06 tok=25.005B dt=673 wait=180 +05:10:51 step 189780 loss=3.0130 lr=9.44e-06 tok=25.007B dt=737 wait=180 +05:11:55 step 189800 loss=3.0136 lr=9.42e-06 tok=25.010B dt=801 wait=180 +05:13:00 step 189820 loss=2.6960 lr=9.40e-06 tok=25.012B dt=866 wait=181 +05:14:04 step 189840 loss=2.9404 lr=9.38e-06 tok=25.015B dt=930 wait=181 +05:15:08 step 189860 loss=2.9242 lr=9.36e-06 tok=25.018B dt=994 wait=181 +05:16:12 step 189880 loss=3.1535 lr=9.35e-06 tok=25.020B dt=1058 wait=181 +05:17:16 step 189900 loss=3.2406 lr=9.33e-06 tok=25.023B dt=1122 wait=181 +05:17:30 pushed ckpt_w190.pt,log_w190.txt +05:18:46 waited 76s for global r=813 +05:18:53 global r=813 load 4.279 -> 4.279 +05:19:59 step 189920 loss=3.0995 lr=9.31e-06 tok=25.026B dt=1285 wait=181 +05:21:02 step 189940 loss=3.0299 lr=9.29e-06 tok=25.028B dt=1348 wait=182 +05:22:05 step 189960 loss=3.0299 lr=9.27e-06 tok=25.031B dt=1411 wait=182 +05:23:10 step 189980 loss=3.0047 lr=9.25e-06 tok=25.033B dt=1476 wait=182 +05:24:13 step 190000 loss=3.0843 lr=9.24e-06 tok=25.036B dt=1539 wait=182 +05:25:16 step 190020 loss=3.0315 lr=9.22e-06 tok=25.039B dt=1602 wait=182 +05:26:20 step 190040 loss=2.8467 lr=9.20e-06 tok=25.041B dt=1665 wait=182 +05:27:03 pushed ckpt_w190.pt,log_w190.txt +05:28:19 waited 76s for global r=814 +05:28:25 global r=814 load 4.282 -> 4.279 +05:28:59 step 190060 loss=2.7546 lr=9.18e-06 tok=25.044B dt=1825 wait=183 +05:30:02 step 190080 loss=2.8684 lr=9.16e-06 tok=25.047B dt=1888 wait=183 +05:31:05 step 190100 loss=2.9161 lr=9.14e-06 tok=25.049B dt=1951 wait=184 +05:32:08 step 190120 loss=2.9653 lr=9.12e-06 tok=25.052B dt=2014 wait=184 +05:33:12 step 190140 loss=2.8911 lr=9.11e-06 tok=25.054B dt=2078 wait=184 +05:34:15 step 190160 loss=2.9660 lr=9.09e-06 tok=25.057B dt=2141 wait=184 +05:35:19 step 190180 loss=2.6348 lr=9.07e-06 tok=25.060B dt=2205 wait=184 +05:36:23 step 190200 loss=2.9816 lr=9.05e-06 tok=25.062B dt=2269 wait=184 +05:36:36 pushed ckpt_w190.pt,log_w190.txt +05:37:51 waited 76s for global r=815 +05:37:57 global r=815 load 4.282 -> 4.279 +05:39:01 step 190220 loss=2.9686 lr=9.03e-06 tok=25.065B dt=2427 wait=184 +05:40:04 step 190240 loss=3.0264 lr=9.01e-06 tok=25.068B dt=2490 wait=185 +05:41:07 step 190260 loss=2.9856 lr=9.00e-06 tok=25.070B dt=2553 wait=185 +05:42:10 step 190280 loss=2.9895 lr=8.98e-06 tok=25.073B dt=2616 wait=185 +05:43:13 step 190300 loss=3.0180 lr=8.96e-06 tok=25.075B dt=2679 wait=185 +05:44:17 step 190320 loss=3.1527 lr=8.94e-06 tok=25.078B dt=2743 wait=185 +05:45:20 step 190340 loss=2.9871 lr=8.92e-06 tok=25.081B dt=2806 wait=185 +05:46:05 pushed ckpt_w190.pt,log_w190.txt +05:47:21 waited 76s for global r=816 +05:47:26 global r=816 load 4.278 -> 4.280 +05:47:59 step 190360 loss=3.2686 lr=8.90e-06 tok=25.083B dt=2965 wait=186 +05:49:03 step 190380 loss=3.1946 lr=8.88e-06 tok=25.086B dt=3029 wait=186 +05:50:06 step 190400 loss=3.1076 lr=8.87e-06 tok=25.088B dt=3092 wait=186 +05:51:10 step 190420 loss=3.1552 lr=8.85e-06 tok=25.091B dt=3156 wait=186 +05:52:14 step 190440 loss=3.1061 lr=8.83e-06 tok=25.094B dt=3220 wait=186 +05:53:18 step 190460 loss=3.1394 lr=8.81e-06 tok=25.096B dt=3284 wait=186 +05:54:22 step 190480 loss=3.1632 lr=8.79e-06 tok=25.099B dt=3348 wait=186 +05:55:26 step 190500 loss=3.1317 lr=8.77e-06 tok=25.102B dt=3412 wait=187 +05:55:38 pushed ckpt_w190.pt,log_w190.txt +05:56:53 waited 76s for global r=817 +05:57:02 global r=817 load 4.281 -> 4.280 +05:58:07 step 190520 loss=3.1321 lr=8.76e-06 tok=25.104B dt=3573 wait=187 +05:59:11 step 190540 loss=3.1867 lr=8.74e-06 tok=25.107B dt=3637 wait=187 +06:00:15 step 190560 loss=3.1460 lr=8.72e-06 tok=25.109B dt=3701 wait=187 +06:01:20 step 190580 loss=3.1298 lr=8.70e-06 tok=25.112B dt=3766 wait=187 +06:02:25 step 190600 loss=3.3037 lr=8.68e-06 tok=25.115B dt=3831 wait=187 +06:03:31 step 190620 loss=3.3531 lr=8.66e-06 tok=25.117B dt=3897 wait=188 +06:04:36 step 190640 loss=3.2381 lr=8.64e-06 tok=25.120B dt=3962 wait=189 +06:05:19 pushed ckpt_w190.pt,log_w190.txt +06:06:35 waited 76s for global r=818 +06:06:40 global r=818 load 4.277 -> 4.279 +06:07:14 step 190660 loss=3.2339 lr=8.63e-06 tok=25.123B dt=4120 wait=189 +06:08:18 step 190680 loss=3.1216 lr=8.61e-06 tok=25.125B dt=4184 wait=189 +06:09:22 step 190700 loss=2.8788 lr=8.59e-06 tok=25.128B dt=4248 wait=189 +06:10:26 step 190720 loss=3.1702 lr=8.57e-06 tok=25.130B dt=4312 wait=189 +06:11:30 step 190740 loss=3.0371 lr=8.55e-06 tok=25.133B dt=4376 wait=189 +06:12:33 step 190760 loss=3.0455 lr=8.53e-06 tok=25.136B dt=4439 wait=189 +06:13:37 step 190780 loss=2.9976 lr=8.52e-06 tok=25.138B dt=4503 wait=190 +06:14:40 step 190800 loss=2.7453 lr=8.50e-06 tok=25.141B dt=4566 wait=190 +06:14:52 pushed ckpt_w190.pt,log_w190.txt +06:16:07 waited 76s for global r=819 +06:16:19 global r=819 load 4.279 -> 4.280 +06:17:23 step 190820 loss=2.8933 lr=8.48e-06 tok=25.144B dt=4729 wait=190 +06:18:27 step 190840 loss=2.8745 lr=8.46e-06 tok=25.146B dt=4793 wait=190 +06:19:29 step 190860 loss=3.0829 lr=8.44e-06 tok=25.149B dt=4855 wait=190 +06:20:33 step 190880 loss=2.9293 lr=8.42e-06 tok=25.151B dt=4919 wait=190 +06:21:37 step 190900 loss=3.0104 lr=8.40e-06 tok=25.154B dt=4983 wait=190 +06:22:39 step 190920 loss=2.9943 lr=8.39e-06 tok=25.157B dt=5045 wait=191 +06:23:42 step 190940 loss=2.8808 lr=8.37e-06 tok=25.159B dt=5108 wait=191 +06:24:28 pushed ckpt_w190.pt,log_w190.txt +06:25:43 waited 76s for global r=820 +06:25:51 global r=820 load 4.279 -> 4.279 +06:26:23 step 190960 loss=3.0092 lr=8.35e-06 tok=25.162B dt=5269 wait=191 +06:27:26 step 190980 loss=2.8863 lr=8.33e-06 tok=25.165B dt=5332 wait=191 +06:28:29 step 191000 loss=2.8162 lr=8.31e-06 tok=25.167B dt=5395 wait=191 +06:29:33 step 191020 loss=2.9496 lr=8.29e-06 tok=25.170B dt=5459 wait=191 +06:30:37 step 191040 loss=2.8746 lr=8.28e-06 tok=25.172B dt=5522 wait=191 +06:31:40 step 191060 loss=2.9381 lr=8.26e-06 tok=25.175B dt=5586 wait=192 +06:32:43 step 191080 loss=2.8933 lr=8.24e-06 tok=25.178B dt=5649 wait=192 +06:33:46 step 191100 loss=2.9360 lr=8.22e-06 tok=25.180B dt=5712 wait=192 +06:33:57 pushed ckpt_w190.pt,log_w190.txt +06:35:13 waited 76s for global r=821 +06:35:21 global r=821 load 4.279 -> 4.279 +06:36:26 step 191120 loss=2.8310 lr=8.20e-06 tok=25.183B dt=5872 wait=192 +06:37:29 step 191140 loss=2.9362 lr=8.18e-06 tok=25.185B dt=5935 wait=192 +06:38:31 step 191160 loss=3.0074 lr=8.16e-06 tok=25.188B dt=5997 wait=192 +06:39:34 step 191180 loss=2.8905 lr=8.15e-06 tok=25.191B dt=6060 wait=192 +06:40:37 step 191200 loss=2.8909 lr=8.13e-06 tok=25.193B dt=6123 wait=193 +06:41:41 step 191220 loss=2.8771 lr=8.11e-06 tok=25.196B dt=6187 wait=193 +06:42:44 step 191240 loss=2.9065 lr=8.09e-06 tok=25.199B dt=6250 wait=193 +06:43:28 pushed ckpt_w190.pt,log_w190.txt +06:44:43 waited 76s for global r=822 +06:44:48 global r=822 load 4.280 -> 4.278 +06:45:22 step 191260 loss=2.9878 lr=8.07e-06 tok=25.201B dt=6407 wait=193 +06:46:25 step 191280 loss=2.9277 lr=8.05e-06 tok=25.204B dt=6471 wait=193 +06:47:29 step 191300 loss=3.0052 lr=8.04e-06 tok=25.206B dt=6534 wait=193 +06:48:32 step 191320 loss=2.9827 lr=8.02e-06 tok=25.209B dt=6598 wait=194 +06:49:37 step 191340 loss=3.0170 lr=8.00e-06 tok=25.212B dt=6663 wait=194 +06:50:41 step 191360 loss=3.0213 lr=7.98e-06 tok=25.214B dt=6727 wait=194 +06:51:45 step 191380 loss=3.0791 lr=7.96e-06 tok=25.217B dt=6791 wait=194 +06:52:49 step 191400 loss=3.0480 lr=7.94e-06 tok=25.220B dt=6855 wait=194 +06:53:01 pushed ckpt_w190.pt,log_w190.txt +06:54:17 waited 76s for global r=823 +06:54:23 global r=823 load 4.280 -> 4.279 +06:55:29 step 191420 loss=2.9672 lr=7.92e-06 tok=25.222B dt=7014 wait=194 +06:56:32 step 191440 loss=3.0423 lr=7.91e-06 tok=25.225B dt=7078 wait=195 +06:57:37 step 191460 loss=3.0149 lr=7.89e-06 tok=25.227B dt=7143 wait=195 +06:58:41 step 191480 loss=3.0772 lr=7.87e-06 tok=25.230B dt=7206 wait=195 +06:59:45 step 191500 loss=3.0833 lr=7.85e-06 tok=25.233B dt=7271 wait=195 +07:00:49 step 191520 loss=3.0724 lr=7.83e-06 tok=25.235B dt=7335 wait=195 +07:01:53 step 191540 loss=2.9451 lr=7.81e-06 tok=25.238B dt=7399 wait=195 +07:02:38 pushed ckpt_w190.pt,log_w190.txt +07:03:54 waited 76s for global r=824 +07:04:00 global r=824 load 4.280 -> 4.279 +07:04:33 step 191560 loss=3.0412 lr=7.80e-06 tok=25.241B dt=7559 wait=195 +07:05:37 step 191580 loss=2.9634 lr=7.78e-06 tok=25.243B dt=7623 wait=196 +07:06:41 step 191600 loss=2.6991 lr=7.76e-06 tok=25.246B dt=7687 wait=196 +07:07:45 step 191620 loss=3.1503 lr=7.74e-06 tok=25.248B dt=7751 wait=196 +07:08:49 step 191640 loss=2.9708 lr=7.72e-06 tok=25.251B dt=7815 wait=196 +07:09:53 step 191660 loss=2.9554 lr=7.70e-06 tok=25.254B dt=7879 wait=196 +07:10:56 step 191680 loss=2.9658 lr=7.68e-06 tok=25.256B dt=7942 wait=196 +07:12:00 step 191700 loss=2.9423 lr=7.67e-06 tok=25.259B dt=8005 wait=197 +07:12:15 pushed ckpt_w190.pt,log_w190.txt +07:13:15 waited 61s for global r=825 +07:13:20 global r=825 load 4.280 -> 4.279 +07:14:26 step 191720 loss=2.6613 lr=7.65e-06 tok=25.262B dt=8152 wait=197 +07:15:30 step 191740 loss=3.0024 lr=7.63e-06 tok=25.264B dt=8216 wait=197 +07:16:34 step 191760 loss=2.9951 lr=7.61e-06 tok=25.267B dt=8280 wait=197 +07:17:38 step 191780 loss=2.9656 lr=7.59e-06 tok=25.269B dt=8344 wait=197 +07:18:42 step 191800 loss=2.9526 lr=7.57e-06 tok=25.272B dt=8408 wait=197 +07:19:47 step 191820 loss=2.9080 lr=7.55e-06 tok=25.275B dt=8473 wait=197 +07:20:51 step 191840 loss=2.9951 lr=7.54e-06 tok=25.277B dt=8536 wait=198 +07:21:36 pushed ckpt_w190.pt,log_w190.txt +07:22:52 waited 76s for global r=826 +07:22:59 global r=826 load 4.281 -> 4.280 +07:23:33 step 191860 loss=2.8432 lr=7.52e-06 tok=25.280B dt=8699 wait=198 +07:24:37 step 191880 loss=2.8641 lr=7.50e-06 tok=25.282B dt=8763 wait=198 +07:25:41 step 191900 loss=2.9570 lr=7.48e-06 tok=25.285B dt=8827 wait=198 +07:26:45 step 191920 loss=3.0448 lr=7.46e-06 tok=25.288B dt=8891 wait=198 +07:27:49 step 191940 loss=3.0023 lr=7.44e-06 tok=25.290B dt=8955 wait=198 +07:28:55 step 191960 loss=2.4980 lr=7.43e-06 tok=25.293B dt=9021 wait=198 +07:29:59 step 191980 loss=2.9151 lr=7.41e-06 tok=25.296B dt=9085 wait=199 +07:31:03 step 192000 loss=2.9970 lr=7.39e-06 tok=25.298B dt=9149 wait=199 +07:31:15 pushed ckpt_w190.pt,log_w190.txt +07:32:15 waited 61s for global r=827 +07:32:21 global r=827 load 4.282 -> 4.280 +07:33:27 step 192020 loss=2.9034 lr=7.37e-06 tok=25.301B dt=9293 wait=199 +07:34:31 step 192040 loss=3.0021 lr=7.35e-06 tok=25.303B dt=9357 wait=199 +07:35:35 step 192060 loss=3.0514 lr=7.33e-06 tok=25.306B dt=9421 wait=199 +07:36:40 step 192080 loss=3.0284 lr=7.31e-06 tok=25.309B dt=9486 wait=199 +07:37:44 step 192100 loss=3.0031 lr=7.30e-06 tok=25.311B dt=9550 wait=199 +07:38:48 step 192120 loss=3.0847 lr=7.28e-06 tok=25.314B dt=9614 wait=200 +07:39:52 step 192140 loss=2.9584 lr=7.26e-06 tok=25.317B dt=9678 wait=200 diff --git a/log_w191.txt b/log_w191.txt new file mode 100644 index 0000000000000000000000000000000000000000..c15212a0ffee292866636e59c8d2b4215f871bab --- /dev/null +++ b/log_w191.txt @@ -0,0 +1,181 @@ +launch +04:58:16 name=w191 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 138/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +04:58:22 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +04:58:22 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +04:58:32 params 1.08B +04:58:33 resumed step=189600 seen=24.99B round=811 +04:58:33 stream salt=189600 +05:00:16 blocks compiled +05:00:16 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:03:52 step 189601 loss=2.6432 lr=9.60e-06 tok=24.986B dt=216 wait=189 +05:04:08 step 189602 loss=2.7796 lr=9.60e-06 tok=24.987B dt=232 wait=189 +05:05:06 step 189620 loss=3.1110 lr=9.59e-06 tok=24.989B dt=289 wait=189 +05:06:10 step 189640 loss=2.9993 lr=9.57e-06 tok=24.991B dt=354 wait=189 +05:07:14 step 189660 loss=3.1316 lr=9.55e-06 tok=24.994B dt=418 wait=189 +05:08:17 step 189680 loss=3.2298 lr=9.53e-06 tok=24.997B dt=481 wait=189 +05:09:04 global r=812 load 4.277 -> 4.278 +05:09:36 step 189760 loss=3.0474 lr=9.46e-06 tok=24.999B dt=560 wait=189 +05:10:41 step 189780 loss=3.0503 lr=9.44e-06 tok=25.002B dt=624 wait=190 +05:11:45 step 189800 loss=2.9487 lr=9.42e-06 tok=25.005B dt=689 wait=190 +05:12:50 step 189820 loss=2.9675 lr=9.40e-06 tok=25.007B dt=754 wait=190 +05:13:55 step 189840 loss=2.9069 lr=9.38e-06 tok=25.010B dt=819 wait=190 +05:15:00 step 189860 loss=2.8776 lr=9.36e-06 tok=25.012B dt=884 wait=190 +05:16:04 step 189880 loss=2.9901 lr=9.35e-06 tok=25.015B dt=947 wait=190 +05:17:08 step 189900 loss=3.0378 lr=9.33e-06 tok=25.018B dt=1012 wait=190 +05:17:21 pushed ckpt_w191.pt,log_w191.txt +05:18:52 waited 91s for global r=813 +05:18:59 global r=813 load 4.278 -> 4.279 +05:20:05 step 189920 loss=3.0577 lr=9.31e-06 tok=25.020B dt=1189 wait=191 +05:21:10 step 189940 loss=2.9904 lr=9.29e-06 tok=25.023B dt=1254 wait=191 +05:22:13 step 189960 loss=3.0081 lr=9.27e-06 tok=25.026B dt=1317 wait=191 +05:23:18 step 189980 loss=2.9735 lr=9.25e-06 tok=25.028B dt=1382 wait=191 +05:24:22 step 190000 loss=3.0720 lr=9.24e-06 tok=25.031B dt=1446 wait=191 +05:25:26 step 190020 loss=3.1370 lr=9.22e-06 tok=25.033B dt=1510 wait=191 +05:26:30 step 190040 loss=2.9654 lr=9.20e-06 tok=25.036B dt=1574 wait=191 +05:27:14 pushed ckpt_w191.pt,log_w191.txt +05:28:30 waited 76s for global r=814 +05:28:35 global r=814 load 4.277 -> 4.279 +05:29:07 step 190060 loss=3.1335 lr=9.18e-06 tok=25.039B dt=1731 wait=192 +05:30:11 step 190080 loss=3.0539 lr=9.16e-06 tok=25.041B dt=1795 wait=192 +05:31:16 step 190100 loss=3.0277 lr=9.14e-06 tok=25.044B dt=1859 wait=192 +05:32:20 step 190120 loss=3.0784 lr=9.12e-06 tok=25.047B dt=1923 wait=192 +05:33:24 step 190140 loss=2.9936 lr=9.11e-06 tok=25.049B dt=1988 wait=192 +05:34:29 step 190160 loss=3.0989 lr=9.09e-06 tok=25.052B dt=2053 wait=192 +05:35:34 step 190180 loss=3.0065 lr=9.07e-06 tok=25.054B dt=2118 wait=192 +05:36:39 step 190200 loss=2.9996 lr=9.05e-06 tok=25.057B dt=2183 wait=193 +05:36:51 pushed ckpt_w191.pt,log_w191.txt +05:37:52 waited 60s for global r=815 +05:37:57 global r=815 load 4.279 -> 4.279 +05:39:03 step 190220 loss=2.9799 lr=9.03e-06 tok=25.060B dt=2327 wait=193 +05:40:08 step 190240 loss=3.0024 lr=9.01e-06 tok=25.062B dt=2392 wait=193 +05:41:12 step 190260 loss=3.0470 lr=9.00e-06 tok=25.065B dt=2456 wait=193 +05:42:17 step 190280 loss=2.9629 lr=8.98e-06 tok=25.068B dt=2521 wait=193 +05:43:22 step 190300 loss=2.9262 lr=8.96e-06 tok=25.070B dt=2586 wait=193 +05:44:26 step 190320 loss=3.0882 lr=8.94e-06 tok=25.073B dt=2650 wait=193 +05:45:30 step 190340 loss=2.8403 lr=8.92e-06 tok=25.075B dt=2714 wait=194 +05:46:17 pushed ckpt_w191.pt,log_w191.txt +05:47:33 waited 76s for global r=816 +05:47:38 global r=816 load 4.279 -> 4.280 +05:48:12 step 190360 loss=2.9870 lr=8.90e-06 tok=25.078B dt=2876 wait=194 +05:49:17 step 190380 loss=2.8708 lr=8.88e-06 tok=25.081B dt=2941 wait=194 +05:50:21 step 190400 loss=2.9136 lr=8.87e-06 tok=25.083B dt=3005 wait=194 +05:51:26 step 190420 loss=2.9281 lr=8.85e-06 tok=25.086B dt=3070 wait=194 +05:52:30 step 190440 loss=2.9386 lr=8.83e-06 tok=25.088B dt=3134 wait=194 +05:53:36 step 190460 loss=3.0026 lr=8.81e-06 tok=25.091B dt=3200 wait=195 +05:54:41 step 190480 loss=2.8275 lr=8.79e-06 tok=25.094B dt=3265 wait=195 +05:55:46 step 190500 loss=3.0100 lr=8.77e-06 tok=25.096B dt=3330 wait=195 +05:55:59 pushed ckpt_w191.pt,log_w191.txt +05:57:00 waited 61s for global r=817 +05:57:06 global r=817 load 4.280 -> 4.280 +05:58:12 step 190520 loss=2.9975 lr=8.76e-06 tok=25.099B dt=3476 wait=195 +05:59:17 step 190540 loss=2.9552 lr=8.74e-06 tok=25.102B dt=3541 wait=195 +06:00:22 step 190560 loss=3.0017 lr=8.72e-06 tok=25.104B dt=3606 wait=195 +06:01:26 step 190580 loss=2.9194 lr=8.70e-06 tok=25.107B dt=3670 wait=195 +06:02:31 step 190600 loss=3.0521 lr=8.68e-06 tok=25.109B dt=3735 wait=195 +06:03:36 step 190620 loss=3.0098 lr=8.66e-06 tok=25.112B dt=3800 wait=196 +06:04:41 step 190640 loss=2.8835 lr=8.64e-06 tok=25.115B dt=3864 wait=196 +06:05:29 pushed ckpt_w191.pt,log_w191.txt +06:06:44 waited 76s for global r=818 +06:06:50 global r=818 load 4.278 -> 4.279 +06:07:24 step 190660 loss=2.9979 lr=8.63e-06 tok=25.117B dt=4028 wait=196 +06:08:29 step 190680 loss=3.0089 lr=8.61e-06 tok=25.120B dt=4093 wait=196 +06:09:33 step 190700 loss=2.9068 lr=8.59e-06 tok=25.123B dt=4157 wait=196 +06:10:38 step 190720 loss=3.0294 lr=8.57e-06 tok=25.125B dt=4222 wait=196 +06:11:42 step 190740 loss=2.9885 lr=8.55e-06 tok=25.128B dt=4286 wait=197 +06:12:46 step 190760 loss=3.0284 lr=8.53e-06 tok=25.130B dt=4349 wait=197 +06:13:50 step 190780 loss=2.9988 lr=8.52e-06 tok=25.133B dt=4414 wait=197 +06:14:56 step 190800 loss=2.7733 lr=8.50e-06 tok=25.136B dt=4479 wait=197 +06:15:09 pushed ckpt_w191.pt,log_w191.txt +06:16:09 waited 60s for global r=819 +06:16:15 global r=819 load 4.280 -> 4.280 +06:17:21 step 190820 loss=2.8139 lr=8.48e-06 tok=25.138B dt=4625 wait=197 +06:18:25 step 190840 loss=2.9813 lr=8.46e-06 tok=25.141B dt=4689 wait=197 +06:19:30 step 190860 loss=2.9311 lr=8.44e-06 tok=25.144B dt=4754 wait=198 +06:20:35 step 190880 loss=2.9174 lr=8.42e-06 tok=25.146B dt=4818 wait=198 +06:21:38 step 190900 loss=3.0245 lr=8.40e-06 tok=25.149B dt=4882 wait=198 +06:22:43 step 190920 loss=3.0414 lr=8.39e-06 tok=25.151B dt=4947 wait=198 +06:23:48 step 190940 loss=2.9427 lr=8.37e-06 tok=25.154B dt=5012 wait=198 +06:24:35 pushed ckpt_w191.pt,log_w191.txt +06:25:36 waited 60s for global r=820 +06:25:42 global r=820 load 4.278 -> 4.279 +06:26:16 step 190960 loss=3.0423 lr=8.35e-06 tok=25.157B dt=5159 wait=198 +06:27:20 step 190980 loss=2.9228 lr=8.33e-06 tok=25.159B dt=5224 wait=198 +06:28:24 step 191000 loss=2.9928 lr=8.31e-06 tok=25.162B dt=5288 wait=198 +06:29:29 step 191020 loss=3.0117 lr=8.29e-06 tok=25.165B dt=5353 wait=199 +06:30:34 step 191040 loss=2.9676 lr=8.28e-06 tok=25.167B dt=5418 wait=199 +06:31:39 step 191060 loss=3.0182 lr=8.26e-06 tok=25.170B dt=5483 wait=199 +06:32:43 step 191080 loss=2.8952 lr=8.24e-06 tok=25.172B dt=5547 wait=199 +06:33:48 step 191100 loss=2.9691 lr=8.22e-06 tok=25.175B dt=5612 wait=199 +06:34:03 pushed ckpt_w191.pt,log_w191.txt +06:35:18 waited 76s for global r=821 +06:35:23 global r=821 load 4.280 -> 4.279 +06:36:29 step 191120 loss=2.9726 lr=8.20e-06 tok=25.178B dt=5772 wait=199 +06:37:33 step 191140 loss=3.0477 lr=8.18e-06 tok=25.180B dt=5836 wait=200 +06:38:36 step 191160 loss=2.9679 lr=8.16e-06 tok=25.183B dt=5900 wait=200 +06:39:41 step 191180 loss=2.9655 lr=8.15e-06 tok=25.185B dt=5964 wait=200 +06:40:44 step 191200 loss=2.8836 lr=8.13e-06 tok=25.188B dt=6028 wait=200 +06:41:48 step 191220 loss=2.9938 lr=8.11e-06 tok=25.191B dt=6092 wait=200 +06:42:52 step 191240 loss=2.7747 lr=8.09e-06 tok=25.193B dt=6156 wait=200 +06:43:39 pushed ckpt_w191.pt,log_w191.txt +06:44:39 waited 61s for global r=822 +06:44:44 global r=822 load 4.279 -> 4.278 +06:45:17 step 191260 loss=2.9571 lr=8.07e-06 tok=25.196B dt=6301 wait=200 +06:46:21 step 191280 loss=2.9131 lr=8.05e-06 tok=25.199B dt=6365 wait=201 +06:47:25 step 191300 loss=2.9227 lr=8.04e-06 tok=25.201B dt=6429 wait=201 +06:48:28 step 191320 loss=3.0152 lr=8.02e-06 tok=25.204B dt=6492 wait=201 +06:49:32 step 191340 loss=2.8447 lr=8.00e-06 tok=25.206B dt=6556 wait=201 +06:50:36 step 191360 loss=2.9043 lr=7.98e-06 tok=25.209B dt=6620 wait=201 +06:51:40 step 191380 loss=2.8728 lr=7.96e-06 tok=25.212B dt=6684 wait=201 +06:52:45 step 191400 loss=2.8053 lr=7.94e-06 tok=25.214B dt=6749 wait=201 +06:52:56 pushed ckpt_w191.pt,log_w191.txt +06:54:12 waited 76s for global r=823 +06:54:16 global r=823 load 4.279 -> 4.279 +06:55:22 step 191420 loss=2.8685 lr=7.92e-06 tok=25.217B dt=6906 wait=202 +06:56:26 step 191440 loss=2.6438 lr=7.91e-06 tok=25.220B dt=6970 wait=202 +06:57:30 step 191460 loss=2.8831 lr=7.89e-06 tok=25.222B dt=7034 wait=202 +06:58:35 step 191480 loss=2.8074 lr=7.87e-06 tok=25.225B dt=7099 wait=202 +06:59:38 step 191500 loss=2.9414 lr=7.85e-06 tok=25.227B dt=7162 wait=202 +07:00:42 step 191520 loss=2.9875 lr=7.83e-06 tok=25.230B dt=7226 wait=202 +07:01:46 step 191540 loss=2.9281 lr=7.81e-06 tok=25.233B dt=7290 wait=202 +07:02:30 pushed ckpt_w191.pt,log_w191.txt +07:03:45 waited 76s for global r=824 +07:03:52 global r=824 load 4.278 -> 4.279 +07:04:25 step 191560 loss=2.9237 lr=7.80e-06 tok=25.235B dt=7449 wait=203 +07:05:29 step 191580 loss=2.8579 lr=7.78e-06 tok=25.238B dt=7512 wait=203 +07:06:32 step 191600 loss=2.9445 lr=7.76e-06 tok=25.241B dt=7576 wait=203 +07:07:36 step 191620 loss=2.9450 lr=7.74e-06 tok=25.243B dt=7640 wait=203 +07:08:40 step 191640 loss=2.8980 lr=7.72e-06 tok=25.246B dt=7704 wait=203 +07:09:44 step 191660 loss=2.9745 lr=7.70e-06 tok=25.248B dt=7768 wait=203 +07:10:47 step 191680 loss=2.9957 lr=7.68e-06 tok=25.251B dt=7831 wait=203 +07:11:51 step 191700 loss=3.0165 lr=7.67e-06 tok=25.254B dt=7895 wait=204 +07:12:04 pushed ckpt_w191.pt,log_w191.txt +07:13:19 waited 76s for global r=825 +07:13:24 global r=825 load 4.280 -> 4.279 +07:14:29 step 191720 loss=2.8896 lr=7.65e-06 tok=25.256B dt=8053 wait=204 +07:15:33 step 191740 loss=2.9840 lr=7.63e-06 tok=25.259B dt=8117 wait=204 +07:16:37 step 191760 loss=3.0080 lr=7.61e-06 tok=25.262B dt=8181 wait=204 +07:17:42 step 191780 loss=2.7666 lr=7.59e-06 tok=25.264B dt=8245 wait=204 +07:18:46 step 191800 loss=2.9928 lr=7.57e-06 tok=25.267B dt=8309 wait=204 +07:19:50 step 191820 loss=2.9819 lr=7.55e-06 tok=25.269B dt=8374 wait=204 +07:20:53 step 191840 loss=2.8538 lr=7.54e-06 tok=25.272B dt=8437 wait=205 +07:21:37 pushed ckpt_w191.pt,log_w191.txt +07:22:53 waited 76s for global r=826 +07:23:01 global r=826 load 4.279 -> 4.280 +07:23:34 step 191860 loss=3.0116 lr=7.52e-06 tok=25.275B dt=8598 wait=205 +07:24:38 step 191880 loss=2.8800 lr=7.50e-06 tok=25.277B dt=8661 wait=205 +07:25:41 step 191900 loss=2.8688 lr=7.48e-06 tok=25.280B dt=8725 wait=205 +07:26:45 step 191920 loss=2.9678 lr=7.46e-06 tok=25.282B dt=8789 wait=205 +07:27:49 step 191940 loss=2.9408 lr=7.44e-06 tok=25.285B dt=8853 wait=205 +07:28:53 step 191960 loss=3.0030 lr=7.43e-06 tok=25.288B dt=8917 wait=205 +07:29:57 step 191980 loss=2.9167 lr=7.41e-06 tok=25.290B dt=8981 wait=206 +07:31:01 step 192000 loss=3.0066 lr=7.39e-06 tok=25.293B dt=9045 wait=206 +07:31:13 pushed ckpt_w191.pt,log_w191.txt +07:32:28 waited 76s for global r=827 +07:32:33 global r=827 load 4.279 -> 4.280 +07:33:39 step 192020 loss=2.7109 lr=7.37e-06 tok=25.296B dt=9203 wait=206 +07:34:43 step 192040 loss=2.9459 lr=7.35e-06 tok=25.298B dt=9267 wait=206 +07:35:48 step 192060 loss=2.9507 lr=7.33e-06 tok=25.301B dt=9332 wait=206 +07:36:52 step 192080 loss=3.0274 lr=7.31e-06 tok=25.303B dt=9396 wait=206 +07:37:57 step 192100 loss=2.9767 lr=7.30e-06 tok=25.306B dt=9461 wait=206 +07:39:02 step 192120 loss=2.9661 lr=7.28e-06 tok=25.309B dt=9526 wait=207 +07:40:06 step 192140 loss=3.0280 lr=7.26e-06 tok=25.311B dt=9589 wait=207 diff --git a/log_w192.txt b/log_w192.txt new file mode 100644 index 0000000000000000000000000000000000000000..cdb0608a58a2f4db982d509786177b62533d4bfc --- /dev/null +++ b/log_w192.txt @@ -0,0 +1,180 @@ +launch +05:00:02 name=w192 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 139/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +05:00:08 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +05:00:09 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +05:00:18 params 1.08B +05:00:20 resumed step=189600 seen=24.99B round=811 +05:00:20 stream salt=189600 +05:02:06 blocks compiled +05:02:06 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +05:05:12 step 189601 loss=3.4349 lr=9.60e-06 tok=24.986B dt=187 wait=146 +05:05:35 step 189602 loss=3.2471 lr=9.60e-06 tok=24.987B dt=209 wait=146 +05:06:38 step 189620 loss=3.4063 lr=9.59e-06 tok=24.989B dt=272 wait=146 +05:07:43 step 189640 loss=3.3481 lr=9.57e-06 tok=24.991B dt=337 wait=146 +05:08:48 step 189660 loss=3.3505 lr=9.55e-06 tok=24.994B dt=402 wait=146 +05:09:03 global r=812 load 4.280 -> 4.278 +05:09:36 step 189760 loss=3.3932 lr=9.46e-06 tok=24.995B dt=450 wait=146 +05:10:39 step 189780 loss=3.2336 lr=9.44e-06 tok=24.998B dt=514 wait=147 +05:11:44 step 189800 loss=3.2769 lr=9.42e-06 tok=25.001B dt=578 wait=147 +05:12:48 step 189820 loss=3.2711 lr=9.40e-06 tok=25.003B dt=643 wait=147 +05:13:53 step 189840 loss=3.2371 lr=9.38e-06 tok=25.006B dt=707 wait=147 +05:14:58 step 189860 loss=3.1929 lr=9.36e-06 tok=25.009B dt=772 wait=147 +05:16:01 step 189880 loss=3.1140 lr=9.35e-06 tok=25.011B dt=836 wait=148 +05:17:06 step 189900 loss=3.1771 lr=9.33e-06 tok=25.014B dt=900 wait=148 +05:17:23 pushed ckpt_w192.pt,log_w192.txt +05:18:53 waited 91s for global r=813 +05:19:02 global r=813 load 4.280 -> 4.279 +05:20:09 step 189920 loss=3.2017 lr=9.31e-06 tok=25.016B dt=1083 wait=148 +05:21:13 step 189940 loss=3.0425 lr=9.29e-06 tok=25.019B dt=1147 wait=148 +05:22:17 step 189960 loss=3.1734 lr=9.27e-06 tok=25.022B dt=1212 wait=148 +05:23:21 step 189980 loss=3.1640 lr=9.25e-06 tok=25.024B dt=1276 wait=148 +05:24:25 step 190000 loss=3.1201 lr=9.24e-06 tok=25.027B dt=1339 wait=148 +05:25:30 step 190020 loss=3.2611 lr=9.22e-06 tok=25.030B dt=1404 wait=149 +05:26:34 step 190040 loss=3.1203 lr=9.20e-06 tok=25.032B dt=1468 wait=149 +05:27:22 pushed ckpt_w192.pt,log_w192.txt +05:28:23 waited 60s for global r=814 +05:28:29 global r=814 load 4.277 -> 4.279 +05:29:03 step 190060 loss=3.3426 lr=9.18e-06 tok=25.035B dt=1617 wait=149 +05:30:07 step 190080 loss=3.3097 lr=9.16e-06 tok=25.037B dt=1681 wait=149 +05:31:11 step 190100 loss=3.1616 lr=9.14e-06 tok=25.040B dt=1745 wait=149 +05:32:15 step 190120 loss=3.1601 lr=9.12e-06 tok=25.043B dt=1810 wait=149 +05:33:20 step 190140 loss=3.2054 lr=9.11e-06 tok=25.045B dt=1874 wait=150 +05:34:25 step 190160 loss=3.3778 lr=9.09e-06 tok=25.048B dt=1939 wait=150 +05:35:30 step 190180 loss=3.1974 lr=9.07e-06 tok=25.050B dt=2004 wait=150 +05:36:35 step 190200 loss=3.2461 lr=9.05e-06 tok=25.053B dt=2069 wait=150 +05:36:52 pushed ckpt_w192.pt,log_w192.txt +05:37:53 waited 61s for global r=815 +05:38:02 global r=815 load 4.278 -> 4.279 +05:39:08 step 190220 loss=3.0907 lr=9.03e-06 tok=25.056B dt=2223 wait=150 +05:40:13 step 190240 loss=2.9664 lr=9.01e-06 tok=25.058B dt=2288 wait=150 +05:41:18 step 190260 loss=3.2795 lr=9.00e-06 tok=25.061B dt=2352 wait=150 +05:42:22 step 190280 loss=3.1627 lr=8.98e-06 tok=25.064B dt=2416 wait=151 +05:43:26 step 190300 loss=3.1127 lr=8.96e-06 tok=25.066B dt=2480 wait=151 +05:44:30 step 190320 loss=3.2081 lr=8.94e-06 tok=25.069B dt=2544 wait=151 +05:45:34 step 190340 loss=3.1465 lr=8.92e-06 tok=25.071B dt=2608 wait=151 +05:46:19 pushed ckpt_w192.pt,log_w192.txt +05:47:20 waited 61s for global r=816 +05:47:29 global r=816 load 4.280 -> 4.280 +05:48:03 step 190360 loss=3.2204 lr=8.90e-06 tok=25.074B dt=2757 wait=151 +05:49:07 step 190380 loss=3.0398 lr=8.88e-06 tok=25.077B dt=2821 wait=151 +05:50:11 step 190400 loss=2.9866 lr=8.87e-06 tok=25.079B dt=2885 wait=152 +05:51:15 step 190420 loss=3.2081 lr=8.85e-06 tok=25.082B dt=2950 wait=152 +05:52:20 step 190440 loss=2.9114 lr=8.83e-06 tok=25.085B dt=3014 wait=152 +05:53:24 step 190460 loss=3.1389 lr=8.81e-06 tok=25.087B dt=3078 wait=152 +05:54:28 step 190480 loss=3.1442 lr=8.79e-06 tok=25.090B dt=3142 wait=152 +05:55:33 step 190500 loss=3.0520 lr=8.77e-06 tok=25.092B dt=3207 wait=152 +05:55:49 pushed ckpt_w192.pt,log_w192.txt +05:57:04 waited 76s for global r=817 +05:57:10 global r=817 load 4.277 -> 4.280 +05:58:15 step 190520 loss=3.1222 lr=8.76e-06 tok=25.095B dt=3369 wait=152 +05:59:20 step 190540 loss=3.1437 lr=8.74e-06 tok=25.098B dt=3434 wait=153 +06:00:24 step 190560 loss=3.1906 lr=8.72e-06 tok=25.100B dt=3499 wait=153 +06:01:29 step 190580 loss=3.1676 lr=8.70e-06 tok=25.103B dt=3563 wait=153 +06:02:47 step 190600 loss=3.1139 lr=8.68e-06 tok=25.106B dt=3642 wait=167 +06:03:52 step 190620 loss=3.3368 lr=8.66e-06 tok=25.108B dt=3706 wait=167 +06:04:56 step 190640 loss=3.1671 lr=8.64e-06 tok=25.111B dt=3771 wait=167 +06:05:41 pushed ckpt_w192.pt,log_w192.txt +06:06:41 waited 60s for global r=818 +06:06:47 global r=818 load 4.278 -> 4.279 +06:07:20 step 190660 loss=3.3193 lr=8.63e-06 tok=25.113B dt=3915 wait=167 +06:08:25 step 190680 loss=3.0739 lr=8.61e-06 tok=25.116B dt=3979 wait=168 +06:09:29 step 190700 loss=3.1391 lr=8.59e-06 tok=25.119B dt=4043 wait=168 +06:10:33 step 190720 loss=3.2267 lr=8.57e-06 tok=25.121B dt=4108 wait=168 +06:11:38 step 190740 loss=3.0205 lr=8.55e-06 tok=25.124B dt=4172 wait=168 +06:12:42 step 190760 loss=3.1283 lr=8.53e-06 tok=25.127B dt=4236 wait=168 +06:13:45 step 190780 loss=3.0551 lr=8.52e-06 tok=25.129B dt=4300 wait=168 +06:14:49 step 190800 loss=3.0600 lr=8.50e-06 tok=25.132B dt=4363 wait=169 +06:15:01 pushed ckpt_w192.pt,log_w192.txt +06:16:16 waited 76s for global r=819 +06:16:22 global r=819 load 4.280 -> 4.280 +06:17:28 step 190820 loss=2.9755 lr=8.48e-06 tok=25.134B dt=4522 wait=169 +06:18:32 step 190840 loss=3.0147 lr=8.46e-06 tok=25.137B dt=4586 wait=169 +06:19:36 step 190860 loss=3.1426 lr=8.44e-06 tok=25.140B dt=4651 wait=169 +06:20:41 step 190880 loss=3.1126 lr=8.42e-06 tok=25.142B dt=4715 wait=169 +06:21:46 step 190900 loss=3.2204 lr=8.40e-06 tok=25.145B dt=4780 wait=169 +06:22:50 step 190920 loss=3.2855 lr=8.39e-06 tok=25.147B dt=4845 wait=170 +06:23:55 step 190940 loss=3.1108 lr=8.37e-06 tok=25.150B dt=4909 wait=170 +06:24:40 pushed ckpt_w192.pt,log_w192.txt +06:25:40 waited 60s for global r=820 +06:25:48 global r=820 load 4.276 -> 4.279 +06:26:23 step 190960 loss=3.3142 lr=8.35e-06 tok=25.153B dt=5057 wait=170 +06:27:33 step 190980 loss=3.1456 lr=8.33e-06 tok=25.155B dt=5128 wait=170 +06:28:37 step 191000 loss=3.1709 lr=8.31e-06 tok=25.158B dt=5192 wait=170 +06:29:42 step 191020 loss=3.3655 lr=8.29e-06 tok=25.161B dt=5256 wait=170 +06:30:47 step 191040 loss=3.2359 lr=8.28e-06 tok=25.163B dt=5321 wait=170 +06:31:51 step 191060 loss=3.2273 lr=8.26e-06 tok=25.166B dt=5385 wait=171 +06:32:55 step 191080 loss=3.1861 lr=8.24e-06 tok=25.168B dt=5450 wait=171 +06:34:00 step 191100 loss=3.1582 lr=8.22e-06 tok=25.171B dt=5514 wait=171 +06:34:12 pushed ckpt_w192.pt,log_w192.txt +06:35:12 waited 61s for global r=821 +06:35:18 global r=821 load 4.276 -> 4.279 +06:36:24 step 191120 loss=3.0663 lr=8.20e-06 tok=25.174B dt=5658 wait=171 +06:37:28 step 191140 loss=3.2094 lr=8.18e-06 tok=25.176B dt=5722 wait=171 +06:38:33 step 191160 loss=3.2994 lr=8.16e-06 tok=25.179B dt=5787 wait=171 +06:39:38 step 191180 loss=3.1617 lr=8.15e-06 tok=25.182B dt=5852 wait=172 +06:40:42 step 191200 loss=3.1557 lr=8.13e-06 tok=25.184B dt=5916 wait=172 +06:41:45 step 191220 loss=3.2304 lr=8.11e-06 tok=25.187B dt=5979 wait=172 +06:42:48 step 191240 loss=3.0009 lr=8.09e-06 tok=25.189B dt=6043 wait=172 +06:43:34 pushed ckpt_w192.pt,log_w192.txt +06:44:49 waited 76s for global r=822 +06:44:55 global r=822 load 4.277 -> 4.278 +06:45:28 step 191260 loss=3.1077 lr=8.07e-06 tok=25.192B dt=6202 wait=172 +06:46:31 step 191280 loss=3.0235 lr=8.05e-06 tok=25.195B dt=6266 wait=172 +06:47:34 step 191300 loss=3.1279 lr=8.04e-06 tok=25.197B dt=6329 wait=172 +06:48:39 step 191320 loss=3.1625 lr=8.02e-06 tok=25.200B dt=6393 wait=173 +06:49:43 step 191340 loss=3.0954 lr=8.00e-06 tok=25.203B dt=6457 wait=173 +06:50:47 step 191360 loss=3.2118 lr=7.98e-06 tok=25.205B dt=6521 wait=173 +06:51:51 step 191380 loss=3.0700 lr=7.96e-06 tok=25.208B dt=6585 wait=173 +06:52:55 step 191400 loss=3.1864 lr=7.94e-06 tok=25.210B dt=6649 wait=173 +06:53:09 pushed ckpt_w192.pt,log_w192.txt +06:54:09 waited 61s for global r=823 +06:54:15 global r=823 load 4.278 -> 4.279 +06:55:21 step 191420 loss=3.1714 lr=7.92e-06 tok=25.213B dt=6796 wait=173 +06:56:25 step 191440 loss=3.1964 lr=7.91e-06 tok=25.216B dt=6860 wait=173 +06:57:30 step 191460 loss=3.2534 lr=7.89e-06 tok=25.218B dt=6925 wait=174 +06:58:36 step 191480 loss=2.9454 lr=7.87e-06 tok=25.221B dt=6990 wait=174 +06:59:41 step 191500 loss=3.0359 lr=7.85e-06 tok=25.223B dt=7055 wait=174 +07:00:46 step 191520 loss=3.0734 lr=7.83e-06 tok=25.226B dt=7121 wait=174 +07:01:51 step 191540 loss=3.1281 lr=7.81e-06 tok=25.229B dt=7185 wait=174 +07:02:37 pushed ckpt_w192.pt,log_w192.txt +07:03:52 waited 76s for global r=824 +07:03:58 global r=824 load 4.277 -> 4.279 +07:04:32 step 191560 loss=3.0493 lr=7.80e-06 tok=25.231B dt=7346 wait=174 +07:05:36 step 191580 loss=3.0143 lr=7.78e-06 tok=25.234B dt=7411 wait=175 +07:06:40 step 191600 loss=2.9941 lr=7.76e-06 tok=25.237B dt=7474 wait=175 +07:07:44 step 191620 loss=3.1014 lr=7.74e-06 tok=25.239B dt=7538 wait=175 +07:08:47 step 191640 loss=2.8600 lr=7.72e-06 tok=25.242B dt=7602 wait=175 +07:09:51 step 191660 loss=2.9737 lr=7.70e-06 tok=25.244B dt=7665 wait=175 +07:10:55 step 191680 loss=2.8495 lr=7.68e-06 tok=25.247B dt=7729 wait=175 +07:11:58 step 191700 loss=3.0104 lr=7.67e-06 tok=25.250B dt=7793 wait=175 +07:12:10 pushed ckpt_w192.pt,log_w192.txt +07:13:26 waited 76s for global r=825 +07:13:33 global r=825 load 4.278 -> 4.279 +07:14:38 step 191720 loss=2.9426 lr=7.65e-06 tok=25.252B dt=7952 wait=176 +07:15:41 step 191740 loss=3.0385 lr=7.63e-06 tok=25.255B dt=8015 wait=176 +07:16:44 step 191760 loss=3.0058 lr=7.61e-06 tok=25.258B dt=8078 wait=176 +07:17:47 step 191780 loss=3.0126 lr=7.59e-06 tok=25.260B dt=8141 wait=176 +07:18:51 step 191800 loss=2.9167 lr=7.57e-06 tok=25.263B dt=8206 wait=176 +07:19:55 step 191820 loss=3.0795 lr=7.55e-06 tok=25.265B dt=8269 wait=176 +07:20:59 step 191840 loss=2.9163 lr=7.54e-06 tok=25.268B dt=8333 wait=176 +07:21:43 pushed ckpt_w192.pt,log_w192.txt +07:22:44 waited 61s for global r=826 +07:22:52 global r=826 load 4.280 -> 4.280 +07:23:25 step 191860 loss=3.0453 lr=7.52e-06 tok=25.271B dt=8480 wait=177 +07:24:30 step 191880 loss=2.9628 lr=7.50e-06 tok=25.273B dt=8545 wait=177 +07:25:35 step 191900 loss=3.0873 lr=7.48e-06 tok=25.276B dt=8609 wait=177 +07:26:40 step 191920 loss=3.1664 lr=7.46e-06 tok=25.279B dt=8674 wait=177 +07:27:45 step 191940 loss=3.1560 lr=7.44e-06 tok=25.281B dt=8739 wait=177 +07:28:49 step 191960 loss=3.1224 lr=7.43e-06 tok=25.284B dt=8803 wait=177 +07:29:53 step 191980 loss=3.0299 lr=7.41e-06 tok=25.286B dt=8867 wait=177 +07:30:58 step 192000 loss=2.9881 lr=7.39e-06 tok=25.289B dt=8932 wait=178 +07:31:10 pushed ckpt_w192.pt,log_w192.txt +07:32:25 waited 76s for global r=827 +07:32:31 global r=827 load 4.278 -> 4.280 +07:33:37 step 192020 loss=3.0714 lr=7.37e-06 tok=25.292B dt=9091 wait=178 +07:34:41 step 192040 loss=3.1281 lr=7.35e-06 tok=25.294B dt=9156 wait=178 +07:35:47 step 192060 loss=3.1207 lr=7.33e-06 tok=25.297B dt=9221 wait=178 +07:36:52 step 192080 loss=3.1280 lr=7.31e-06 tok=25.300B dt=9286 wait=178 +07:37:58 step 192100 loss=3.1325 lr=7.30e-06 tok=25.302B dt=9352 wait=179 +07:39:02 step 192120 loss=3.2534 lr=7.28e-06 tok=25.305B dt=9416 wait=179 +07:40:08 step 192140 loss=2.8986 lr=7.26e-06 tok=25.307B dt=9482 wait=179 diff --git a/log_w193.txt b/log_w193.txt new file mode 100644 index 0000000000000000000000000000000000000000..6363ab0afc98bb50e2a77b067c5d358bbb116e70 --- /dev/null +++ b/log_w193.txt @@ -0,0 +1,187 @@ +launch +12:54:31 name=w193 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 140/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:54:37 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:54:37 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:54:47 params 1.08B +12:54:48 resumed step=196050 seen=25.85B round=854 +12:54:48 stream salt=196050 +12:56:55 blocks compiled +12:56:55 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:02:44 step 196051 loss=2.8474 lr=3.65e-06 tok=25.846B dt=349 wait=320 +13:03:00 step 196052 loss=3.1427 lr=3.65e-06 tok=25.846B dt=365 wait=320 +13:03:25 step 196060 loss=3.2715 lr=3.64e-06 tok=25.847B dt=390 wait=320 +13:04:29 step 196080 loss=3.1230 lr=3.62e-06 tok=25.850B dt=454 wait=320 +13:05:34 step 196100 loss=3.1319 lr=3.60e-06 tok=25.853B dt=519 wait=320 +13:06:38 step 196120 loss=3.1348 lr=3.58e-06 tok=25.855B dt=583 wait=320 +13:07:41 step 196140 loss=3.2296 lr=3.57e-06 tok=25.858B dt=646 wait=320 +13:08:45 step 196160 loss=3.1938 lr=3.55e-06 tok=25.861B dt=710 wait=320 +13:09:50 step 196180 loss=3.0690 lr=3.53e-06 tok=25.863B dt=774 wait=321 +13:10:53 step 196200 loss=3.1815 lr=3.51e-06 tok=25.866B dt=838 wait=321 +13:11:06 pushed ckpt_w193.pt,log_w193.txt +13:11:52 waited 45s for global r=855 +13:12:09 global r=855 load 4.280 -> 4.281 +13:13:15 step 196220 loss=3.0381 lr=3.49e-06 tok=25.868B dt=980 wait=321 +13:14:19 step 196240 loss=3.0422 lr=3.47e-06 tok=25.871B dt=1044 wait=321 +13:15:23 step 196260 loss=3.1009 lr=3.45e-06 tok=25.874B dt=1107 wait=321 +13:16:26 step 196280 loss=2.9753 lr=3.44e-06 tok=25.876B dt=1171 wait=321 +13:17:30 step 196300 loss=3.0155 lr=3.42e-06 tok=25.879B dt=1235 wait=321 +13:18:35 step 196320 loss=3.0826 lr=3.40e-06 tok=25.881B dt=1300 wait=322 +13:19:39 step 196340 loss=3.0442 lr=3.38e-06 tok=25.884B dt=1364 wait=322 +13:20:22 pushed ckpt_w193.pt,log_w193.txt +13:21:53 waited 91s for global r=856 +13:22:02 global r=856 load 4.281 -> 4.281 +13:22:37 step 196360 loss=2.3374 lr=3.36e-06 tok=25.887B dt=1542 wait=322 +13:23:43 step 196380 loss=2.7786 lr=3.34e-06 tok=25.889B dt=1608 wait=322 +13:24:47 step 196400 loss=3.0620 lr=3.33e-06 tok=25.892B dt=1672 wait=322 +13:25:51 step 196420 loss=3.1807 lr=3.31e-06 tok=25.895B dt=1736 wait=322 +13:26:55 step 196440 loss=3.0141 lr=3.29e-06 tok=25.897B dt=1800 wait=323 +13:27:58 step 196460 loss=3.0848 lr=3.27e-06 tok=25.900B dt=1863 wait=323 +13:29:03 step 196480 loss=3.0635 lr=3.25e-06 tok=25.902B dt=1928 wait=323 +13:30:07 step 196500 loss=3.0399 lr=3.23e-06 tok=25.905B dt=1992 wait=323 +13:30:18 pushed ckpt_w193.pt,log_w193.txt +13:31:34 waited 76s for global r=857 +13:31:42 global r=857 load 4.281 -> 4.281 +13:32:47 step 196520 loss=3.0611 lr=3.21e-06 tok=25.908B dt=2152 wait=323 +13:33:51 step 196540 loss=3.0124 lr=3.20e-06 tok=25.910B dt=2216 wait=323 +13:34:56 step 196560 loss=3.1180 lr=3.18e-06 tok=25.913B dt=2281 wait=323 +13:36:01 step 196580 loss=3.0673 lr=3.16e-06 tok=25.916B dt=2346 wait=324 +13:37:06 step 196600 loss=3.0454 lr=3.14e-06 tok=25.918B dt=2411 wait=324 +13:38:10 step 196620 loss=3.1169 lr=3.12e-06 tok=25.921B dt=2475 wait=324 +13:39:14 step 196640 loss=3.0149 lr=3.10e-06 tok=25.923B dt=2539 wait=324 +13:39:58 pushed ckpt_w193.pt,log_w193.txt +13:41:29 waited 91s for global r=858 +13:41:42 global r=858 load 4.281 -> 4.280 +13:42:15 step 196660 loss=3.1586 lr=3.09e-06 tok=25.926B dt=2720 wait=324 +13:43:19 step 196680 loss=3.0815 lr=3.07e-06 tok=25.929B dt=2784 wait=324 +13:44:24 step 196700 loss=3.0917 lr=3.05e-06 tok=25.931B dt=2849 wait=324 +13:45:27 step 196720 loss=3.1370 lr=3.03e-06 tok=25.934B dt=2912 wait=325 +13:46:31 step 196740 loss=3.0677 lr=3.01e-06 tok=25.937B dt=2976 wait=325 +13:47:36 step 196760 loss=3.0901 lr=2.99e-06 tok=25.939B dt=3040 wait=325 +13:48:39 step 196780 loss=3.0000 lr=2.97e-06 tok=25.942B dt=3104 wait=325 +13:49:43 step 196800 loss=3.0466 lr=2.96e-06 tok=25.944B dt=3168 wait=325 +13:49:57 pushed ckpt_w193.pt,log_w193.txt +13:51:27 waited 91s for global r=859 +13:51:38 global r=859 load 4.280 -> 4.281 +13:52:43 step 196820 loss=3.0420 lr=2.94e-06 tok=25.947B dt=3347 wait=325 +13:53:46 step 196840 loss=2.9742 lr=2.92e-06 tok=25.950B dt=3411 wait=326 +13:54:50 step 196860 loss=3.0113 lr=2.90e-06 tok=25.952B dt=3475 wait=326 +13:55:53 step 196880 loss=2.9975 lr=2.88e-06 tok=25.955B dt=3538 wait=326 +13:56:57 step 196900 loss=3.0025 lr=2.86e-06 tok=25.957B dt=3602 wait=326 +13:58:00 step 196920 loss=2.9724 lr=2.85e-06 tok=25.960B dt=3665 wait=326 +13:59:04 step 196940 loss=2.9090 lr=2.83e-06 tok=25.963B dt=3729 wait=326 +13:59:47 pushed ckpt_w193.pt,log_w193.txt +14:01:18 waited 91s for global r=860 +14:01:23 global r=860 load 4.280 -> 4.281 +14:01:56 step 196960 loss=3.0244 lr=2.81e-06 tok=25.965B dt=3901 wait=326 +14:03:00 step 196980 loss=2.9425 lr=2.79e-06 tok=25.968B dt=3965 wait=327 +14:04:04 step 197000 loss=2.9986 lr=2.77e-06 tok=25.971B dt=4029 wait=327 +14:05:08 step 197020 loss=3.0089 lr=2.75e-06 tok=25.973B dt=4093 wait=327 +14:06:12 step 197040 loss=2.9175 lr=2.73e-06 tok=25.976B dt=4157 wait=327 +14:07:16 step 197060 loss=3.0066 lr=2.72e-06 tok=25.978B dt=4221 wait=327 +14:08:20 step 197080 loss=2.9214 lr=2.70e-06 tok=25.981B dt=4285 wait=327 +14:09:24 step 197100 loss=2.9826 lr=2.68e-06 tok=25.984B dt=4349 wait=328 +14:09:39 pushed ckpt_w193.pt,log_w193.txt +14:11:10 waited 91s for global r=861 +14:11:17 global r=861 load 4.280 -> 4.280 +14:12:23 step 197120 loss=3.0057 lr=2.66e-06 tok=25.986B dt=4528 wait=328 +14:13:26 step 197140 loss=3.0809 lr=2.64e-06 tok=25.989B dt=4591 wait=328 +14:14:30 step 197160 loss=3.0801 lr=2.62e-06 tok=25.992B dt=4655 wait=328 +14:15:34 step 197180 loss=3.0495 lr=2.61e-06 tok=25.994B dt=4719 wait=328 +14:16:39 step 197200 loss=2.9704 lr=2.59e-06 tok=25.997B dt=4784 wait=328 +14:17:43 step 197220 loss=3.0346 lr=2.57e-06 tok=25.999B dt=4848 wait=328 +14:18:46 step 197240 loss=2.9573 lr=2.55e-06 tok=26.002B dt=4911 wait=329 +14:19:31 pushed ckpt_w193.pt,log_w193.txt +14:20:47 waited 76s for global r=862 +14:20:55 global r=862 load 4.280 -> 4.281 +14:21:28 step 197260 loss=2.9845 lr=2.53e-06 tok=26.005B dt=5073 wait=329 +14:22:32 step 197280 loss=3.0209 lr=2.51e-06 tok=26.007B dt=5137 wait=329 +14:23:35 step 197300 loss=2.8454 lr=2.49e-06 tok=26.010B dt=5200 wait=329 +14:24:39 step 197320 loss=2.9662 lr=2.48e-06 tok=26.013B dt=5264 wait=329 +14:25:44 step 197340 loss=2.7655 lr=2.46e-06 tok=26.015B dt=5329 wait=329 +14:26:47 step 197360 loss=3.0553 lr=2.44e-06 tok=26.018B dt=5392 wait=330 +14:27:50 step 197380 loss=2.8720 lr=2.42e-06 tok=26.020B dt=5455 wait=330 +14:28:53 step 197400 loss=2.9638 lr=2.40e-06 tok=26.023B dt=5518 wait=330 +14:29:03 pushed ckpt_w193.pt,log_w193.txt +14:30:34 waited 91s for global r=863 +14:30:40 global r=863 load 4.280 -> 4.280 +14:31:44 step 197420 loss=2.9035 lr=2.38e-06 tok=26.026B dt=5689 wait=330 +14:32:48 step 197440 loss=2.8833 lr=2.37e-06 tok=26.028B dt=5753 wait=330 +14:33:52 step 197460 loss=2.8978 lr=2.35e-06 tok=26.031B dt=5817 wait=330 +14:34:55 step 197480 loss=2.8322 lr=2.33e-06 tok=26.034B dt=5880 wait=331 +14:35:59 step 197500 loss=2.9523 lr=2.31e-06 tok=26.036B dt=5944 wait=331 +14:37:03 step 197520 loss=3.0068 lr=2.29e-06 tok=26.039B dt=6008 wait=331 +14:38:06 step 197540 loss=2.9792 lr=2.27e-06 tok=26.041B dt=6071 wait=331 +14:38:51 pushed ckpt_w193.pt,log_w193.txt +14:40:22 waited 91s for global r=864 +14:40:29 global r=864 load 4.280 -> 4.280 +14:41:02 step 197560 loss=3.0520 lr=2.25e-06 tok=26.044B dt=6247 wait=331 +14:42:05 step 197580 loss=2.8990 lr=2.24e-06 tok=26.047B dt=6310 wait=331 +14:43:08 step 197600 loss=2.9399 lr=2.22e-06 tok=26.049B dt=6373 wait=332 +14:44:11 step 197620 loss=3.1128 lr=2.20e-06 tok=26.052B dt=6436 wait=332 +14:45:15 step 197640 loss=2.8893 lr=2.18e-06 tok=26.054B dt=6500 wait=332 +14:46:19 step 197660 loss=3.0568 lr=2.16e-06 tok=26.057B dt=6563 wait=332 +14:47:22 step 197680 loss=3.0193 lr=2.14e-06 tok=26.060B dt=6627 wait=332 +14:48:26 step 197700 loss=3.0461 lr=2.12e-06 tok=26.062B dt=6691 wait=332 +14:48:37 pushed ckpt_w193.pt,log_w193.txt +14:50:08 waited 91s for global r=865 +14:50:13 global r=865 load 4.280 -> 4.281 +14:51:18 step 197720 loss=2.9674 lr=2.11e-06 tok=26.065B dt=6863 wait=332 +14:52:22 step 197740 loss=2.9811 lr=2.09e-06 tok=26.068B dt=6927 wait=333 +14:53:26 step 197760 loss=2.8611 lr=2.07e-06 tok=26.070B dt=6991 wait=333 +14:54:31 step 197780 loss=2.8496 lr=2.05e-06 tok=26.073B dt=7056 wait=333 +14:55:35 step 197800 loss=2.8105 lr=2.03e-06 tok=26.075B dt=7120 wait=333 +14:56:39 step 197820 loss=2.9158 lr=2.01e-06 tok=26.078B dt=7184 wait=333 +14:57:44 step 197840 loss=2.8299 lr=2.00e-06 tok=26.081B dt=7249 wait=333 +14:58:30 pushed ckpt_w193.pt,log_w193.txt +14:59:46 waited 76s for global r=866 +14:59:52 global r=866 load 4.280 -> 4.280 +15:00:26 step 197860 loss=2.9122 lr=1.98e-06 tok=26.083B dt=7411 wait=334 +15:01:30 step 197880 loss=2.8672 lr=1.96e-06 tok=26.086B dt=7475 wait=334 +15:02:35 step 197900 loss=2.8293 lr=1.94e-06 tok=26.089B dt=7540 wait=334 +15:03:40 step 197920 loss=2.9947 lr=1.92e-06 tok=26.091B dt=7604 wait=334 +15:04:44 step 197940 loss=2.8212 lr=1.90e-06 tok=26.094B dt=7669 wait=334 +15:05:49 step 197960 loss=2.9884 lr=1.88e-06 tok=26.096B dt=7734 wait=334 +15:06:53 step 197980 loss=2.7309 lr=1.87e-06 tok=26.099B dt=7798 wait=334 +15:07:58 step 198000 loss=2.9470 lr=1.85e-06 tok=26.102B dt=7863 wait=335 +15:08:11 pushed ckpt_w193.pt,log_w193.txt +15:09:26 waited 76s for global r=867 +15:09:32 global r=867 load 4.280 -> 4.280 +15:10:37 step 198020 loss=2.8875 lr=1.83e-06 tok=26.104B dt=8022 wait=335 +15:11:42 step 198040 loss=2.9347 lr=1.81e-06 tok=26.107B dt=8086 wait=335 +15:12:46 step 198060 loss=2.9420 lr=1.79e-06 tok=26.110B dt=8151 wait=335 +15:13:50 step 198080 loss=2.8181 lr=1.77e-06 tok=26.112B dt=8215 wait=335 +15:14:54 step 198100 loss=2.9664 lr=1.76e-06 tok=26.115B dt=8279 wait=335 +15:15:58 step 198120 loss=2.9269 lr=1.74e-06 tok=26.117B dt=8343 wait=336 +15:17:02 step 198140 loss=2.9384 lr=1.72e-06 tok=26.120B dt=8407 wait=336 +15:17:46 pushed ckpt_w193.pt,log_w193.txt +15:19:02 waited 76s for global r=868 +15:19:08 global r=868 load 4.281 -> 4.280 +15:19:42 step 198160 loss=2.9641 lr=1.70e-06 tok=26.123B dt=8567 wait=336 +15:20:46 step 198180 loss=2.7843 lr=1.68e-06 tok=26.125B dt=8631 wait=336 +15:21:50 step 198200 loss=2.7342 lr=1.66e-06 tok=26.128B dt=8695 wait=336 +15:22:55 step 198220 loss=2.9716 lr=1.64e-06 tok=26.131B dt=8760 wait=336 +15:24:00 step 198240 loss=2.8607 lr=1.63e-06 tok=26.133B dt=8825 wait=337 +15:25:04 step 198260 loss=2.9707 lr=1.61e-06 tok=26.136B dt=8889 wait=337 +15:26:09 step 198280 loss=2.9053 lr=1.59e-06 tok=26.138B dt=8954 wait=337 +15:27:14 step 198300 loss=3.0116 lr=1.57e-06 tok=26.141B dt=9019 wait=337 +15:27:29 pushed ckpt_w193.pt,log_w193.txt +15:28:45 waited 76s for global r=869 +15:28:52 global r=869 load 4.281 -> 4.281 +15:29:57 step 198320 loss=2.9720 lr=1.55e-06 tok=26.144B dt=9182 wait=337 +15:31:02 step 198340 loss=2.9304 lr=1.53e-06 tok=26.146B dt=9247 wait=337 +15:32:06 step 198360 loss=2.9705 lr=1.52e-06 tok=26.149B dt=9311 wait=337 +15:33:10 step 198380 loss=2.9519 lr=1.50e-06 tok=26.151B dt=9375 wait=338 +15:34:15 step 198400 loss=2.9996 lr=1.48e-06 tok=26.154B dt=9440 wait=338 +15:35:19 step 198420 loss=3.0722 lr=1.46e-06 tok=26.157B dt=9504 wait=338 +15:36:22 step 198440 loss=2.9513 lr=1.44e-06 tok=26.159B dt=9567 wait=338 +15:37:07 pushed ckpt_w193.pt,log_w193.txt +15:38:23 waited 76s for global r=870 +15:38:33 global r=870 load 4.280 -> 4.281 +15:39:06 step 198460 loss=3.0941 lr=1.42e-06 tok=26.162B dt=9731 wait=338 +15:40:09 step 198480 loss=2.9214 lr=1.40e-06 tok=26.165B dt=9794 wait=338 +15:41:14 step 198500 loss=2.8455 lr=1.39e-06 tok=26.167B dt=9858 wait=339 +15:42:18 step 198520 loss=2.8069 lr=1.37e-06 tok=26.170B dt=9923 wait=339 +15:43:22 step 198540 loss=2.7700 lr=1.35e-06 tok=26.172B dt=9986 wait=339 +15:44:25 step 198560 loss=3.0313 lr=1.33e-06 tok=26.175B dt=10050 wait=339 +15:45:29 step 198580 loss=2.9207 lr=1.31e-06 tok=26.178B dt=10114 wait=339 +15:46:32 step 198600 loss=2.9737 lr=1.29e-06 tok=26.180B dt=10177 wait=340 diff --git a/log_w194.txt b/log_w194.txt new file mode 100644 index 0000000000000000000000000000000000000000..80f78c149c09d361d0e14aedddeae539189b4dfc --- /dev/null +++ b/log_w194.txt @@ -0,0 +1,286 @@ +launch +12:56:06 name=w194 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 141/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:56:12 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:56:12 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:56:22 params 1.08B +12:56:23 resumed step=196050 seen=25.85B round=854 +12:56:23 stream salt=196050 +12:58:36 blocks compiled +12:58:36 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:04:04 step 196051 loss=3.0925 lr=3.65e-06 tok=25.846B dt=328 wait=298 +13:04:20 step 196052 loss=3.3680 lr=3.65e-06 tok=25.846B dt=344 wait=298 +13:04:45 step 196060 loss=3.4924 lr=3.64e-06 tok=25.847B dt=369 wait=298 +13:05:50 step 196080 loss=3.1878 lr=3.62e-06 tok=25.850B dt=434 wait=298 +13:06:55 step 196100 loss=3.2377 lr=3.60e-06 tok=25.853B dt=499 wait=298 +13:07:58 step 196120 loss=3.2392 lr=3.58e-06 tok=25.855B dt=563 wait=298 +13:09:02 step 196140 loss=3.4326 lr=3.57e-06 tok=25.858B dt=627 wait=298 +13:10:07 step 196160 loss=3.2246 lr=3.55e-06 tok=25.861B dt=692 wait=299 +13:11:12 step 196180 loss=3.2472 lr=3.53e-06 tok=25.863B dt=756 wait=299 +13:12:01 global r=855 load 4.281 -> 4.281 +13:13:06 step 196220 loss=3.2142 lr=3.49e-06 tok=25.867B dt=870 wait=299 +13:14:10 step 196240 loss=3.1051 lr=3.47e-06 tok=25.870B dt=934 wait=299 +13:15:14 step 196260 loss=3.2439 lr=3.45e-06 tok=25.872B dt=998 wait=299 +13:16:19 step 196280 loss=3.0698 lr=3.44e-06 tok=25.875B dt=1063 wait=299 +13:17:23 step 196300 loss=3.0968 lr=3.42e-06 tok=25.878B dt=1127 wait=300 +13:18:27 step 196320 loss=3.1053 lr=3.40e-06 tok=25.880B dt=1191 wait=300 +13:19:32 step 196340 loss=3.0641 lr=3.38e-06 tok=25.883B dt=1256 wait=300 +13:20:20 pushed ckpt_w194.pt,log_w194.txt +13:21:51 waited 91s for global r=856 +13:21:58 global r=856 load 4.281 -> 4.281 +13:22:32 step 196360 loss=3.2527 lr=3.36e-06 tok=25.885B dt=1437 wait=300 +13:23:37 step 196380 loss=2.9841 lr=3.34e-06 tok=25.888B dt=1501 wait=300 +13:24:41 step 196400 loss=3.0520 lr=3.33e-06 tok=25.891B dt=1565 wait=300 +13:25:45 step 196420 loss=3.0655 lr=3.31e-06 tok=25.893B dt=1629 wait=301 +13:26:50 step 196440 loss=2.5961 lr=3.29e-06 tok=25.896B dt=1694 wait=301 +13:27:53 step 196460 loss=3.1008 lr=3.27e-06 tok=25.899B dt=1757 wait=301 +13:28:57 step 196480 loss=3.0623 lr=3.25e-06 tok=25.901B dt=1822 wait=301 +13:30:02 step 196500 loss=3.0063 lr=3.23e-06 tok=25.904B dt=1886 wait=301 +13:30:14 pushed ckpt_w194.pt,log_w194.txt +13:31:44 waited 91s for global r=857 +13:31:51 global r=857 load 4.281 -> 4.281 +13:32:56 step 196520 loss=3.0085 lr=3.21e-06 tok=25.906B dt=2060 wait=301 +13:34:00 step 196540 loss=3.1485 lr=3.20e-06 tok=25.909B dt=2124 wait=301 +13:35:04 step 196560 loss=3.0640 lr=3.18e-06 tok=25.912B dt=2188 wait=302 +13:36:08 step 196580 loss=3.1112 lr=3.16e-06 tok=25.914B dt=2252 wait=302 +13:37:11 step 196600 loss=3.1399 lr=3.14e-06 tok=25.917B dt=2316 wait=302 +13:38:16 step 196620 loss=3.2360 lr=3.12e-06 tok=25.919B dt=2380 wait=302 +13:39:19 step 196640 loss=3.1068 lr=3.10e-06 tok=25.922B dt=2443 wait=302 +13:40:05 pushed ckpt_w194.pt,log_w194.txt +13:41:21 waited 76s for global r=858 +13:41:30 global r=858 load 4.280 -> 4.280 +13:42:04 step 196660 loss=3.2966 lr=3.09e-06 tok=25.925B dt=2608 wait=302 +13:43:08 step 196680 loss=3.2340 lr=3.07e-06 tok=25.927B dt=2672 wait=302 +13:44:12 step 196700 loss=3.2028 lr=3.05e-06 tok=25.930B dt=2737 wait=303 +13:45:16 step 196720 loss=3.1859 lr=3.03e-06 tok=25.933B dt=2800 wait=303 +13:46:20 step 196740 loss=3.1887 lr=3.01e-06 tok=25.935B dt=2864 wait=303 +13:47:23 step 196760 loss=3.2135 lr=2.99e-06 tok=25.938B dt=2927 wait=303 +13:48:28 step 196780 loss=3.0963 lr=2.97e-06 tok=25.940B dt=2992 wait=303 +13:49:33 step 196800 loss=2.9992 lr=2.96e-06 tok=25.943B dt=3057 wait=303 +13:49:45 pushed ckpt_w194.pt,log_w194.txt +13:51:16 waited 91s for global r=859 +13:51:29 global r=859 load 4.280 -> 4.281 +13:52:34 step 196820 loss=3.1064 lr=2.94e-06 tok=25.946B dt=3238 wait=303 +13:53:38 step 196840 loss=3.0840 lr=2.92e-06 tok=25.948B dt=3302 wait=304 +13:54:42 step 196860 loss=3.0668 lr=2.90e-06 tok=25.951B dt=3366 wait=304 +13:55:47 step 196880 loss=2.9949 lr=2.88e-06 tok=25.954B dt=3431 wait=304 +13:56:51 step 196900 loss=3.0442 lr=2.86e-06 tok=25.956B dt=3495 wait=304 +13:57:55 step 196920 loss=2.9938 lr=2.85e-06 tok=25.959B dt=3559 wait=304 +13:58:59 step 196940 loss=2.9955 lr=2.83e-06 tok=25.961B dt=3623 wait=304 +13:59:45 pushed ckpt_w194.pt,log_w194.txt +14:01:15 waited 91s for global r=860 +14:01:21 global r=860 load 4.280 -> 4.281 +14:01:55 step 196960 loss=2.7420 lr=2.81e-06 tok=25.964B dt=3799 wait=305 +14:02:59 step 196980 loss=2.8570 lr=2.79e-06 tok=25.967B dt=3863 wait=305 +14:04:03 step 197000 loss=3.0069 lr=2.77e-06 tok=25.969B dt=3928 wait=305 +14:05:09 step 197020 loss=2.9997 lr=2.75e-06 tok=25.972B dt=3993 wait=305 +14:06:13 step 197040 loss=3.1268 lr=2.73e-06 tok=25.975B dt=4057 wait=305 +14:07:18 step 197060 loss=3.0445 lr=2.72e-06 tok=25.977B dt=4122 wait=305 +14:08:22 step 197080 loss=3.0149 lr=2.70e-06 tok=25.980B dt=4187 wait=306 +14:09:27 step 197100 loss=2.9944 lr=2.68e-06 tok=25.982B dt=4252 wait=306 +14:09:41 pushed ckpt_w194.pt,log_w194.txt +14:11:12 waited 91s for global r=861 +14:11:19 global r=861 load 4.280 -> 4.280 +14:12:25 step 197120 loss=3.0093 lr=2.66e-06 tok=25.985B dt=4429 wait=306 +14:13:28 step 197140 loss=2.9504 lr=2.64e-06 tok=25.988B dt=4492 wait=306 +14:14:32 step 197160 loss=2.8522 lr=2.62e-06 tok=25.990B dt=4557 wait=306 +14:15:37 step 197180 loss=2.9028 lr=2.61e-06 tok=25.993B dt=4621 wait=306 +14:16:41 step 197200 loss=2.8941 lr=2.59e-06 tok=25.996B dt=4685 wait=307 +14:17:45 step 197220 loss=3.0053 lr=2.57e-06 tok=25.998B dt=4749 wait=307 +14:18:49 step 197240 loss=2.9279 lr=2.55e-06 tok=26.001B dt=4813 wait=307 +14:19:35 pushed ckpt_w194.pt,log_w194.txt +14:20:51 waited 76s for global r=862 +14:20:59 global r=862 load 4.281 -> 4.281 +14:21:32 step 197260 loss=2.9393 lr=2.53e-06 tok=26.003B dt=4976 wait=307 +14:22:37 step 197280 loss=2.8378 lr=2.51e-06 tok=26.006B dt=5041 wait=307 +14:23:41 step 197300 loss=2.6280 lr=2.49e-06 tok=26.009B dt=5105 wait=307 +14:24:45 step 197320 loss=2.8685 lr=2.48e-06 tok=26.011B dt=5170 wait=307 +14:25:50 step 197340 loss=2.7876 lr=2.46e-06 tok=26.014B dt=5234 wait=308 +14:26:54 step 197360 loss=2.9374 lr=2.44e-06 tok=26.016B dt=5298 wait=308 +14:27:57 step 197380 loss=2.8608 lr=2.42e-06 tok=26.019B dt=5361 wait=308 +14:29:00 step 197400 loss=2.8061 lr=2.40e-06 tok=26.022B dt=5424 wait=308 +14:29:14 pushed ckpt_w194.pt,log_w194.txt +14:30:30 waited 76s for global r=863 +14:30:38 global r=863 load 4.281 -> 4.280 +14:31:43 step 197420 loss=2.8828 lr=2.38e-06 tok=26.024B dt=5587 wait=308 +14:32:46 step 197440 loss=2.8899 lr=2.37e-06 tok=26.027B dt=5651 wait=308 +14:33:51 step 197460 loss=2.8763 lr=2.35e-06 tok=26.030B dt=5715 wait=308 +14:34:54 step 197480 loss=2.8268 lr=2.33e-06 tok=26.032B dt=5778 wait=309 +14:35:58 step 197500 loss=2.7567 lr=2.31e-06 tok=26.035B dt=5842 wait=309 +14:37:02 step 197520 loss=2.9989 lr=2.29e-06 tok=26.037B dt=5906 wait=309 +14:38:06 step 197540 loss=2.8285 lr=2.27e-06 tok=26.040B dt=5971 wait=309 +14:38:50 pushed ckpt_w194.pt,log_w194.txt +14:40:21 waited 91s for global r=864 +14:40:28 global r=864 load 4.280 -> 4.280 +14:41:02 step 197560 loss=3.0197 lr=2.25e-06 tok=26.043B dt=6146 wait=309 +14:42:06 step 197580 loss=2.9283 lr=2.24e-06 tok=26.045B dt=6210 wait=309 +14:43:10 step 197600 loss=2.9498 lr=2.22e-06 tok=26.048B dt=6274 wait=309 +14:44:13 step 197620 loss=3.0642 lr=2.20e-06 tok=26.051B dt=6338 wait=310 +14:45:18 step 197640 loss=2.8519 lr=2.18e-06 tok=26.053B dt=6402 wait=310 +14:46:22 step 197660 loss=3.1248 lr=2.16e-06 tok=26.056B dt=6466 wait=310 +14:47:26 step 197680 loss=3.0566 lr=2.14e-06 tok=26.058B dt=6530 wait=310 +14:48:30 step 197700 loss=2.9583 lr=2.12e-06 tok=26.061B dt=6594 wait=310 +14:48:43 pushed ckpt_w194.pt,log_w194.txt +14:50:13 waited 91s for global r=865 +14:50:21 global r=865 load 4.281 -> 4.281 +14:51:25 step 197720 loss=3.0396 lr=2.11e-06 tok=26.064B dt=6769 wait=310 +14:52:29 step 197740 loss=2.9171 lr=2.09e-06 tok=26.066B dt=6833 wait=310 +14:53:33 step 197760 loss=2.9032 lr=2.07e-06 tok=26.069B dt=6897 wait=311 +14:54:37 step 197780 loss=2.9646 lr=2.05e-06 tok=26.072B dt=6961 wait=311 +14:55:41 step 197800 loss=2.9164 lr=2.03e-06 tok=26.074B dt=7025 wait=311 +14:56:44 step 197820 loss=2.9671 lr=2.01e-06 tok=26.077B dt=7088 wait=311 +14:57:48 step 197840 loss=2.8627 lr=2.00e-06 tok=26.079B dt=7152 wait=311 +14:58:33 pushed ckpt_w194.pt,log_w194.txt +14:59:48 waited 76s for global r=866 +14:59:55 global r=866 load 4.280 -> 4.280 +15:00:29 step 197860 loss=2.9158 lr=1.98e-06 tok=26.082B dt=7313 wait=311 +15:01:33 step 197880 loss=2.7936 lr=1.96e-06 tok=26.085B dt=7377 wait=311 +15:02:36 step 197900 loss=2.8834 lr=1.94e-06 tok=26.087B dt=7440 wait=312 +15:03:39 step 197920 loss=2.8662 lr=1.92e-06 tok=26.090B dt=7503 wait=312 +15:04:42 step 197940 loss=2.7840 lr=1.90e-06 tok=26.093B dt=7566 wait=312 +15:05:46 step 197960 loss=2.6957 lr=1.88e-06 tok=26.095B dt=7630 wait=312 +15:06:48 step 197980 loss=2.8306 lr=1.87e-06 tok=26.098B dt=7692 wait=312 +15:07:51 step 198000 loss=2.8372 lr=1.85e-06 tok=26.100B dt=7755 wait=312 +15:08:02 pushed ckpt_w194.pt,log_w194.txt +15:09:18 waited 76s for global r=867 +15:09:24 global r=867 load 4.280 -> 4.280 +15:10:29 step 198020 loss=2.8523 lr=1.83e-06 tok=26.103B dt=7913 wait=313 +15:11:32 step 198040 loss=3.0105 lr=1.81e-06 tok=26.106B dt=7976 wait=313 +15:12:35 step 198060 loss=2.9828 lr=1.79e-06 tok=26.108B dt=8039 wait=313 +15:13:38 step 198080 loss=2.8802 lr=1.77e-06 tok=26.111B dt=8102 wait=313 +15:14:41 step 198100 loss=2.8992 lr=1.76e-06 tok=26.113B dt=8166 wait=313 +15:15:45 step 198120 loss=3.0060 lr=1.74e-06 tok=26.116B dt=8229 wait=313 +15:16:48 step 198140 loss=2.9748 lr=1.72e-06 tok=26.119B dt=8292 wait=314 +15:17:32 pushed ckpt_w194.pt,log_w194.txt +15:19:02 waited 91s for global r=868 +15:19:09 global r=868 load 4.280 -> 4.280 +15:19:42 step 198160 loss=2.9874 lr=1.70e-06 tok=26.121B dt=8466 wait=314 +15:20:46 step 198180 loss=2.9153 lr=1.68e-06 tok=26.124B dt=8530 wait=314 +15:21:50 step 198200 loss=2.9423 lr=1.66e-06 tok=26.127B dt=8594 wait=314 +15:22:53 step 198220 loss=2.9005 lr=1.64e-06 tok=26.129B dt=8658 wait=314 +15:23:57 step 198240 loss=2.8269 lr=1.63e-06 tok=26.132B dt=8721 wait=314 +15:25:01 step 198260 loss=2.9071 lr=1.61e-06 tok=26.134B dt=8785 wait=314 +15:26:04 step 198280 loss=2.8002 lr=1.59e-06 tok=26.137B dt=8848 wait=315 +15:27:08 step 198300 loss=2.8615 lr=1.57e-06 tok=26.140B dt=8912 wait=315 +15:27:22 pushed ckpt_w194.pt,log_w194.txt +15:28:37 waited 76s for global r=869 +15:28:44 global r=869 load 4.281 -> 4.281 +15:29:48 step 198320 loss=2.8499 lr=1.55e-06 tok=26.142B dt=9072 wait=315 +15:30:52 step 198340 loss=2.8833 lr=1.53e-06 tok=26.145B dt=9136 wait=315 +15:31:55 step 198360 loss=2.8993 lr=1.52e-06 tok=26.148B dt=9199 wait=315 +15:32:59 step 198380 loss=2.8235 lr=1.50e-06 tok=26.150B dt=9263 wait=316 +15:34:02 step 198400 loss=2.6372 lr=1.48e-06 tok=26.153B dt=9326 wait=316 +15:35:05 step 198420 loss=3.0178 lr=1.46e-06 tok=26.155B dt=9390 wait=316 +15:36:09 step 198440 loss=2.8736 lr=1.44e-06 tok=26.158B dt=9453 wait=316 +15:36:53 pushed ckpt_w194.pt,log_w194.txt +15:38:24 waited 91s for global r=870 +15:38:34 global r=870 load 4.280 -> 4.281 +15:39:08 step 198460 loss=3.0147 lr=1.42e-06 tok=26.161B dt=9632 wait=316 +15:40:11 step 198480 loss=2.7337 lr=1.40e-06 tok=26.163B dt=9695 wait=316 +15:41:14 step 198500 loss=2.9915 lr=1.39e-06 tok=26.166B dt=9758 wait=317 +15:42:18 step 198520 loss=2.9431 lr=1.37e-06 tok=26.169B dt=9823 wait=317 +15:43:22 step 198540 loss=2.7363 lr=1.35e-06 tok=26.171B dt=9886 wait=317 +15:44:25 step 198560 loss=2.8749 lr=1.33e-06 tok=26.174B dt=9949 wait=317 +15:45:29 step 198580 loss=2.9425 lr=1.31e-06 tok=26.176B dt=10013 wait=317 +15:46:34 step 198600 loss=2.7593 lr=1.29e-06 tok=26.179B dt=10078 wait=317 +15:46:45 pushed ckpt_w194.pt,log_w194.txt +15:48:01 waited 76s for global r=871 +15:48:07 global r=871 load 4.280 -> 4.281 +15:49:12 step 198620 loss=2.9000 lr=1.28e-06 tok=26.182B dt=10236 wait=318 +15:50:16 step 198640 loss=2.8443 lr=1.26e-06 tok=26.184B dt=10300 wait=318 +15:51:19 step 198660 loss=2.8715 lr=1.24e-06 tok=26.187B dt=10363 wait=318 +15:52:23 step 198680 loss=2.9159 lr=1.22e-06 tok=26.189B dt=10427 wait=318 +15:53:27 step 198700 loss=2.8313 lr=1.20e-06 tok=26.192B dt=10491 wait=318 +15:54:30 step 198720 loss=2.9423 lr=1.18e-06 tok=26.195B dt=10555 wait=318 +15:55:34 step 198740 loss=2.8022 lr=1.16e-06 tok=26.197B dt=10618 wait=319 +15:56:18 pushed ckpt_w194.pt,log_w194.txt +15:59:52 step 198760 loss=2.9591 lr=1.15e-06 tok=26.200B dt=10876 wait=319 +16:00:01 global r=872 load 4.281 -> 4.281 +16:00:33 step 198760 loss=2.8925 lr=1.15e-06 tok=26.201B dt=10917 wait=319 +16:01:37 step 198780 loss=2.8168 lr=1.13e-06 tok=26.204B dt=10981 wait=319 +16:02:42 step 198800 loss=2.8177 lr=1.11e-06 tok=26.207B dt=11046 wait=319 +16:03:46 step 198820 loss=2.8840 lr=1.09e-06 tok=26.209B dt=11110 wait=319 +16:04:49 step 198840 loss=2.7192 lr=1.07e-06 tok=26.212B dt=11174 wait=319 +16:05:54 step 198860 loss=2.8800 lr=1.05e-06 tok=26.214B dt=11238 wait=320 +16:06:57 step 198880 loss=2.7778 lr=1.04e-06 tok=26.217B dt=11302 wait=320 +16:08:01 step 198900 loss=2.8129 lr=1.02e-06 tok=26.220B dt=11365 wait=320 +16:08:13 pushed ckpt_w194.pt,log_w194.txt +16:11:58 global r=873 load 4.281 -> 4.281 +16:13:01 step 198920 loss=2.8003 lr=9.98e-07 tok=26.224B dt=11665 wait=320 +16:14:05 step 198940 loss=2.8657 lr=9.80e-07 tok=26.226B dt=11729 wait=320 +16:15:19 step 198960 loss=2.6071 lr=9.61e-07 tok=26.229B dt=11803 wait=321 +16:16:23 step 198980 loss=2.8200 lr=9.43e-07 tok=26.231B dt=11867 wait=321 +16:17:27 step 199000 loss=2.7337 lr=9.24e-07 tok=26.234B dt=11931 wait=321 +16:18:30 step 199020 loss=2.7233 lr=9.06e-07 tok=26.237B dt=11995 wait=321 +16:19:34 step 199040 loss=2.7796 lr=8.87e-07 tok=26.239B dt=12059 wait=321 +16:20:18 pushed ckpt_w194.pt,log_w194.txt +16:23:54 step 199060 loss=2.6367 lr=8.69e-07 tok=26.242B dt=12318 wait=321 +16:24:01 global r=874 load 4.280 -> 4.280 +16:24:32 step 199060 loss=2.7673 lr=8.69e-07 tok=26.243B dt=12356 wait=322 +16:25:36 step 199080 loss=2.7956 lr=8.51e-07 tok=26.246B dt=12420 wait=322 +16:26:39 step 199100 loss=2.7842 lr=8.32e-07 tok=26.248B dt=12484 wait=322 +16:27:43 step 199120 loss=2.8518 lr=8.14e-07 tok=26.251B dt=12548 wait=322 +16:28:46 step 199140 loss=2.9091 lr=7.95e-07 tok=26.254B dt=12611 wait=322 +16:29:50 step 199160 loss=2.9103 lr=7.77e-07 tok=26.256B dt=12674 wait=322 +16:30:55 step 199180 loss=2.9542 lr=7.58e-07 tok=26.259B dt=12739 wait=322 +16:32:01 step 199200 loss=2.8892 lr=7.40e-07 tok=26.262B dt=12805 wait=322 +16:32:13 pushed ckpt_w194.pt,log_w194.txt +16:33:29 waited 76s for global r=875 +16:33:39 global r=875 load 4.280 -> 4.281 +16:34:46 step 199220 loss=2.9433 lr=7.21e-07 tok=26.264B dt=12970 wait=323 +16:35:50 step 199240 loss=3.0566 lr=7.03e-07 tok=26.267B dt=13034 wait=323 +16:36:56 step 199260 loss=2.9457 lr=6.84e-07 tok=26.269B dt=13100 wait=323 +16:37:59 step 199280 loss=2.9368 lr=6.66e-07 tok=26.272B dt=13164 wait=323 +16:39:04 step 199300 loss=2.9817 lr=6.47e-07 tok=26.275B dt=13228 wait=323 +16:40:09 step 199320 loss=3.0370 lr=6.29e-07 tok=26.277B dt=13293 wait=324 +16:41:13 step 199340 loss=2.9405 lr=6.10e-07 tok=26.280B dt=13357 wait=324 +16:41:59 pushed ckpt_w194.pt,log_w194.txt +16:43:30 waited 91s for global r=876 +16:43:39 global r=876 load 4.281 -> 4.280 +16:44:13 step 199360 loss=2.9698 lr=5.92e-07 tok=26.283B dt=13537 wait=324 +16:45:16 step 199380 loss=3.0284 lr=5.73e-07 tok=26.285B dt=13600 wait=324 +16:46:20 step 199400 loss=2.9564 lr=5.55e-07 tok=26.288B dt=13664 wait=324 +16:47:24 step 199420 loss=3.0907 lr=5.37e-07 tok=26.290B dt=13728 wait=324 +16:48:28 step 199440 loss=2.9009 lr=5.18e-07 tok=26.293B dt=13792 wait=324 +16:49:32 step 199460 loss=3.0402 lr=5.00e-07 tok=26.296B dt=13857 wait=325 +16:50:36 step 199480 loss=2.9874 lr=4.81e-07 tok=26.298B dt=13920 wait=325 +16:51:40 step 199500 loss=3.0394 lr=4.63e-07 tok=26.301B dt=13984 wait=325 +16:51:54 pushed ckpt_w194.pt,log_w194.txt +16:55:38 global r=877 load 4.280 -> 4.281 +16:56:42 step 199520 loss=2.9398 lr=4.44e-07 tok=26.305B dt=14286 wait=325 +16:57:46 step 199540 loss=2.9146 lr=4.26e-07 tok=26.307B dt=14350 wait=325 +16:58:50 step 199560 loss=2.8328 lr=4.07e-07 tok=26.310B dt=14414 wait=325 +16:59:54 step 199580 loss=2.7247 lr=3.89e-07 tok=26.313B dt=14478 wait=326 +17:00:57 step 199600 loss=2.9072 lr=3.70e-07 tok=26.315B dt=14541 wait=326 +17:02:00 step 199620 loss=3.0294 lr=3.52e-07 tok=26.318B dt=14605 wait=326 +17:03:04 step 199640 loss=2.8797 lr=3.33e-07 tok=26.321B dt=14668 wait=326 +17:03:51 pushed ckpt_w194.pt,log_w194.txt +17:06:53 waited 182s for global r=878 +17:07:02 global r=878 load 4.281 -> 4.280 +17:07:36 step 199660 loss=2.9100 lr=3.15e-07 tok=26.323B dt=14940 wait=326 +17:08:39 step 199680 loss=2.8541 lr=2.96e-07 tok=26.326B dt=15004 wait=326 +17:09:43 step 199700 loss=2.9170 lr=2.78e-07 tok=26.328B dt=15067 wait=326 +17:10:47 step 199720 loss=3.1110 lr=2.59e-07 tok=26.331B dt=15131 wait=327 +17:11:51 step 199740 loss=2.9198 lr=2.41e-07 tok=26.334B dt=15195 wait=327 +17:12:55 step 199760 loss=2.9193 lr=2.23e-07 tok=26.336B dt=15259 wait=327 +17:13:59 step 199780 loss=2.8654 lr=2.04e-07 tok=26.339B dt=15323 wait=327 +17:15:02 step 199800 loss=3.0108 lr=1.86e-07 tok=26.342B dt=15387 wait=327 +17:15:16 pushed ckpt_w194.pt,log_w194.txt +17:19:06 global r=879 load 4.280 -> 4.280 +17:20:10 step 199820 loss=2.9505 lr=1.67e-07 tok=26.345B dt=15694 wait=327 +17:21:14 step 199840 loss=2.9639 lr=1.49e-07 tok=26.348B dt=15758 wait=327 +17:22:18 step 199860 loss=2.9840 lr=1.30e-07 tok=26.351B dt=15822 wait=328 +17:23:23 step 199880 loss=2.8831 lr=1.12e-07 tok=26.353B dt=15887 wait=328 +17:24:28 step 199900 loss=2.8799 lr=9.33e-08 tok=26.356B dt=15952 wait=328 +17:25:32 step 199920 loss=3.1182 lr=7.48e-08 tok=26.359B dt=16016 wait=328 +17:26:36 step 199940 loss=2.8201 lr=5.63e-08 tok=26.361B dt=16080 wait=328 +17:27:34 pushed ckpt_w194.pt,log_w194.txt +17:30:05 waited 151s for global r=880 +17:30:16 global r=880 load 4.280 -> 4.280 +17:30:50 step 199960 loss=2.9530 lr=3.79e-08 tok=26.364B dt=16334 wait=329 +17:31:53 step 199980 loss=2.8355 lr=1.94e-08 tok=26.366B dt=16397 wait=329 +17:32:57 step 200000 loss=2.9328 lr=9.23e-10 tok=26.369B dt=16461 wait=329 +17:34:01 step 200020 loss=2.9276 lr=0.00e+00 tok=26.372B dt=16525 wait=329 +17:35:06 step 200040 loss=2.7843 lr=0.00e+00 tok=26.374B dt=16590 wait=329 +17:36:10 step 200060 loss=2.9737 lr=0.00e+00 tok=26.377B dt=16654 wait=329 +17:37:13 step 200080 loss=2.8731 lr=0.00e+00 tok=26.380B dt=16717 wait=329 +17:38:18 step 200100 loss=2.9200 lr=0.00e+00 tok=26.382B dt=16782 wait=330 diff --git a/log_w195.txt b/log_w195.txt new file mode 100644 index 0000000000000000000000000000000000000000..aa5f5ef8eb3616d37f1d3981e89ec432e27ace80 --- /dev/null +++ b/log_w195.txt @@ -0,0 +1,184 @@ +launch +12:57:40 name=w195 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 142/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:57:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:57:46 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:57:55 params 1.08B +12:57:57 resumed step=196050 seen=25.85B round=854 +12:57:57 stream salt=196050 +13:00:08 blocks compiled +13:00:08 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:04:43 step 196051 loss=2.6286 lr=3.65e-06 tok=25.846B dt=274 wait=247 +13:04:59 step 196052 loss=2.6066 lr=3.65e-06 tok=25.846B dt=290 wait=247 +13:05:24 step 196060 loss=3.2656 lr=3.64e-06 tok=25.847B dt=316 wait=247 +13:06:29 step 196080 loss=2.9618 lr=3.62e-06 tok=25.850B dt=381 wait=247 +13:07:34 step 196100 loss=3.0958 lr=3.60e-06 tok=25.853B dt=446 wait=247 +13:08:38 step 196120 loss=3.1244 lr=3.58e-06 tok=25.855B dt=509 wait=248 +13:09:42 step 196140 loss=3.0692 lr=3.57e-06 tok=25.858B dt=574 wait=248 +13:10:47 step 196160 loss=3.2166 lr=3.55e-06 tok=25.861B dt=639 wait=248 +13:11:52 step 196180 loss=3.1412 lr=3.53e-06 tok=25.863B dt=703 wait=248 +13:12:06 global r=855 load 4.281 -> 4.281 +13:13:10 step 196220 loss=3.0981 lr=3.49e-06 tok=25.866B dt=781 wait=248 +13:14:15 step 196240 loss=3.1661 lr=3.47e-06 tok=25.868B dt=846 wait=248 +13:15:20 step 196260 loss=3.2171 lr=3.45e-06 tok=25.871B dt=911 wait=249 +13:16:24 step 196280 loss=3.1116 lr=3.44e-06 tok=25.874B dt=976 wait=249 +13:17:29 step 196300 loss=3.0986 lr=3.42e-06 tok=25.876B dt=1041 wait=249 +13:18:34 step 196320 loss=3.1686 lr=3.40e-06 tok=25.879B dt=1106 wait=249 +13:19:39 step 196340 loss=3.0201 lr=3.38e-06 tok=25.881B dt=1170 wait=249 +13:20:26 pushed ckpt_w195.pt,log_w195.txt +13:21:57 waited 91s for global r=856 +13:22:06 global r=856 load 4.281 -> 4.281 +13:22:41 step 196360 loss=3.1964 lr=3.36e-06 tok=25.884B dt=1352 wait=249 +13:23:45 step 196380 loss=3.1284 lr=3.34e-06 tok=25.887B dt=1416 wait=249 +13:24:49 step 196400 loss=3.1142 lr=3.33e-06 tok=25.889B dt=1480 wait=250 +13:25:53 step 196420 loss=3.1423 lr=3.31e-06 tok=25.892B dt=1545 wait=250 +13:26:58 step 196440 loss=3.0747 lr=3.29e-06 tok=25.895B dt=1609 wait=250 +13:28:01 step 196460 loss=3.2278 lr=3.27e-06 tok=25.897B dt=1673 wait=250 +13:29:04 step 196480 loss=3.1309 lr=3.25e-06 tok=25.900B dt=1736 wait=250 +13:30:10 step 196500 loss=2.9545 lr=3.23e-06 tok=25.902B dt=1802 wait=250 +13:30:22 pushed ckpt_w195.pt,log_w195.txt +13:31:38 waited 76s for global r=857 +13:31:44 global r=857 load 4.281 -> 4.281 +13:32:49 step 196520 loss=3.0099 lr=3.21e-06 tok=25.905B dt=1961 wait=251 +13:33:53 step 196540 loss=3.1134 lr=3.20e-06 tok=25.908B dt=2024 wait=251 +13:34:57 step 196560 loss=3.0809 lr=3.18e-06 tok=25.910B dt=2089 wait=251 +13:36:02 step 196580 loss=3.0328 lr=3.16e-06 tok=25.913B dt=2153 wait=251 +13:37:07 step 196600 loss=2.8443 lr=3.14e-06 tok=25.916B dt=2219 wait=251 +13:38:12 step 196620 loss=3.2403 lr=3.12e-06 tok=25.918B dt=2283 wait=251 +13:39:17 step 196640 loss=3.1230 lr=3.10e-06 tok=25.921B dt=2348 wait=252 +13:40:01 pushed ckpt_w195.pt,log_w195.txt +13:41:32 waited 91s for global r=858 +13:41:40 global r=858 load 4.280 -> 4.280 +13:42:13 step 196660 loss=3.1447 lr=3.09e-06 tok=25.923B dt=2525 wait=252 +13:43:18 step 196680 loss=3.1578 lr=3.07e-06 tok=25.926B dt=2590 wait=252 +13:44:23 step 196700 loss=3.1546 lr=3.05e-06 tok=25.929B dt=2654 wait=252 +13:45:27 step 196720 loss=3.1969 lr=3.03e-06 tok=25.931B dt=2719 wait=252 +13:46:32 step 196740 loss=3.1031 lr=3.01e-06 tok=25.934B dt=2783 wait=252 +13:47:37 step 196760 loss=2.9482 lr=2.99e-06 tok=25.937B dt=2848 wait=253 +13:48:41 step 196780 loss=3.1306 lr=2.97e-06 tok=25.939B dt=2912 wait=253 +13:49:46 step 196800 loss=3.1097 lr=2.96e-06 tok=25.942B dt=2977 wait=253 +13:50:01 pushed ckpt_w195.pt,log_w195.txt +13:51:16 waited 76s for global r=859 +13:51:23 global r=859 load 4.281 -> 4.281 +13:52:29 step 196820 loss=3.1430 lr=2.94e-06 tok=25.944B dt=3141 wait=253 +13:53:33 step 196840 loss=3.1589 lr=2.92e-06 tok=25.947B dt=3205 wait=253 +13:54:37 step 196860 loss=3.0137 lr=2.90e-06 tok=25.950B dt=3268 wait=253 +13:55:41 step 196880 loss=2.9134 lr=2.88e-06 tok=25.952B dt=3333 wait=253 +13:56:45 step 196900 loss=3.0331 lr=2.86e-06 tok=25.955B dt=3397 wait=254 +13:57:49 step 196920 loss=3.0155 lr=2.85e-06 tok=25.957B dt=3461 wait=254 +13:58:53 step 196940 loss=2.9515 lr=2.83e-06 tok=25.960B dt=3525 wait=254 +13:59:38 pushed ckpt_w195.pt,log_w195.txt +14:01:09 waited 91s for global r=860 +14:01:15 global r=860 load 4.281 -> 4.281 +14:01:48 step 196960 loss=3.1150 lr=2.81e-06 tok=25.963B dt=3700 wait=254 +14:02:52 step 196980 loss=3.0666 lr=2.79e-06 tok=25.965B dt=3764 wait=254 +14:03:56 step 197000 loss=3.0412 lr=2.77e-06 tok=25.968B dt=3827 wait=254 +14:05:00 step 197020 loss=3.1083 lr=2.75e-06 tok=25.971B dt=3891 wait=255 +14:06:05 step 197040 loss=2.8788 lr=2.73e-06 tok=25.973B dt=3957 wait=255 +14:07:09 step 197060 loss=3.1477 lr=2.72e-06 tok=25.976B dt=4021 wait=255 +14:08:12 step 197080 loss=3.0882 lr=2.70e-06 tok=25.978B dt=4084 wait=255 +14:09:18 step 197100 loss=3.1777 lr=2.68e-06 tok=25.981B dt=4149 wait=255 +14:09:34 pushed ckpt_w195.pt,log_w195.txt +14:11:04 waited 91s for global r=861 +14:11:11 global r=861 load 4.281 -> 4.280 +14:12:17 step 197120 loss=3.1745 lr=2.66e-06 tok=25.984B dt=4329 wait=255 +14:13:22 step 197140 loss=3.0306 lr=2.64e-06 tok=25.986B dt=4394 wait=255 +14:14:27 step 197160 loss=2.7930 lr=2.62e-06 tok=25.989B dt=4459 wait=256 +14:15:31 step 197180 loss=3.0339 lr=2.61e-06 tok=25.992B dt=4523 wait=256 +14:16:36 step 197200 loss=3.1302 lr=2.59e-06 tok=25.994B dt=4588 wait=256 +14:17:41 step 197220 loss=3.1253 lr=2.57e-06 tok=25.997B dt=4653 wait=256 +14:18:46 step 197240 loss=2.9694 lr=2.55e-06 tok=25.999B dt=4718 wait=256 +14:19:31 pushed ckpt_w195.pt,log_w195.txt +14:20:47 waited 76s for global r=862 +14:20:55 global r=862 load 4.281 -> 4.281 +14:21:28 step 197260 loss=3.1190 lr=2.53e-06 tok=26.002B dt=4879 wait=256 +14:22:32 step 197280 loss=3.0304 lr=2.51e-06 tok=26.005B dt=4944 wait=257 +14:23:36 step 197300 loss=3.0914 lr=2.49e-06 tok=26.007B dt=5008 wait=257 +14:24:40 step 197320 loss=3.0495 lr=2.48e-06 tok=26.010B dt=5072 wait=257 +14:25:45 step 197340 loss=3.0438 lr=2.46e-06 tok=26.013B dt=5137 wait=257 +14:26:50 step 197360 loss=3.0692 lr=2.44e-06 tok=26.015B dt=5201 wait=257 +14:27:54 step 197380 loss=3.0833 lr=2.42e-06 tok=26.018B dt=5265 wait=257 +14:28:58 step 197400 loss=3.0577 lr=2.40e-06 tok=26.020B dt=5329 wait=258 +14:29:14 pushed ckpt_w195.pt,log_w195.txt +14:30:30 waited 76s for global r=863 +14:30:36 global r=863 load 4.280 -> 4.280 +14:31:40 step 197420 loss=3.0389 lr=2.38e-06 tok=26.023B dt=5492 wait=258 +14:32:44 step 197440 loss=3.0045 lr=2.37e-06 tok=26.026B dt=5556 wait=258 +14:33:49 step 197460 loss=3.1163 lr=2.35e-06 tok=26.028B dt=5620 wait=258 +14:34:53 step 197480 loss=2.9787 lr=2.33e-06 tok=26.031B dt=5684 wait=258 +14:35:57 step 197500 loss=3.0543 lr=2.31e-06 tok=26.034B dt=5748 wait=258 +14:37:01 step 197520 loss=3.1003 lr=2.29e-06 tok=26.036B dt=5813 wait=258 +14:38:05 step 197540 loss=2.9473 lr=2.27e-06 tok=26.039B dt=5877 wait=259 +14:38:50 pushed ckpt_w195.pt,log_w195.txt +14:40:20 waited 91s for global r=864 +14:40:30 global r=864 load 4.280 -> 4.280 +14:41:03 step 197560 loss=3.0519 lr=2.25e-06 tok=26.041B dt=6055 wait=259 +14:42:07 step 197580 loss=3.0729 lr=2.24e-06 tok=26.044B dt=6119 wait=259 +14:43:11 step 197600 loss=3.0410 lr=2.22e-06 tok=26.047B dt=6182 wait=259 +14:44:14 step 197620 loss=3.1350 lr=2.20e-06 tok=26.049B dt=6246 wait=259 +14:45:18 step 197640 loss=3.0764 lr=2.18e-06 tok=26.052B dt=6310 wait=259 +14:46:22 step 197660 loss=3.1246 lr=2.16e-06 tok=26.054B dt=6374 wait=260 +14:47:26 step 197680 loss=3.0728 lr=2.14e-06 tok=26.057B dt=6437 wait=260 +14:48:30 step 197700 loss=3.1167 lr=2.12e-06 tok=26.060B dt=6502 wait=260 +14:48:45 pushed ckpt_w195.pt,log_w195.txt +14:50:01 waited 76s for global r=865 +14:50:08 global r=865 load 4.280 -> 4.281 +14:51:13 step 197720 loss=3.0495 lr=2.11e-06 tok=26.062B dt=6665 wait=260 +14:52:17 step 197740 loss=3.0763 lr=2.09e-06 tok=26.065B dt=6728 wait=260 +14:53:20 step 197760 loss=3.0366 lr=2.07e-06 tok=26.068B dt=6792 wait=260 +14:54:24 step 197780 loss=3.0196 lr=2.05e-06 tok=26.070B dt=6856 wait=261 +14:55:28 step 197800 loss=2.9938 lr=2.03e-06 tok=26.073B dt=6920 wait=261 +14:56:32 step 197820 loss=3.0740 lr=2.01e-06 tok=26.075B dt=6984 wait=261 +14:57:35 step 197840 loss=2.9160 lr=2.00e-06 tok=26.078B dt=7047 wait=261 +14:58:21 pushed ckpt_w195.pt,log_w195.txt +14:59:37 waited 76s for global r=866 +14:59:44 global r=866 load 4.280 -> 4.280 +15:00:17 step 197860 loss=2.9753 lr=1.98e-06 tok=26.081B dt=7209 wait=261 +15:01:21 step 197880 loss=2.9029 lr=1.96e-06 tok=26.083B dt=7273 wait=261 +15:02:25 step 197900 loss=2.9134 lr=1.94e-06 tok=26.086B dt=7336 wait=261 +15:03:28 step 197920 loss=2.9521 lr=1.92e-06 tok=26.089B dt=7400 wait=262 +15:04:32 step 197940 loss=2.8698 lr=1.90e-06 tok=26.091B dt=7464 wait=262 +15:05:35 step 197960 loss=2.9431 lr=1.88e-06 tok=26.094B dt=7527 wait=262 +15:06:39 step 197980 loss=2.8846 lr=1.87e-06 tok=26.096B dt=7590 wait=262 +15:07:42 step 198000 loss=2.9015 lr=1.85e-06 tok=26.099B dt=7654 wait=262 +15:07:54 pushed ckpt_w195.pt,log_w195.txt +15:09:25 waited 91s for global r=867 +15:09:32 global r=867 load 4.281 -> 4.280 +15:10:36 step 198020 loss=2.9015 lr=1.83e-06 tok=26.102B dt=7828 wait=262 +15:11:40 step 198040 loss=2.9610 lr=1.81e-06 tok=26.104B dt=7891 wait=263 +15:12:43 step 198060 loss=2.9227 lr=1.79e-06 tok=26.107B dt=7954 wait=263 +15:13:46 step 198080 loss=2.8709 lr=1.77e-06 tok=26.110B dt=8017 wait=263 +15:14:49 step 198100 loss=2.8813 lr=1.76e-06 tok=26.112B dt=8081 wait=263 +15:15:52 step 198120 loss=2.9790 lr=1.74e-06 tok=26.115B dt=8143 wait=263 +15:16:55 step 198140 loss=2.8297 lr=1.72e-06 tok=26.117B dt=8206 wait=263 +15:17:41 pushed ckpt_w195.pt,log_w195.txt +15:18:57 waited 76s for global r=868 +15:19:07 global r=868 load 4.281 -> 4.280 +15:19:40 step 198160 loss=2.7787 lr=1.70e-06 tok=26.120B dt=8372 wait=263 +15:20:44 step 198180 loss=2.7574 lr=1.68e-06 tok=26.123B dt=8435 wait=264 +15:21:47 step 198200 loss=2.7055 lr=1.66e-06 tok=26.125B dt=8499 wait=264 +15:22:50 step 198220 loss=2.8008 lr=1.64e-06 tok=26.128B dt=8562 wait=264 +15:23:53 step 198240 loss=2.8042 lr=1.63e-06 tok=26.131B dt=8625 wait=264 +15:24:56 step 198260 loss=2.8779 lr=1.61e-06 tok=26.133B dt=8688 wait=264 +15:25:59 step 198280 loss=2.8482 lr=1.59e-06 tok=26.136B dt=8751 wait=264 +15:27:03 step 198300 loss=2.8468 lr=1.57e-06 tok=26.138B dt=8815 wait=264 +15:27:17 pushed ckpt_w195.pt,log_w195.txt +15:28:33 waited 76s for global r=869 +15:28:41 global r=869 load 4.281 -> 4.281 +15:29:46 step 198320 loss=2.8309 lr=1.55e-06 tok=26.141B dt=8977 wait=265 +15:30:49 step 198340 loss=2.9214 lr=1.53e-06 tok=26.144B dt=9040 wait=265 +15:31:52 step 198360 loss=2.9153 lr=1.52e-06 tok=26.146B dt=9104 wait=265 +15:32:56 step 198380 loss=2.8780 lr=1.50e-06 tok=26.149B dt=9168 wait=265 +15:33:59 step 198400 loss=2.9178 lr=1.48e-06 tok=26.151B dt=9231 wait=265 +15:35:03 step 198420 loss=2.9788 lr=1.46e-06 tok=26.154B dt=9295 wait=265 +15:36:07 step 198440 loss=2.8826 lr=1.44e-06 tok=26.157B dt=9358 wait=265 +15:36:52 pushed ckpt_w195.pt,log_w195.txt +15:38:23 waited 91s for global r=870 +15:38:29 global r=870 load 4.281 -> 4.281 +15:39:02 step 198460 loss=3.0518 lr=1.42e-06 tok=26.159B dt=9534 wait=266 +15:40:06 step 198480 loss=2.9629 lr=1.40e-06 tok=26.162B dt=9598 wait=266 +15:41:10 step 198500 loss=3.0359 lr=1.39e-06 tok=26.165B dt=9661 wait=266 +15:42:14 step 198520 loss=2.9704 lr=1.37e-06 tok=26.167B dt=9725 wait=266 +15:43:17 step 198540 loss=2.9873 lr=1.35e-06 tok=26.170B dt=9789 wait=266 +15:44:21 step 198560 loss=3.1141 lr=1.33e-06 tok=26.172B dt=9853 wait=266 +15:45:25 step 198580 loss=2.9959 lr=1.31e-06 tok=26.175B dt=9917 wait=267 +15:46:29 step 198600 loss=2.9787 lr=1.29e-06 tok=26.178B dt=9981 wait=267 diff --git a/log_w196.txt b/log_w196.txt new file mode 100644 index 0000000000000000000000000000000000000000..46d7f5d351204027694e097557ad7039cdc9b5fc --- /dev/null +++ b/log_w196.txt @@ -0,0 +1,182 @@ +launch +12:59:13 name=w196 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 143/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:59:19 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:59:19 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:59:29 params 1.08B +12:59:30 resumed step=196050 seen=25.85B round=854 +12:59:30 stream salt=196050 +13:01:35 blocks compiled +13:01:35 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:06:28 step 196051 loss=2.7837 lr=3.65e-06 tok=25.846B dt=294 wait=264 +13:06:45 step 196052 loss=2.9625 lr=3.65e-06 tok=25.846B dt=310 wait=264 +13:07:10 step 196060 loss=3.2582 lr=3.64e-06 tok=25.847B dt=336 wait=264 +13:08:16 step 196080 loss=3.2056 lr=3.62e-06 tok=25.850B dt=401 wait=264 +13:09:21 step 196100 loss=3.1889 lr=3.60e-06 tok=25.853B dt=466 wait=265 +13:10:25 step 196120 loss=3.0904 lr=3.58e-06 tok=25.855B dt=531 wait=265 +13:11:30 step 196140 loss=3.2229 lr=3.57e-06 tok=25.858B dt=595 wait=265 +13:12:18 global r=855 load 4.281 -> 4.281 +13:13:22 step 196220 loss=3.0835 lr=3.49e-06 tok=25.862B dt=708 wait=265 +13:14:27 step 196240 loss=3.0249 lr=3.47e-06 tok=25.864B dt=773 wait=265 +13:15:32 step 196260 loss=2.8818 lr=3.45e-06 tok=25.867B dt=837 wait=265 +13:16:37 step 196280 loss=3.0373 lr=3.44e-06 tok=25.870B dt=902 wait=266 +13:17:41 step 196300 loss=3.0956 lr=3.42e-06 tok=25.872B dt=966 wait=266 +13:18:45 step 196320 loss=3.2375 lr=3.40e-06 tok=25.875B dt=1031 wait=266 +13:19:49 step 196340 loss=2.9686 lr=3.38e-06 tok=25.878B dt=1095 wait=266 +13:20:36 pushed ckpt_w196.pt,log_w196.txt +13:21:52 waited 76s for global r=856 +13:22:00 global r=856 load 4.281 -> 4.281 +13:22:34 step 196360 loss=3.1205 lr=3.36e-06 tok=25.880B dt=1259 wait=266 +13:23:39 step 196380 loss=2.9876 lr=3.34e-06 tok=25.883B dt=1324 wait=266 +13:24:43 step 196400 loss=3.0722 lr=3.33e-06 tok=25.885B dt=1388 wait=266 +13:25:47 step 196420 loss=3.1298 lr=3.31e-06 tok=25.888B dt=1453 wait=267 +13:26:51 step 196440 loss=3.0419 lr=3.29e-06 tok=25.891B dt=1517 wait=267 +13:27:55 step 196460 loss=3.1470 lr=3.27e-06 tok=25.893B dt=1581 wait=267 +13:28:59 step 196480 loss=3.0500 lr=3.25e-06 tok=25.896B dt=1645 wait=267 +13:30:02 step 196500 loss=3.0964 lr=3.23e-06 tok=25.899B dt=1708 wait=267 +13:30:14 pushed ckpt_w196.pt,log_w196.txt +13:31:45 waited 91s for global r=857 +13:31:52 global r=857 load 4.281 -> 4.281 +13:32:58 step 196520 loss=2.9512 lr=3.21e-06 tok=25.901B dt=1884 wait=267 +13:34:02 step 196540 loss=3.0172 lr=3.20e-06 tok=25.904B dt=1948 wait=268 +13:35:07 step 196560 loss=3.0619 lr=3.18e-06 tok=25.906B dt=2012 wait=268 +13:36:11 step 196580 loss=3.1566 lr=3.16e-06 tok=25.909B dt=2077 wait=268 +13:37:16 step 196600 loss=3.0525 lr=3.14e-06 tok=25.912B dt=2142 wait=268 +13:38:21 step 196620 loss=3.0903 lr=3.12e-06 tok=25.914B dt=2207 wait=268 +13:39:26 step 196640 loss=2.9860 lr=3.10e-06 tok=25.917B dt=2272 wait=268 +13:40:11 pushed ckpt_w196.pt,log_w196.txt +13:41:27 waited 76s for global r=858 +13:41:34 global r=858 load 4.281 -> 4.280 +13:42:08 step 196660 loss=3.1664 lr=3.09e-06 tok=25.919B dt=2433 wait=268 +13:43:12 step 196680 loss=3.0031 lr=3.07e-06 tok=25.922B dt=2497 wait=269 +13:44:16 step 196700 loss=2.9934 lr=3.05e-06 tok=25.925B dt=2561 wait=269 +13:45:19 step 196720 loss=3.1514 lr=3.03e-06 tok=25.927B dt=2625 wait=269 +13:46:23 step 196740 loss=3.0340 lr=3.01e-06 tok=25.930B dt=2689 wait=269 +13:47:27 step 196760 loss=3.1251 lr=2.99e-06 tok=25.933B dt=2753 wait=269 +13:48:31 step 196780 loss=3.1880 lr=2.97e-06 tok=25.935B dt=2817 wait=269 +13:49:36 step 196800 loss=3.3122 lr=2.96e-06 tok=25.938B dt=2881 wait=270 +13:49:47 pushed ckpt_w196.pt,log_w196.txt +13:51:18 waited 91s for global r=859 +13:51:26 global r=859 load 4.280 -> 4.281 +13:52:33 step 196820 loss=3.2224 lr=2.94e-06 tok=25.940B dt=3058 wait=270 +13:53:37 step 196840 loss=3.1313 lr=2.92e-06 tok=25.943B dt=3123 wait=270 +13:54:41 step 196860 loss=3.0730 lr=2.90e-06 tok=25.946B dt=3187 wait=270 +13:55:46 step 196880 loss=3.1786 lr=2.88e-06 tok=25.948B dt=3252 wait=270 +13:56:51 step 196900 loss=2.9946 lr=2.86e-06 tok=25.951B dt=3317 wait=270 +13:57:56 step 196920 loss=3.0718 lr=2.85e-06 tok=25.954B dt=3382 wait=270 +13:59:02 step 196940 loss=3.0737 lr=2.83e-06 tok=25.956B dt=3447 wait=271 +13:59:47 pushed ckpt_w196.pt,log_w196.txt +14:01:18 waited 91s for global r=860 +14:01:25 global r=860 load 4.281 -> 4.281 +14:01:59 step 196960 loss=3.0861 lr=2.81e-06 tok=25.959B dt=3624 wait=271 +14:03:04 step 196980 loss=2.9963 lr=2.79e-06 tok=25.961B dt=3690 wait=271 +14:04:08 step 197000 loss=2.9857 lr=2.77e-06 tok=25.964B dt=3754 wait=271 +14:05:13 step 197020 loss=2.9960 lr=2.75e-06 tok=25.967B dt=3819 wait=271 +14:06:18 step 197040 loss=3.0070 lr=2.73e-06 tok=25.969B dt=3884 wait=271 +14:07:23 step 197060 loss=3.0200 lr=2.72e-06 tok=25.972B dt=3949 wait=271 +14:08:28 step 197080 loss=2.9119 lr=2.70e-06 tok=25.975B dt=4014 wait=272 +14:09:33 step 197100 loss=3.0328 lr=2.68e-06 tok=25.977B dt=4079 wait=272 +14:09:47 pushed ckpt_w196.pt,log_w196.txt +14:11:03 waited 76s for global r=861 +14:11:11 global r=861 load 4.281 -> 4.280 +14:12:17 step 197120 loss=2.9863 lr=2.66e-06 tok=25.980B dt=4242 wait=272 +14:13:22 step 197140 loss=2.8594 lr=2.64e-06 tok=25.982B dt=4307 wait=272 +14:14:27 step 197160 loss=2.9053 lr=2.62e-06 tok=25.985B dt=4373 wait=272 +14:15:32 step 197180 loss=2.9352 lr=2.61e-06 tok=25.988B dt=4438 wait=272 +14:16:36 step 197200 loss=2.9516 lr=2.59e-06 tok=25.990B dt=4502 wait=273 +14:17:41 step 197220 loss=3.0165 lr=2.57e-06 tok=25.993B dt=4567 wait=273 +14:18:46 step 197240 loss=2.9242 lr=2.55e-06 tok=25.996B dt=4631 wait=273 +14:19:32 pushed ckpt_w196.pt,log_w196.txt +14:20:48 waited 76s for global r=862 +14:21:02 global r=862 load 4.280 -> 4.281 +14:21:35 step 197260 loss=3.0419 lr=2.53e-06 tok=25.998B dt=4801 wait=273 +14:22:41 step 197280 loss=2.9423 lr=2.51e-06 tok=26.001B dt=4866 wait=273 +14:23:45 step 197300 loss=2.9679 lr=2.49e-06 tok=26.003B dt=4930 wait=273 +14:24:50 step 197320 loss=2.9814 lr=2.48e-06 tok=26.006B dt=4995 wait=273 +14:25:54 step 197340 loss=2.9343 lr=2.46e-06 tok=26.009B dt=5059 wait=274 +14:26:59 step 197360 loss=3.0028 lr=2.44e-06 tok=26.011B dt=5125 wait=274 +14:28:05 step 197380 loss=2.8486 lr=2.42e-06 tok=26.014B dt=5190 wait=274 +14:29:09 step 197400 loss=2.9000 lr=2.40e-06 tok=26.016B dt=5255 wait=274 +14:29:23 pushed ckpt_w196.pt,log_w196.txt +14:30:38 waited 76s for global r=863 +14:30:44 global r=863 load 4.281 -> 4.280 +14:31:50 step 197420 loss=2.9101 lr=2.38e-06 tok=26.019B dt=5415 wait=274 +14:32:54 step 197440 loss=2.9004 lr=2.37e-06 tok=26.022B dt=5479 wait=274 +14:33:58 step 197460 loss=2.8964 lr=2.35e-06 tok=26.024B dt=5544 wait=275 +14:35:03 step 197480 loss=2.8219 lr=2.33e-06 tok=26.027B dt=5608 wait=275 +14:36:08 step 197500 loss=2.7942 lr=2.31e-06 tok=26.030B dt=5673 wait=275 +14:37:13 step 197520 loss=2.9228 lr=2.29e-06 tok=26.032B dt=5739 wait=275 +14:38:17 step 197540 loss=2.8059 lr=2.27e-06 tok=26.035B dt=5803 wait=275 +14:39:02 pushed ckpt_w196.pt,log_w196.txt +14:40:17 waited 76s for global r=864 +14:40:25 global r=864 load 4.281 -> 4.280 +14:40:59 step 197560 loss=2.8841 lr=2.25e-06 tok=26.037B dt=5965 wait=275 +14:42:04 step 197580 loss=2.7846 lr=2.24e-06 tok=26.040B dt=6029 wait=275 +14:43:08 step 197600 loss=2.8577 lr=2.22e-06 tok=26.043B dt=6093 wait=275 +14:44:12 step 197620 loss=2.8655 lr=2.20e-06 tok=26.045B dt=6157 wait=276 +14:45:16 step 197640 loss=2.9023 lr=2.18e-06 tok=26.048B dt=6221 wait=276 +14:46:20 step 197660 loss=3.0116 lr=2.16e-06 tok=26.051B dt=6285 wait=276 +14:47:23 step 197680 loss=2.9803 lr=2.14e-06 tok=26.053B dt=6349 wait=276 +14:48:28 step 197700 loss=2.9041 lr=2.12e-06 tok=26.056B dt=6414 wait=276 +14:48:40 pushed ckpt_w196.pt,log_w196.txt +14:50:11 waited 91s for global r=865 +14:50:16 global r=865 load 4.281 -> 4.281 +14:51:22 step 197720 loss=3.0114 lr=2.11e-06 tok=26.058B dt=6587 wait=276 +14:52:26 step 197740 loss=2.9777 lr=2.09e-06 tok=26.061B dt=6651 wait=277 +14:53:30 step 197760 loss=3.0559 lr=2.07e-06 tok=26.064B dt=6715 wait=277 +14:54:34 step 197780 loss=2.9265 lr=2.05e-06 tok=26.066B dt=6779 wait=277 +14:55:38 step 197800 loss=2.9620 lr=2.03e-06 tok=26.069B dt=6844 wait=277 +14:56:43 step 197820 loss=2.9713 lr=2.01e-06 tok=26.072B dt=6908 wait=277 +14:57:48 step 197840 loss=2.8136 lr=2.00e-06 tok=26.074B dt=6973 wait=277 +14:58:31 pushed ckpt_w196.pt,log_w196.txt +14:59:47 waited 76s for global r=866 +14:59:56 global r=866 load 4.281 -> 4.280 +15:00:30 step 197860 loss=2.9742 lr=1.98e-06 tok=26.077B dt=7135 wait=277 +15:01:34 step 197880 loss=2.9008 lr=1.96e-06 tok=26.079B dt=7200 wait=278 +15:02:38 step 197900 loss=2.8322 lr=1.94e-06 tok=26.082B dt=7264 wait=278 +15:03:42 step 197920 loss=2.9506 lr=1.92e-06 tok=26.085B dt=7328 wait=278 +15:04:46 step 197940 loss=2.7858 lr=1.90e-06 tok=26.087B dt=7391 wait=278 +15:05:49 step 197960 loss=2.9559 lr=1.88e-06 tok=26.090B dt=7455 wait=278 +15:06:53 step 197980 loss=2.8113 lr=1.87e-06 tok=26.093B dt=7519 wait=278 +15:07:58 step 198000 loss=2.9158 lr=1.85e-06 tok=26.095B dt=7583 wait=278 +15:08:09 pushed ckpt_w196.pt,log_w196.txt +15:09:25 waited 76s for global r=867 +15:09:30 global r=867 load 4.281 -> 4.280 +15:10:35 step 198020 loss=2.8001 lr=1.83e-06 tok=26.098B dt=7741 wait=279 +15:11:39 step 198040 loss=2.8814 lr=1.81e-06 tok=26.100B dt=7804 wait=279 +15:12:43 step 198060 loss=2.8917 lr=1.79e-06 tok=26.103B dt=7868 wait=279 +15:13:47 step 198080 loss=2.8705 lr=1.77e-06 tok=26.106B dt=7932 wait=279 +15:14:50 step 198100 loss=2.8936 lr=1.76e-06 tok=26.108B dt=7996 wait=279 +15:15:54 step 198120 loss=2.9718 lr=1.74e-06 tok=26.111B dt=8059 wait=279 +15:16:58 step 198140 loss=2.9045 lr=1.72e-06 tok=26.113B dt=8124 wait=279 +15:17:44 pushed ckpt_w196.pt,log_w196.txt +15:18:59 waited 76s for global r=868 +15:19:05 global r=868 load 4.281 -> 4.280 +15:19:39 step 198160 loss=2.9794 lr=1.70e-06 tok=26.116B dt=8284 wait=280 +15:20:42 step 198180 loss=2.8516 lr=1.68e-06 tok=26.119B dt=8347 wait=280 +15:21:46 step 198200 loss=2.8393 lr=1.66e-06 tok=26.121B dt=8411 wait=280 +15:22:50 step 198220 loss=3.0082 lr=1.64e-06 tok=26.124B dt=8475 wait=280 +15:23:53 step 198240 loss=2.8476 lr=1.63e-06 tok=26.127B dt=8539 wait=280 +15:24:57 step 198260 loss=2.8456 lr=1.61e-06 tok=26.129B dt=8603 wait=280 +15:26:01 step 198280 loss=2.7616 lr=1.59e-06 tok=26.132B dt=8667 wait=280 +15:27:05 step 198300 loss=2.8719 lr=1.57e-06 tok=26.134B dt=8731 wait=281 +15:27:21 pushed ckpt_w196.pt,log_w196.txt +15:28:37 waited 76s for global r=869 +15:28:44 global r=869 load 4.281 -> 4.281 +15:29:48 step 198320 loss=2.7698 lr=1.55e-06 tok=26.137B dt=8894 wait=281 +15:30:52 step 198340 loss=2.8734 lr=1.53e-06 tok=26.140B dt=8958 wait=281 +15:31:56 step 198360 loss=2.8729 lr=1.52e-06 tok=26.142B dt=9021 wait=281 +15:32:59 step 198380 loss=2.7923 lr=1.50e-06 tok=26.145B dt=9085 wait=281 +15:34:02 step 198400 loss=2.8323 lr=1.48e-06 tok=26.148B dt=9148 wait=281 +15:35:07 step 198420 loss=2.8252 lr=1.46e-06 tok=26.150B dt=9212 wait=281 +15:36:10 step 198440 loss=2.8357 lr=1.44e-06 tok=26.153B dt=9275 wait=282 +15:36:54 pushed ckpt_w196.pt,log_w196.txt +15:38:24 waited 91s for global r=870 +15:38:31 global r=870 load 4.280 -> 4.281 +15:39:04 step 198460 loss=2.8754 lr=1.42e-06 tok=26.155B dt=9449 wait=282 +15:40:08 step 198480 loss=2.7340 lr=1.40e-06 tok=26.158B dt=9513 wait=282 +15:41:11 step 198500 loss=2.8512 lr=1.39e-06 tok=26.161B dt=9577 wait=282 +15:42:15 step 198520 loss=2.9361 lr=1.37e-06 tok=26.163B dt=9640 wait=282 +15:43:18 step 198540 loss=2.8459 lr=1.35e-06 tok=26.166B dt=9703 wait=282 +15:44:22 step 198560 loss=2.8660 lr=1.33e-06 tok=26.169B dt=9767 wait=283 +15:45:25 step 198580 loss=2.9377 lr=1.31e-06 tok=26.171B dt=9830 wait=283 +15:46:28 step 198600 loss=2.8659 lr=1.29e-06 tok=26.174B dt=9894 wait=283 diff --git a/log_w197.txt b/log_w197.txt new file mode 100644 index 0000000000000000000000000000000000000000..1348405843dbd1670a70864b4777a51d8370323f --- /dev/null +++ b/log_w197.txt @@ -0,0 +1,283 @@ +launch +13:00:51 name=w197 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 144/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:00:57 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:00:57 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:01:07 params 1.08B +13:01:08 resumed step=196050 seen=25.85B round=854 +13:01:08 stream salt=196050 +13:03:00 blocks compiled +13:03:00 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:08:03 step 196051 loss=2.8155 lr=3.65e-06 tok=25.846B dt=303 wait=275 +13:08:19 step 196052 loss=2.8374 lr=3.65e-06 tok=25.846B dt=319 wait=275 +13:08:44 step 196060 loss=3.0794 lr=3.64e-06 tok=25.847B dt=344 wait=275 +13:09:48 step 196080 loss=2.9052 lr=3.62e-06 tok=25.850B dt=409 wait=276 +13:10:52 step 196100 loss=2.9642 lr=3.60e-06 tok=25.853B dt=473 wait=276 +13:11:56 step 196120 loss=2.9702 lr=3.58e-06 tok=25.855B dt=536 wait=276 +13:12:12 global r=855 load 4.282 -> 4.281 +13:13:16 step 196220 loss=2.9207 lr=3.49e-06 tok=25.858B dt=616 wait=276 +13:14:20 step 196240 loss=2.9288 lr=3.47e-06 tok=25.861B dt=681 wait=276 +13:15:24 step 196260 loss=2.9145 lr=3.45e-06 tok=25.863B dt=744 wait=276 +13:16:28 step 196280 loss=2.9062 lr=3.44e-06 tok=25.866B dt=808 wait=277 +13:17:32 step 196300 loss=3.0280 lr=3.42e-06 tok=25.868B dt=872 wait=277 +13:18:36 step 196320 loss=3.0942 lr=3.40e-06 tok=25.871B dt=936 wait=277 +13:19:40 step 196340 loss=2.9448 lr=3.38e-06 tok=25.874B dt=1001 wait=277 +13:20:33 pushed ckpt_w197.pt,log_w197.txt +13:22:04 waited 91s for global r=856 +13:22:11 global r=856 load 4.281 -> 4.281 +13:22:46 step 196360 loss=3.1001 lr=3.36e-06 tok=25.876B dt=1186 wait=277 +13:23:50 step 196380 loss=3.0483 lr=3.34e-06 tok=25.879B dt=1250 wait=277 +13:24:54 step 196400 loss=3.0299 lr=3.33e-06 tok=25.881B dt=1315 wait=277 +13:25:59 step 196420 loss=2.8784 lr=3.31e-06 tok=25.884B dt=1380 wait=278 +13:27:03 step 196440 loss=3.0798 lr=3.29e-06 tok=25.887B dt=1444 wait=278 +13:28:08 step 196460 loss=3.1547 lr=3.27e-06 tok=25.889B dt=1508 wait=278 +13:29:12 step 196480 loss=3.1011 lr=3.25e-06 tok=25.892B dt=1572 wait=278 +13:30:17 step 196500 loss=3.1146 lr=3.23e-06 tok=25.895B dt=1637 wait=278 +13:30:28 pushed ckpt_w197.pt,log_w197.txt +13:31:44 waited 76s for global r=857 +13:31:51 global r=857 load 4.280 -> 4.281 +13:32:57 step 196520 loss=3.0019 lr=3.21e-06 tok=25.897B dt=1798 wait=278 +13:34:01 step 196540 loss=3.0921 lr=3.20e-06 tok=25.900B dt=1861 wait=279 +13:35:06 step 196560 loss=3.1210 lr=3.18e-06 tok=25.902B dt=1927 wait=279 +13:36:11 step 196580 loss=3.1721 lr=3.16e-06 tok=25.905B dt=1991 wait=279 +13:37:15 step 196600 loss=3.1249 lr=3.14e-06 tok=25.908B dt=2056 wait=279 +13:38:21 step 196620 loss=3.2038 lr=3.12e-06 tok=25.910B dt=2121 wait=279 +13:39:26 step 196640 loss=2.9691 lr=3.10e-06 tok=25.913B dt=2187 wait=279 +13:40:13 pushed ckpt_w197.pt,log_w197.txt +13:41:29 waited 76s for global r=858 +13:41:35 global r=858 load 4.281 -> 4.280 +13:42:09 step 196660 loss=3.0593 lr=3.09e-06 tok=25.916B dt=2349 wait=279 +13:43:14 step 196680 loss=2.9047 lr=3.07e-06 tok=25.918B dt=2414 wait=280 +13:44:18 step 196700 loss=2.9960 lr=3.05e-06 tok=25.921B dt=2479 wait=280 +13:45:23 step 196720 loss=3.0719 lr=3.03e-06 tok=25.923B dt=2543 wait=280 +13:46:26 step 196740 loss=3.0406 lr=3.01e-06 tok=25.926B dt=2607 wait=280 +13:47:31 step 196760 loss=2.8475 lr=2.99e-06 tok=25.929B dt=2672 wait=280 +13:48:36 step 196780 loss=2.8354 lr=2.97e-06 tok=25.931B dt=2736 wait=280 +13:49:40 step 196800 loss=3.0068 lr=2.96e-06 tok=25.934B dt=2800 wait=280 +13:49:54 pushed ckpt_w197.pt,log_w197.txt +13:51:25 waited 91s for global r=859 +13:51:36 global r=859 load 4.281 -> 4.281 +13:52:41 step 196820 loss=2.9777 lr=2.94e-06 tok=25.937B dt=2981 wait=281 +13:53:45 step 196840 loss=3.0476 lr=2.92e-06 tok=25.939B dt=3045 wait=281 +13:54:50 step 196860 loss=3.0102 lr=2.90e-06 tok=25.942B dt=3110 wait=281 +13:55:54 step 196880 loss=2.9703 lr=2.88e-06 tok=25.944B dt=3174 wait=281 +13:56:57 step 196900 loss=2.9739 lr=2.86e-06 tok=25.947B dt=3237 wait=281 +13:58:01 step 196920 loss=3.0780 lr=2.85e-06 tok=25.950B dt=3301 wait=281 +13:59:06 step 196940 loss=2.8898 lr=2.83e-06 tok=25.952B dt=3366 wait=282 +13:59:53 pushed ckpt_w197.pt,log_w197.txt +14:01:24 waited 91s for global r=860 +14:01:30 global r=860 load 4.281 -> 4.281 +14:02:03 step 196960 loss=3.0248 lr=2.81e-06 tok=25.955B dt=3543 wait=282 +14:03:07 step 196980 loss=2.9101 lr=2.79e-06 tok=25.957B dt=3608 wait=282 +14:04:11 step 197000 loss=3.0901 lr=2.77e-06 tok=25.960B dt=3672 wait=282 +14:05:16 step 197020 loss=3.1365 lr=2.75e-06 tok=25.963B dt=3736 wait=282 +14:06:20 step 197040 loss=3.0118 lr=2.73e-06 tok=25.965B dt=3801 wait=282 +14:07:25 step 197060 loss=3.0913 lr=2.72e-06 tok=25.968B dt=3865 wait=283 +14:08:29 step 197080 loss=2.9215 lr=2.70e-06 tok=25.971B dt=3929 wait=283 +14:09:33 step 197100 loss=3.0345 lr=2.68e-06 tok=25.973B dt=3993 wait=283 +14:09:46 pushed ckpt_w197.pt,log_w197.txt +14:11:02 waited 76s for global r=861 +14:11:08 global r=861 load 4.281 -> 4.280 +14:12:14 step 197120 loss=2.8149 lr=2.66e-06 tok=25.976B dt=4154 wait=283 +14:13:18 step 197140 loss=3.0211 lr=2.64e-06 tok=25.978B dt=4218 wait=283 +14:14:22 step 197160 loss=2.8818 lr=2.62e-06 tok=25.981B dt=4282 wait=283 +14:15:25 step 197180 loss=2.8405 lr=2.61e-06 tok=25.984B dt=4346 wait=284 +14:16:30 step 197200 loss=2.8930 lr=2.59e-06 tok=25.986B dt=4410 wait=284 +14:17:33 step 197220 loss=2.9544 lr=2.57e-06 tok=25.989B dt=4473 wait=284 +14:18:37 step 197240 loss=2.9016 lr=2.55e-06 tok=25.992B dt=4537 wait=284 +14:19:21 pushed ckpt_w197.pt,log_w197.txt +14:20:52 waited 91s for global r=862 +14:20:58 global r=862 load 4.281 -> 4.281 +14:21:31 step 197260 loss=2.8640 lr=2.53e-06 tok=25.994B dt=4711 wait=284 +14:22:35 step 197280 loss=2.7686 lr=2.51e-06 tok=25.997B dt=4775 wait=284 +14:23:39 step 197300 loss=2.7647 lr=2.49e-06 tok=25.999B dt=4839 wait=284 +14:24:42 step 197320 loss=2.8871 lr=2.48e-06 tok=26.002B dt=4902 wait=285 +14:25:46 step 197340 loss=2.7949 lr=2.46e-06 tok=26.005B dt=4966 wait=285 +14:26:50 step 197360 loss=2.8169 lr=2.44e-06 tok=26.007B dt=5030 wait=285 +14:27:54 step 197380 loss=2.7362 lr=2.42e-06 tok=26.010B dt=5095 wait=285 +14:28:58 step 197400 loss=2.8229 lr=2.40e-06 tok=26.013B dt=5158 wait=285 +14:29:14 pushed ckpt_w197.pt,log_w197.txt +14:30:30 waited 76s for global r=863 +14:30:36 global r=863 load 4.281 -> 4.280 +14:31:41 step 197420 loss=2.7719 lr=2.38e-06 tok=26.015B dt=5321 wait=285 +14:32:44 step 197440 loss=2.9150 lr=2.37e-06 tok=26.018B dt=5385 wait=286 +14:33:48 step 197460 loss=2.7504 lr=2.35e-06 tok=26.020B dt=5448 wait=286 +14:34:51 step 197480 loss=2.8290 lr=2.33e-06 tok=26.023B dt=5512 wait=286 +14:35:55 step 197500 loss=2.7627 lr=2.31e-06 tok=26.026B dt=5575 wait=286 +14:36:58 step 197520 loss=2.8309 lr=2.29e-06 tok=26.028B dt=5639 wait=286 +14:38:02 step 197540 loss=2.7454 lr=2.27e-06 tok=26.031B dt=5702 wait=286 +14:38:47 pushed ckpt_w197.pt,log_w197.txt +14:40:18 waited 91s for global r=864 +14:40:26 global r=864 load 4.281 -> 4.280 +14:40:59 step 197560 loss=2.8234 lr=2.25e-06 tok=26.034B dt=5880 wait=287 +14:42:03 step 197580 loss=2.8424 lr=2.24e-06 tok=26.036B dt=5943 wait=287 +14:43:06 step 197600 loss=2.7871 lr=2.22e-06 tok=26.039B dt=6006 wait=287 +14:44:09 step 197620 loss=2.8850 lr=2.20e-06 tok=26.041B dt=6069 wait=287 +14:45:13 step 197640 loss=2.7657 lr=2.18e-06 tok=26.044B dt=6133 wait=287 +14:46:16 step 197660 loss=2.8781 lr=2.16e-06 tok=26.047B dt=6196 wait=287 +14:47:19 step 197680 loss=2.7995 lr=2.14e-06 tok=26.049B dt=6259 wait=287 +14:48:22 step 197700 loss=2.8390 lr=2.12e-06 tok=26.052B dt=6323 wait=288 +14:48:35 pushed ckpt_w197.pt,log_w197.txt +14:50:06 waited 91s for global r=865 +14:50:14 global r=865 load 4.281 -> 4.281 +14:51:19 step 197720 loss=2.8721 lr=2.11e-06 tok=26.054B dt=6500 wait=288 +14:52:23 step 197740 loss=2.9181 lr=2.09e-06 tok=26.057B dt=6563 wait=288 +14:53:27 step 197760 loss=2.9841 lr=2.07e-06 tok=26.060B dt=6628 wait=288 +14:54:32 step 197780 loss=2.7015 lr=2.05e-06 tok=26.062B dt=6693 wait=288 +14:55:36 step 197800 loss=2.8800 lr=2.03e-06 tok=26.065B dt=6756 wait=288 +14:56:40 step 197820 loss=2.9748 lr=2.01e-06 tok=26.068B dt=6820 wait=289 +14:57:45 step 197840 loss=3.0216 lr=2.00e-06 tok=26.070B dt=6885 wait=289 +14:58:29 pushed ckpt_w197.pt,log_w197.txt +14:59:45 waited 76s for global r=866 +14:59:54 global r=866 load 4.281 -> 4.280 +15:00:28 step 197860 loss=3.0165 lr=1.98e-06 tok=26.073B dt=7048 wait=289 +15:01:32 step 197880 loss=2.9329 lr=1.96e-06 tok=26.075B dt=7112 wait=289 +15:02:37 step 197900 loss=2.9870 lr=1.94e-06 tok=26.078B dt=7177 wait=289 +15:03:42 step 197920 loss=3.0481 lr=1.92e-06 tok=26.081B dt=7242 wait=289 +15:04:46 step 197940 loss=2.9427 lr=1.90e-06 tok=26.083B dt=7307 wait=290 +15:05:50 step 197960 loss=2.9396 lr=1.88e-06 tok=26.086B dt=7371 wait=290 +15:06:54 step 197980 loss=2.8531 lr=1.87e-06 tok=26.089B dt=7435 wait=290 +15:07:59 step 198000 loss=2.8483 lr=1.85e-06 tok=26.091B dt=7499 wait=290 +15:08:11 pushed ckpt_w197.pt,log_w197.txt +15:09:27 waited 76s for global r=867 +15:09:34 global r=867 load 4.281 -> 4.280 +15:10:39 step 198020 loss=2.9434 lr=1.83e-06 tok=26.094B dt=7660 wait=290 +15:11:43 step 198040 loss=2.8794 lr=1.81e-06 tok=26.096B dt=7723 wait=290 +15:12:46 step 198060 loss=2.9276 lr=1.79e-06 tok=26.099B dt=7787 wait=291 +15:13:50 step 198080 loss=2.7401 lr=1.77e-06 tok=26.102B dt=7850 wait=291 +15:14:53 step 198100 loss=2.8160 lr=1.76e-06 tok=26.104B dt=7913 wait=291 +15:15:56 step 198120 loss=2.8381 lr=1.74e-06 tok=26.107B dt=7977 wait=291 +15:16:59 step 198140 loss=2.7421 lr=1.72e-06 tok=26.110B dt=8039 wait=291 +15:17:43 pushed ckpt_w197.pt,log_w197.txt +15:18:59 waited 76s for global r=868 +15:19:05 global r=868 load 4.280 -> 4.280 +15:19:39 step 198160 loss=2.4078 lr=1.70e-06 tok=26.112B dt=8200 wait=291 +15:20:42 step 198180 loss=2.8370 lr=1.68e-06 tok=26.115B dt=8262 wait=291 +15:21:45 step 198200 loss=2.8482 lr=1.66e-06 tok=26.117B dt=8325 wait=292 +15:22:48 step 198220 loss=2.9588 lr=1.64e-06 tok=26.120B dt=8388 wait=292 +15:23:52 step 198240 loss=2.7998 lr=1.63e-06 tok=26.123B dt=8452 wait=292 +15:24:55 step 198260 loss=2.9465 lr=1.61e-06 tok=26.125B dt=8515 wait=292 +15:25:58 step 198280 loss=2.9517 lr=1.59e-06 tok=26.128B dt=8578 wait=292 +15:27:02 step 198300 loss=2.9772 lr=1.57e-06 tok=26.131B dt=8642 wait=292 +15:27:16 pushed ckpt_w197.pt,log_w197.txt +15:28:46 waited 91s for global r=869 +15:28:53 global r=869 load 4.280 -> 4.281 +15:29:58 step 198320 loss=2.8971 lr=1.55e-06 tok=26.133B dt=8818 wait=293 +15:31:01 step 198340 loss=2.8819 lr=1.53e-06 tok=26.136B dt=8881 wait=293 +15:32:05 step 198360 loss=2.8631 lr=1.52e-06 tok=26.138B dt=8945 wait=293 +15:33:09 step 198380 loss=2.8527 lr=1.50e-06 tok=26.141B dt=9009 wait=293 +15:34:12 step 198400 loss=2.9273 lr=1.48e-06 tok=26.144B dt=9072 wait=293 +15:35:16 step 198420 loss=2.9436 lr=1.46e-06 tok=26.146B dt=9136 wait=293 +15:36:20 step 198440 loss=2.8117 lr=1.44e-06 tok=26.149B dt=9200 wait=294 +15:37:04 pushed ckpt_w197.pt,log_w197.txt +15:38:20 waited 76s for global r=870 +15:38:25 global r=870 load 4.281 -> 4.281 +15:38:59 step 198460 loss=2.9915 lr=1.42e-06 tok=26.151B dt=9359 wait=294 +15:40:03 step 198480 loss=2.8028 lr=1.40e-06 tok=26.154B dt=9423 wait=294 +15:41:07 step 198500 loss=2.8400 lr=1.39e-06 tok=26.157B dt=9487 wait=294 +15:42:10 step 198520 loss=3.0809 lr=1.37e-06 tok=26.159B dt=9551 wait=294 +15:43:14 step 198540 loss=2.9044 lr=1.35e-06 tok=26.162B dt=9614 wait=294 +15:44:17 step 198560 loss=2.9125 lr=1.33e-06 tok=26.165B dt=9678 wait=294 +15:45:21 step 198580 loss=2.8869 lr=1.31e-06 tok=26.167B dt=9742 wait=295 +15:46:25 step 198600 loss=2.9224 lr=1.29e-06 tok=26.170B dt=9806 wait=295 +15:46:37 pushed ckpt_w197.pt,log_w197.txt +15:47:53 waited 76s for global r=871 +15:48:01 global r=871 load 4.281 -> 4.281 +15:49:05 step 198620 loss=2.9467 lr=1.28e-06 tok=26.172B dt=9966 wait=295 +15:50:09 step 198640 loss=2.9357 lr=1.26e-06 tok=26.175B dt=10030 wait=295 +15:51:13 step 198660 loss=2.9499 lr=1.24e-06 tok=26.178B dt=10094 wait=295 +15:52:17 step 198680 loss=2.9375 lr=1.22e-06 tok=26.180B dt=10157 wait=295 +15:53:21 step 198700 loss=2.9563 lr=1.20e-06 tok=26.183B dt=10221 wait=296 +15:54:25 step 198720 loss=3.0324 lr=1.18e-06 tok=26.186B dt=10285 wait=296 +15:55:28 step 198740 loss=2.9267 lr=1.16e-06 tok=26.188B dt=10348 wait=296 +15:56:12 pushed ckpt_w197.pt,log_w197.txt +15:59:46 step 198760 loss=3.0674 lr=1.15e-06 tok=26.191B dt=10607 wait=296 +15:59:53 global r=872 load 4.281 -> 4.281 +16:00:25 step 198760 loss=3.0093 lr=1.15e-06 tok=26.192B dt=10645 wait=296 +16:01:28 step 198780 loss=2.9936 lr=1.13e-06 tok=26.195B dt=10709 wait=296 +16:02:32 step 198800 loss=2.9711 lr=1.11e-06 tok=26.197B dt=10772 wait=296 +16:03:36 step 198820 loss=3.0050 lr=1.09e-06 tok=26.200B dt=10836 wait=296 +16:04:39 step 198840 loss=2.9445 lr=1.07e-06 tok=26.203B dt=10900 wait=297 +16:05:43 step 198860 loss=3.1046 lr=1.05e-06 tok=26.205B dt=10964 wait=297 +16:06:47 step 198880 loss=2.9762 lr=1.04e-06 tok=26.208B dt=11027 wait=297 +16:07:51 step 198900 loss=3.0821 lr=1.02e-06 tok=26.210B dt=11091 wait=297 +16:08:06 pushed ckpt_w197.pt,log_w197.txt +16:11:50 global r=873 load 4.281 -> 4.281 +16:12:54 step 198920 loss=2.9199 lr=9.98e-07 tok=26.214B dt=11394 wait=297 +16:13:57 step 198940 loss=2.9662 lr=9.80e-07 tok=26.217B dt=11458 wait=297 +16:15:29 step 198960 loss=2.9009 lr=9.61e-07 tok=26.220B dt=11550 wait=327 +16:16:33 step 198980 loss=2.9285 lr=9.43e-07 tok=26.222B dt=11614 wait=327 +16:17:37 step 199000 loss=2.9326 lr=9.24e-07 tok=26.225B dt=11677 wait=327 +16:18:40 step 199020 loss=2.9562 lr=9.06e-07 tok=26.228B dt=11740 wait=327 +16:19:44 step 199040 loss=2.9058 lr=8.87e-07 tok=26.230B dt=11804 wait=327 +16:20:28 pushed ckpt_w197.pt,log_w197.txt +16:23:31 waited 182s for global r=874 +16:23:39 global r=874 load 4.281 -> 4.280 +16:24:13 step 199060 loss=2.9691 lr=8.69e-07 tok=26.233B dt=12073 wait=327 +16:25:16 step 199080 loss=2.9267 lr=8.51e-07 tok=26.235B dt=12136 wait=327 +16:26:19 step 199100 loss=2.8813 lr=8.32e-07 tok=26.238B dt=12200 wait=328 +16:27:23 step 199120 loss=2.9810 lr=8.14e-07 tok=26.241B dt=12264 wait=328 +16:28:27 step 199140 loss=2.9457 lr=7.95e-07 tok=26.243B dt=12327 wait=328 +16:29:30 step 199160 loss=2.9947 lr=7.77e-07 tok=26.246B dt=12391 wait=328 +16:30:34 step 199180 loss=3.0136 lr=7.58e-07 tok=26.248B dt=12454 wait=328 +16:31:38 step 199200 loss=2.9869 lr=7.40e-07 tok=26.251B dt=12518 wait=328 +16:31:51 pushed ckpt_w197.pt,log_w197.txt +16:33:37 waited 106s for global r=875 +16:33:45 global r=875 load 4.280 -> 4.281 +16:34:50 step 199220 loss=3.0249 lr=7.21e-07 tok=26.254B dt=12710 wait=329 +16:35:53 step 199240 loss=2.9796 lr=7.03e-07 tok=26.256B dt=12773 wait=329 +16:36:57 step 199260 loss=2.9120 lr=6.84e-07 tok=26.259B dt=12837 wait=329 +16:38:01 step 199280 loss=2.9461 lr=6.66e-07 tok=26.262B dt=12902 wait=329 +16:39:06 step 199300 loss=3.0047 lr=6.47e-07 tok=26.264B dt=12966 wait=329 +16:40:10 step 199320 loss=2.8749 lr=6.29e-07 tok=26.267B dt=13030 wait=329 +16:41:15 step 199340 loss=2.9693 lr=6.10e-07 tok=26.269B dt=13095 wait=329 +16:41:59 pushed ckpt_w197.pt,log_w197.txt +16:43:30 waited 91s for global r=876 +16:43:37 global r=876 load 4.281 -> 4.280 +16:44:10 step 199360 loss=3.0805 lr=5.92e-07 tok=26.272B dt=13270 wait=330 +16:45:14 step 199380 loss=2.9763 lr=5.73e-07 tok=26.275B dt=13334 wait=330 +16:46:18 step 199400 loss=2.9510 lr=5.55e-07 tok=26.277B dt=13398 wait=330 +16:47:22 step 199420 loss=3.0916 lr=5.37e-07 tok=26.280B dt=13463 wait=330 +16:48:26 step 199440 loss=2.8918 lr=5.18e-07 tok=26.283B dt=13527 wait=330 +16:49:30 step 199460 loss=2.9643 lr=5.00e-07 tok=26.285B dt=13590 wait=330 +16:50:35 step 199480 loss=2.5645 lr=4.81e-07 tok=26.288B dt=13656 wait=330 +16:51:40 step 199500 loss=2.9597 lr=4.63e-07 tok=26.290B dt=13720 wait=331 +16:51:54 pushed ckpt_w197.pt,log_w197.txt +16:55:38 global r=877 load 4.281 -> 4.281 +16:56:42 step 199520 loss=2.8766 lr=4.44e-07 tok=26.294B dt=14022 wait=331 +16:57:46 step 199540 loss=2.8963 lr=4.26e-07 tok=26.297B dt=14086 wait=331 +16:58:50 step 199560 loss=2.8556 lr=4.07e-07 tok=26.300B dt=14150 wait=331 +16:59:54 step 199580 loss=2.8172 lr=3.89e-07 tok=26.302B dt=14215 wait=331 +17:00:58 step 199600 loss=2.9515 lr=3.70e-07 tok=26.305B dt=14278 wait=332 +17:02:02 step 199620 loss=3.0418 lr=3.52e-07 tok=26.307B dt=14342 wait=332 +17:03:07 step 199640 loss=2.5956 lr=3.33e-07 tok=26.310B dt=14407 wait=332 +17:03:51 pushed ckpt_w197.pt,log_w197.txt +17:06:52 waited 181s for global r=878 +17:06:59 global r=878 load 4.280 -> 4.280 +17:07:32 step 199660 loss=2.9824 lr=3.15e-07 tok=26.313B dt=14673 wait=332 +17:08:36 step 199680 loss=2.9110 lr=2.96e-07 tok=26.315B dt=14736 wait=332 +17:09:39 step 199700 loss=2.9074 lr=2.78e-07 tok=26.318B dt=14799 wait=332 +17:10:42 step 199720 loss=3.0040 lr=2.59e-07 tok=26.321B dt=14862 wait=332 +17:11:46 step 199740 loss=2.9293 lr=2.41e-07 tok=26.323B dt=14926 wait=333 +17:12:49 step 199760 loss=2.9907 lr=2.23e-07 tok=26.326B dt=14989 wait=333 +17:13:53 step 199780 loss=2.9223 lr=2.04e-07 tok=26.328B dt=15054 wait=333 +17:14:58 step 199800 loss=2.8045 lr=1.86e-07 tok=26.331B dt=15118 wait=333 +17:15:10 pushed ckpt_w197.pt,log_w197.txt +17:18:56 global r=879 load 4.281 -> 4.280 +17:20:00 step 199820 loss=3.0064 lr=1.67e-07 tok=26.335B dt=15420 wait=333 +17:21:03 step 199840 loss=3.0673 lr=1.49e-07 tok=26.338B dt=15484 wait=334 +17:22:07 step 199860 loss=3.0447 lr=1.30e-07 tok=26.340B dt=15547 wait=334 +17:23:10 step 199880 loss=3.0828 lr=1.12e-07 tok=26.343B dt=15611 wait=334 +17:24:14 step 199900 loss=3.0535 lr=9.33e-08 tok=26.345B dt=15674 wait=334 +17:25:18 step 199920 loss=3.0919 lr=7.48e-08 tok=26.348B dt=15738 wait=334 +17:26:22 step 199940 loss=3.0704 lr=5.63e-08 tok=26.351B dt=15802 wait=334 +17:27:06 pushed ckpt_w197.pt,log_w197.txt +17:30:08 waited 181s for global r=880 +17:30:17 global r=880 load 4.280 -> 4.280 +17:30:51 step 199960 loss=3.0742 lr=3.79e-08 tok=26.353B dt=16071 wait=335 +17:31:55 step 199980 loss=2.9712 lr=1.94e-08 tok=26.356B dt=16136 wait=335 +17:32:58 step 200000 loss=2.9868 lr=9.23e-10 tok=26.359B dt=16199 wait=335 +17:34:03 step 200020 loss=3.1065 lr=0.00e+00 tok=26.361B dt=16263 wait=335 +17:35:06 step 200040 loss=2.9812 lr=0.00e+00 tok=26.364B dt=16326 wait=335 +17:36:09 step 200060 loss=3.1190 lr=0.00e+00 tok=26.366B dt=16390 wait=335 +17:37:13 step 200080 loss=2.9930 lr=0.00e+00 tok=26.369B dt=16453 wait=335 +17:38:17 step 200100 loss=2.9632 lr=0.00e+00 tok=26.372B dt=16517 wait=336 diff --git a/log_w198.txt b/log_w198.txt new file mode 100644 index 0000000000000000000000000000000000000000..9e63dafcb1fb2dd7f9906b7c7ac910f8c8afbe3c --- /dev/null +++ b/log_w198.txt @@ -0,0 +1,180 @@ +launch +13:02:22 name=w198 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 145/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:02:28 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:02:28 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:02:38 params 1.08B +13:02:39 resumed step=196050 seen=25.85B round=854 +13:02:39 stream salt=196050 +13:04:29 blocks compiled +13:04:29 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:09:16 step 196051 loss=2.7829 lr=3.65e-06 tok=25.846B dt=287 wait=261 +13:09:32 step 196052 loss=2.7383 lr=3.65e-06 tok=25.846B dt=303 wait=261 +13:09:57 step 196060 loss=3.0610 lr=3.64e-06 tok=25.847B dt=328 wait=261 +13:11:03 step 196080 loss=2.8470 lr=3.62e-06 tok=25.850B dt=394 wait=261 +13:12:09 step 196100 loss=2.7922 lr=3.60e-06 tok=25.853B dt=460 wait=261 +13:12:25 global r=855 load 4.282 -> 4.281 +13:13:29 step 196220 loss=2.8726 lr=3.49e-06 tok=25.855B dt=541 wait=261 +13:14:34 step 196240 loss=2.9634 lr=3.47e-06 tok=25.858B dt=605 wait=261 +13:15:40 step 196260 loss=2.8536 lr=3.45e-06 tok=25.861B dt=671 wait=261 +13:16:45 step 196280 loss=2.8353 lr=3.44e-06 tok=25.863B dt=736 wait=262 +13:17:51 step 196300 loss=2.7795 lr=3.42e-06 tok=25.866B dt=802 wait=262 +13:18:56 step 196320 loss=2.8333 lr=3.40e-06 tok=25.868B dt=867 wait=262 +13:20:01 step 196340 loss=2.8186 lr=3.38e-06 tok=25.871B dt=932 wait=262 +13:20:50 pushed ckpt_w198.pt,log_w198.txt +13:21:51 waited 61s for global r=856 +13:21:58 global r=856 load 4.281 -> 4.281 +13:22:33 step 196360 loss=2.9209 lr=3.36e-06 tok=25.874B dt=1084 wait=262 +13:23:39 step 196380 loss=2.8563 lr=3.34e-06 tok=25.876B dt=1150 wait=262 +13:24:44 step 196400 loss=2.8219 lr=3.33e-06 tok=25.879B dt=1215 wait=263 +13:25:50 step 196420 loss=2.8707 lr=3.31e-06 tok=25.881B dt=1281 wait=263 +13:26:56 step 196440 loss=2.9721 lr=3.29e-06 tok=25.884B dt=1347 wait=263 +13:28:01 step 196460 loss=3.0332 lr=3.27e-06 tok=25.887B dt=1412 wait=263 +13:29:07 step 196480 loss=2.9035 lr=3.25e-06 tok=25.889B dt=1478 wait=263 +13:30:12 step 196500 loss=2.9515 lr=3.23e-06 tok=25.892B dt=1543 wait=263 +13:30:24 pushed ckpt_w198.pt,log_w198.txt +13:31:40 waited 76s for global r=857 +13:31:47 global r=857 load 4.281 -> 4.281 +13:32:53 step 196520 loss=2.9626 lr=3.21e-06 tok=25.895B dt=1704 wait=263 +13:33:58 step 196540 loss=2.9564 lr=3.20e-06 tok=25.897B dt=1769 wait=264 +13:35:03 step 196560 loss=3.0067 lr=3.18e-06 tok=25.900B dt=1834 wait=264 +13:36:09 step 196580 loss=2.9015 lr=3.16e-06 tok=25.902B dt=1900 wait=264 +13:37:14 step 196600 loss=2.9819 lr=3.14e-06 tok=25.905B dt=1965 wait=264 +13:38:19 step 196620 loss=2.9562 lr=3.12e-06 tok=25.908B dt=2030 wait=264 +13:39:25 step 196640 loss=2.7809 lr=3.10e-06 tok=25.910B dt=2096 wait=264 +13:40:10 pushed ckpt_w198.pt,log_w198.txt +13:41:26 waited 76s for global r=858 +13:41:34 global r=858 load 4.280 -> 4.280 +13:42:08 step 196660 loss=2.9413 lr=3.09e-06 tok=25.913B dt=2259 wait=265 +13:43:14 step 196680 loss=2.9400 lr=3.07e-06 tok=25.916B dt=2325 wait=265 +13:44:19 step 196700 loss=2.8635 lr=3.05e-06 tok=25.918B dt=2390 wait=265 +13:45:24 step 196720 loss=2.9134 lr=3.03e-06 tok=25.921B dt=2455 wait=265 +13:46:30 step 196740 loss=2.8382 lr=3.01e-06 tok=25.923B dt=2521 wait=265 +13:47:35 step 196760 loss=2.9207 lr=2.99e-06 tok=25.926B dt=2586 wait=265 +13:48:40 step 196780 loss=2.8066 lr=2.97e-06 tok=25.929B dt=2651 wait=265 +13:49:46 step 196800 loss=2.8131 lr=2.96e-06 tok=25.931B dt=2717 wait=266 +13:49:58 pushed ckpt_w198.pt,log_w198.txt +13:51:13 waited 76s for global r=859 +13:51:25 global r=859 load 4.281 -> 4.281 +13:52:32 step 196820 loss=2.8085 lr=2.94e-06 tok=25.934B dt=2883 wait=266 +13:53:37 step 196840 loss=2.8446 lr=2.92e-06 tok=25.937B dt=2948 wait=266 +13:54:43 step 196860 loss=2.7985 lr=2.90e-06 tok=25.939B dt=3014 wait=266 +13:55:50 step 196880 loss=2.7721 lr=2.88e-06 tok=25.942B dt=3081 wait=266 +13:56:55 step 196900 loss=2.8171 lr=2.86e-06 tok=25.944B dt=3146 wait=266 +13:58:01 step 196920 loss=2.8999 lr=2.85e-06 tok=25.947B dt=3212 wait=267 +13:59:07 step 196940 loss=2.8208 lr=2.83e-06 tok=25.950B dt=3278 wait=267 +13:59:53 pushed ckpt_w198.pt,log_w198.txt +14:01:09 waited 76s for global r=860 +14:01:15 global r=860 load 4.281 -> 4.281 +14:01:50 step 196960 loss=2.8417 lr=2.81e-06 tok=25.952B dt=3441 wait=267 +14:02:56 step 196980 loss=2.8452 lr=2.79e-06 tok=25.955B dt=3507 wait=267 +14:04:02 step 197000 loss=2.8082 lr=2.77e-06 tok=25.957B dt=3573 wait=267 +14:05:07 step 197020 loss=2.9986 lr=2.75e-06 tok=25.960B dt=3639 wait=267 +14:06:14 step 197040 loss=2.9128 lr=2.73e-06 tok=25.963B dt=3705 wait=267 +14:07:20 step 197060 loss=3.0267 lr=2.72e-06 tok=25.965B dt=3771 wait=268 +14:08:25 step 197080 loss=2.8623 lr=2.70e-06 tok=25.968B dt=3837 wait=268 +14:09:31 step 197100 loss=2.9438 lr=2.68e-06 tok=25.971B dt=3902 wait=268 +14:09:47 pushed ckpt_w198.pt,log_w198.txt +14:11:03 waited 76s for global r=861 +14:11:14 global r=861 load 4.281 -> 4.280 +14:12:22 step 197120 loss=2.8802 lr=2.66e-06 tok=25.973B dt=4073 wait=268 +14:13:29 step 197140 loss=2.5864 lr=2.64e-06 tok=25.976B dt=4140 wait=268 +14:14:35 step 197160 loss=2.8920 lr=2.62e-06 tok=25.978B dt=4206 wait=268 +14:15:41 step 197180 loss=2.8357 lr=2.61e-06 tok=25.981B dt=4272 wait=269 +14:16:47 step 197200 loss=2.8537 lr=2.59e-06 tok=25.984B dt=4338 wait=269 +14:17:53 step 197220 loss=2.8925 lr=2.57e-06 tok=25.986B dt=4404 wait=269 +14:18:59 step 197240 loss=2.6120 lr=2.55e-06 tok=25.989B dt=4470 wait=269 +14:19:46 pushed ckpt_w198.pt,log_w198.txt +14:20:46 waited 60s for global r=862 +14:20:54 global r=862 load 4.280 -> 4.281 +14:21:29 step 197260 loss=2.9523 lr=2.53e-06 tok=25.992B dt=4620 wait=269 +14:22:35 step 197280 loss=2.7097 lr=2.51e-06 tok=25.994B dt=4686 wait=269 +14:23:41 step 197300 loss=2.7537 lr=2.49e-06 tok=25.997B dt=4753 wait=269 +14:24:47 step 197320 loss=2.8422 lr=2.48e-06 tok=25.999B dt=4818 wait=270 +14:25:53 step 197340 loss=2.7302 lr=2.46e-06 tok=26.002B dt=4885 wait=270 +14:27:00 step 197360 loss=2.7664 lr=2.44e-06 tok=26.005B dt=4951 wait=270 +14:28:05 step 197380 loss=2.8441 lr=2.42e-06 tok=26.007B dt=5016 wait=270 +14:29:12 step 197400 loss=2.9476 lr=2.40e-06 tok=26.010B dt=5083 wait=270 +14:29:25 pushed ckpt_w198.pt,log_w198.txt +14:30:25 waited 61s for global r=863 +14:30:33 global r=863 load 4.280 -> 4.280 +14:31:42 step 197420 loss=2.8506 lr=2.38e-06 tok=26.013B dt=5233 wait=270 +14:32:49 step 197440 loss=3.0089 lr=2.37e-06 tok=26.015B dt=5300 wait=271 +14:33:55 step 197460 loss=2.8695 lr=2.35e-06 tok=26.018B dt=5366 wait=271 +14:35:01 step 197480 loss=2.8155 lr=2.33e-06 tok=26.020B dt=5432 wait=271 +14:36:06 step 197500 loss=2.8576 lr=2.31e-06 tok=26.023B dt=5497 wait=271 +14:37:12 step 197520 loss=2.9345 lr=2.29e-06 tok=26.026B dt=5563 wait=271 +14:38:17 step 197540 loss=2.7563 lr=2.27e-06 tok=26.028B dt=5628 wait=271 +14:39:04 pushed ckpt_w198.pt,log_w198.txt +14:40:20 waited 76s for global r=864 +14:40:30 global r=864 load 4.280 -> 4.280 +14:41:04 step 197560 loss=2.8546 lr=2.25e-06 tok=26.031B dt=5795 wait=272 +14:42:09 step 197580 loss=2.8161 lr=2.24e-06 tok=26.034B dt=5860 wait=272 +14:43:14 step 197600 loss=2.8534 lr=2.22e-06 tok=26.036B dt=5925 wait=272 +14:44:19 step 197620 loss=2.8474 lr=2.20e-06 tok=26.039B dt=5990 wait=272 +14:45:23 step 197640 loss=2.9097 lr=2.18e-06 tok=26.041B dt=6055 wait=272 +14:46:28 step 197660 loss=2.9054 lr=2.16e-06 tok=26.044B dt=6120 wait=272 +14:47:32 step 197680 loss=2.9175 lr=2.14e-06 tok=26.047B dt=6183 wait=272 +14:48:37 step 197700 loss=2.8838 lr=2.12e-06 tok=26.049B dt=6248 wait=273 +14:48:53 pushed ckpt_w198.pt,log_w198.txt +14:50:09 waited 76s for global r=865 +14:50:16 global r=865 load 4.281 -> 4.281 +14:51:21 step 197720 loss=2.8228 lr=2.11e-06 tok=26.052B dt=6412 wait=273 +14:52:26 step 197740 loss=2.9149 lr=2.09e-06 tok=26.054B dt=6477 wait=273 +14:53:29 step 197760 loss=2.9386 lr=2.07e-06 tok=26.057B dt=6540 wait=273 +14:54:34 step 197780 loss=2.8967 lr=2.05e-06 tok=26.060B dt=6605 wait=273 +14:55:38 step 197800 loss=2.8793 lr=2.03e-06 tok=26.062B dt=6669 wait=273 +14:56:42 step 197820 loss=3.0827 lr=2.01e-06 tok=26.065B dt=6733 wait=273 +14:57:47 step 197840 loss=2.8776 lr=2.00e-06 tok=26.068B dt=6798 wait=273 +14:58:33 pushed ckpt_w198.pt,log_w198.txt +14:59:49 waited 76s for global r=866 +14:59:55 global r=866 load 4.281 -> 4.280 +15:00:29 step 197860 loss=3.0158 lr=1.98e-06 tok=26.070B dt=6960 wait=274 +15:01:35 step 197880 loss=2.8665 lr=1.96e-06 tok=26.073B dt=7026 wait=274 +15:02:39 step 197900 loss=2.9484 lr=1.94e-06 tok=26.075B dt=7090 wait=274 +15:03:44 step 197920 loss=3.1206 lr=1.92e-06 tok=26.078B dt=7155 wait=274 +15:04:49 step 197940 loss=2.8903 lr=1.90e-06 tok=26.081B dt=7220 wait=274 +15:05:54 step 197960 loss=2.9782 lr=1.88e-06 tok=26.083B dt=7285 wait=274 +15:06:58 step 197980 loss=2.9316 lr=1.87e-06 tok=26.086B dt=7349 wait=275 +15:08:03 step 198000 loss=2.9755 lr=1.85e-06 tok=26.089B dt=7414 wait=275 +15:08:15 pushed ckpt_w198.pt,log_w198.txt +15:09:31 waited 76s for global r=867 +15:09:37 global r=867 load 4.281 -> 4.280 +15:10:43 step 198020 loss=3.0496 lr=1.83e-06 tok=26.091B dt=7574 wait=275 +15:11:48 step 198040 loss=3.0767 lr=1.81e-06 tok=26.094B dt=7639 wait=275 +15:12:53 step 198060 loss=2.9683 lr=1.79e-06 tok=26.096B dt=7704 wait=275 +15:13:57 step 198080 loss=2.9402 lr=1.77e-06 tok=26.099B dt=7768 wait=275 +15:15:02 step 198100 loss=3.0299 lr=1.76e-06 tok=26.102B dt=7833 wait=276 +15:16:06 step 198120 loss=3.1752 lr=1.74e-06 tok=26.104B dt=7897 wait=276 +15:17:10 step 198140 loss=2.9338 lr=1.72e-06 tok=26.107B dt=7962 wait=276 +15:17:55 pushed ckpt_w198.pt,log_w198.txt +15:19:11 waited 76s for global r=868 +15:19:17 global r=868 load 4.281 -> 4.280 +15:19:50 step 198160 loss=3.0611 lr=1.70e-06 tok=26.110B dt=8121 wait=276 +15:20:55 step 198180 loss=2.9559 lr=1.68e-06 tok=26.112B dt=8186 wait=276 +15:21:58 step 198200 loss=2.9743 lr=1.66e-06 tok=26.115B dt=8250 wait=276 +15:23:03 step 198220 loss=3.0476 lr=1.64e-06 tok=26.117B dt=8314 wait=277 +15:24:08 step 198240 loss=2.9728 lr=1.63e-06 tok=26.120B dt=8379 wait=277 +15:25:12 step 198260 loss=2.9760 lr=1.61e-06 tok=26.123B dt=8443 wait=277 +15:26:17 step 198280 loss=3.0210 lr=1.59e-06 tok=26.125B dt=8508 wait=277 +15:27:21 step 198300 loss=2.9090 lr=1.57e-06 tok=26.128B dt=8572 wait=277 +15:27:36 pushed ckpt_w198.pt,log_w198.txt +15:28:36 waited 60s for global r=869 +15:28:43 global r=869 load 4.281 -> 4.281 +15:29:49 step 198320 loss=2.8838 lr=1.55e-06 tok=26.131B dt=8720 wait=277 +15:30:54 step 198340 loss=2.9203 lr=1.53e-06 tok=26.133B dt=8785 wait=277 +15:31:58 step 198360 loss=2.9161 lr=1.52e-06 tok=26.136B dt=8849 wait=278 +15:33:03 step 198380 loss=2.8524 lr=1.50e-06 tok=26.138B dt=8914 wait=278 +15:34:09 step 198400 loss=2.8364 lr=1.48e-06 tok=26.141B dt=8980 wait=278 +15:35:13 step 198420 loss=3.0207 lr=1.46e-06 tok=26.144B dt=9044 wait=278 +15:36:18 step 198440 loss=2.8260 lr=1.44e-06 tok=26.146B dt=9109 wait=278 +15:37:03 pushed ckpt_w198.pt,log_w198.txt +15:38:19 waited 76s for global r=870 +15:38:26 global r=870 load 4.281 -> 4.281 +15:39:01 step 198460 loss=3.0355 lr=1.42e-06 tok=26.149B dt=9272 wait=278 +15:40:06 step 198480 loss=3.0105 lr=1.40e-06 tok=26.151B dt=9337 wait=278 +15:41:11 step 198500 loss=2.8749 lr=1.39e-06 tok=26.154B dt=9402 wait=279 +15:42:16 step 198520 loss=2.8964 lr=1.37e-06 tok=26.157B dt=9467 wait=279 +15:43:20 step 198540 loss=2.9457 lr=1.35e-06 tok=26.159B dt=9531 wait=279 +15:44:24 step 198560 loss=3.0688 lr=1.33e-06 tok=26.162B dt=9595 wait=279 +15:45:28 step 198580 loss=2.9462 lr=1.31e-06 tok=26.165B dt=9659 wait=279 +15:46:33 step 198600 loss=2.9359 lr=1.29e-06 tok=26.167B dt=9724 wait=279 diff --git a/log_w199.txt b/log_w199.txt new file mode 100644 index 0000000000000000000000000000000000000000..78298567e8ff73c208d338b43184516610457810 --- /dev/null +++ b/log_w199.txt @@ -0,0 +1,178 @@ +launch +13:03:59 name=w199 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 146/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:04:05 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:04:05 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +13:04:15 params 1.08B +13:04:16 resumed step=196050 seen=25.85B round=854 +13:04:16 stream salt=196050 +13:06:08 blocks compiled +13:06:08 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +13:10:36 step 196051 loss=2.7282 lr=3.65e-06 tok=25.846B dt=268 wait=240 +13:10:53 step 196052 loss=2.8266 lr=3.65e-06 tok=25.846B dt=284 wait=240 +13:11:19 step 196060 loss=3.1826 lr=3.64e-06 tok=25.847B dt=310 wait=240 +13:12:08 global r=855 load 4.282 -> 4.281 +13:13:14 step 196220 loss=2.8639 lr=3.49e-06 tok=25.851B dt=425 wait=240 +13:14:21 step 196240 loss=2.4399 lr=3.47e-06 tok=25.854B dt=492 wait=240 +13:15:26 step 196260 loss=2.9075 lr=3.45e-06 tok=25.857B dt=558 wait=240 +13:16:32 step 196280 loss=2.9241 lr=3.44e-06 tok=25.859B dt=623 wait=241 +13:17:38 step 196300 loss=2.8871 lr=3.42e-06 tok=25.862B dt=689 wait=241 +13:18:44 step 196320 loss=2.9714 lr=3.40e-06 tok=25.864B dt=756 wait=241 +13:19:49 step 196340 loss=2.8977 lr=3.38e-06 tok=25.867B dt=821 wait=241 +13:20:36 pushed ckpt_w199.pt,log_w199.txt +13:21:51 waited 76s for global r=856 +13:21:59 global r=856 load 4.281 -> 4.281 +13:22:35 step 196360 loss=2.9050 lr=3.36e-06 tok=25.870B dt=986 wait=241 +13:23:40 step 196380 loss=2.9406 lr=3.34e-06 tok=25.872B dt=1052 wait=241 +13:24:47 step 196400 loss=2.9250 lr=3.33e-06 tok=25.875B dt=1118 wait=241 +13:25:53 step 196420 loss=2.9344 lr=3.31e-06 tok=25.878B dt=1184 wait=242 +13:26:59 step 196440 loss=2.8704 lr=3.29e-06 tok=25.880B dt=1251 wait=242 +13:28:05 step 196460 loss=2.9506 lr=3.27e-06 tok=25.883B dt=1317 wait=242 +13:29:10 step 196480 loss=2.8906 lr=3.25e-06 tok=25.885B dt=1382 wait=242 +13:30:16 step 196500 loss=2.9181 lr=3.23e-06 tok=25.888B dt=1447 wait=242 +13:30:30 pushed ckpt_w199.pt,log_w199.txt +13:31:46 waited 76s for global r=857 +13:31:53 global r=857 load 4.282 -> 4.281 +13:33:00 step 196520 loss=2.8699 lr=3.21e-06 tok=25.891B dt=1611 wait=242 +13:34:05 step 196540 loss=2.9569 lr=3.20e-06 tok=25.893B dt=1677 wait=242 +13:35:11 step 196560 loss=2.9264 lr=3.18e-06 tok=25.896B dt=1742 wait=243 +13:36:15 step 196580 loss=2.9584 lr=3.16e-06 tok=25.899B dt=1807 wait=243 +13:37:21 step 196600 loss=2.8819 lr=3.14e-06 tok=25.901B dt=1872 wait=243 +13:38:27 step 196620 loss=3.0411 lr=3.12e-06 tok=25.904B dt=1938 wait=243 +13:39:32 step 196640 loss=2.8601 lr=3.10e-06 tok=25.906B dt=2003 wait=243 +13:40:17 pushed ckpt_w199.pt,log_w199.txt +13:41:17 waited 61s for global r=858 +13:41:28 global r=858 load 4.281 -> 4.280 +13:42:02 step 196660 loss=2.9997 lr=3.09e-06 tok=25.909B dt=2154 wait=243 +13:43:08 step 196680 loss=2.9001 lr=3.07e-06 tok=25.912B dt=2220 wait=243 +13:44:14 step 196700 loss=2.9487 lr=3.05e-06 tok=25.914B dt=2286 wait=244 +13:45:20 step 196720 loss=3.0034 lr=3.03e-06 tok=25.917B dt=2352 wait=244 +13:46:26 step 196740 loss=2.8689 lr=3.01e-06 tok=25.919B dt=2418 wait=244 +13:47:32 step 196760 loss=2.9935 lr=2.99e-06 tok=25.922B dt=2484 wait=244 +13:48:38 step 196780 loss=3.1560 lr=2.97e-06 tok=25.925B dt=2549 wait=244 +13:49:43 step 196800 loss=3.1107 lr=2.96e-06 tok=25.927B dt=2615 wait=244 +13:49:55 pushed ckpt_w199.pt,log_w199.txt +13:51:26 waited 91s for global r=859 +13:51:35 global r=859 load 4.281 -> 4.281 +13:52:42 step 196820 loss=3.0872 lr=2.94e-06 tok=25.930B dt=2794 wait=244 +13:53:47 step 196840 loss=3.1797 lr=2.92e-06 tok=25.933B dt=2859 wait=245 +13:54:53 step 196860 loss=3.0592 lr=2.90e-06 tok=25.935B dt=2925 wait=245 +13:55:58 step 196880 loss=3.2773 lr=2.88e-06 tok=25.938B dt=2990 wait=245 +13:57:05 step 196900 loss=3.2213 lr=2.86e-06 tok=25.940B dt=3056 wait=245 +13:58:10 step 196920 loss=3.3745 lr=2.85e-06 tok=25.943B dt=3121 wait=245 +13:59:15 step 196940 loss=3.1354 lr=2.83e-06 tok=25.946B dt=3186 wait=245 +14:00:02 pushed ckpt_w199.pt,log_w199.txt +14:01:18 waited 76s for global r=860 +14:01:25 global r=860 load 4.280 -> 4.281 +14:01:59 step 196960 loss=3.3609 lr=2.81e-06 tok=25.948B dt=3351 wait=245 +14:03:03 step 196980 loss=3.1034 lr=2.79e-06 tok=25.951B dt=3415 wait=246 +14:04:09 step 197000 loss=3.2554 lr=2.77e-06 tok=25.954B dt=3480 wait=246 +14:05:13 step 197020 loss=3.3397 lr=2.75e-06 tok=25.956B dt=3545 wait=246 +14:06:17 step 197040 loss=3.1153 lr=2.73e-06 tok=25.959B dt=3608 wait=246 +14:07:21 step 197060 loss=2.9528 lr=2.72e-06 tok=25.961B dt=3673 wait=246 +14:08:25 step 197080 loss=3.0987 lr=2.70e-06 tok=25.964B dt=3736 wait=246 +14:09:28 step 197100 loss=3.1513 lr=2.68e-06 tok=25.967B dt=3800 wait=246 +14:09:42 pushed ckpt_w199.pt,log_w199.txt +14:10:58 waited 76s for global r=861 +14:11:06 global r=861 load 4.280 -> 4.280 +14:12:10 step 197120 loss=3.1716 lr=2.66e-06 tok=25.969B dt=3962 wait=247 +14:13:15 step 197140 loss=3.0613 lr=2.64e-06 tok=25.972B dt=4026 wait=247 +14:14:18 step 197160 loss=3.2872 lr=2.62e-06 tok=25.975B dt=4089 wait=247 +14:15:21 step 197180 loss=3.0734 lr=2.61e-06 tok=25.977B dt=4153 wait=247 +14:16:24 step 197200 loss=3.0969 lr=2.59e-06 tok=25.980B dt=4215 wait=247 +14:17:27 step 197220 loss=3.0850 lr=2.57e-06 tok=25.982B dt=4279 wait=247 +14:18:31 step 197240 loss=3.0768 lr=2.55e-06 tok=25.985B dt=4342 wait=247 +14:19:14 pushed ckpt_w199.pt,log_w199.txt +14:20:44 waited 91s for global r=862 +14:20:52 global r=862 load 4.280 -> 4.281 +14:21:25 step 197260 loss=3.1520 lr=2.53e-06 tok=25.988B dt=4516 wait=248 +14:22:28 step 197280 loss=3.0244 lr=2.51e-06 tok=25.990B dt=4579 wait=248 +14:23:32 step 197300 loss=2.8164 lr=2.49e-06 tok=25.993B dt=4644 wait=248 +14:24:37 step 197320 loss=3.1143 lr=2.48e-06 tok=25.996B dt=4708 wait=248 +14:25:40 step 197340 loss=2.9923 lr=2.46e-06 tok=25.998B dt=4772 wait=248 +14:26:44 step 197360 loss=3.1483 lr=2.44e-06 tok=26.001B dt=4836 wait=248 +14:27:47 step 197380 loss=3.1837 lr=2.42e-06 tok=26.003B dt=4899 wait=249 +14:28:52 step 197400 loss=3.1666 lr=2.40e-06 tok=26.006B dt=4964 wait=249 +14:29:06 pushed ckpt_w199.pt,log_w199.txt +14:30:36 waited 91s for global r=863 +14:30:42 global r=863 load 4.280 -> 4.280 +14:31:47 step 197420 loss=3.1068 lr=2.38e-06 tok=26.009B dt=5139 wait=249 +14:32:51 step 197440 loss=3.1908 lr=2.37e-06 tok=26.011B dt=5203 wait=249 +14:33:57 step 197460 loss=2.8181 lr=2.35e-06 tok=26.014B dt=5269 wait=249 +14:35:01 step 197480 loss=3.2280 lr=2.33e-06 tok=26.016B dt=5333 wait=249 +14:36:05 step 197500 loss=3.1664 lr=2.31e-06 tok=26.019B dt=5397 wait=249 +14:37:09 step 197520 loss=3.3035 lr=2.29e-06 tok=26.022B dt=5460 wait=250 +14:38:13 step 197540 loss=3.0235 lr=2.27e-06 tok=26.024B dt=5524 wait=250 +14:38:59 pushed ckpt_w199.pt,log_w199.txt +14:40:15 waited 76s for global r=864 +14:40:25 global r=864 load 4.280 -> 4.280 +14:40:59 step 197560 loss=3.2746 lr=2.25e-06 tok=26.027B dt=5691 wait=250 +14:42:04 step 197580 loss=3.1032 lr=2.24e-06 tok=26.030B dt=5755 wait=250 +14:43:08 step 197600 loss=3.0859 lr=2.22e-06 tok=26.032B dt=5820 wait=250 +14:44:13 step 197620 loss=3.3131 lr=2.20e-06 tok=26.035B dt=5884 wait=250 +14:45:17 step 197640 loss=3.0148 lr=2.18e-06 tok=26.037B dt=5949 wait=250 +14:46:21 step 197660 loss=3.1623 lr=2.16e-06 tok=26.040B dt=6013 wait=251 +14:47:25 step 197680 loss=3.0250 lr=2.14e-06 tok=26.043B dt=6077 wait=251 +14:48:28 step 197700 loss=2.9499 lr=2.12e-06 tok=26.045B dt=6140 wait=251 +14:48:43 pushed ckpt_w199.pt,log_w199.txt +14:50:14 waited 91s for global r=865 +14:50:20 global r=865 load 4.280 -> 4.281 +14:51:25 step 197720 loss=2.9811 lr=2.11e-06 tok=26.048B dt=6317 wait=251 +14:52:28 step 197740 loss=3.0459 lr=2.09e-06 tok=26.051B dt=6380 wait=251 +14:53:32 step 197760 loss=2.9975 lr=2.07e-06 tok=26.053B dt=6444 wait=251 +14:54:36 step 197780 loss=3.0215 lr=2.05e-06 tok=26.056B dt=6507 wait=251 +14:55:39 step 197800 loss=3.0074 lr=2.03e-06 tok=26.058B dt=6571 wait=252 +14:56:44 step 197820 loss=3.0750 lr=2.01e-06 tok=26.061B dt=6635 wait=252 +14:57:47 step 197840 loss=2.9085 lr=2.00e-06 tok=26.064B dt=6699 wait=252 +14:58:33 pushed ckpt_w199.pt,log_w199.txt +14:59:48 waited 76s for global r=866 +14:59:54 global r=866 load 4.280 -> 4.280 +15:00:28 step 197860 loss=3.0508 lr=1.98e-06 tok=26.066B dt=6860 wait=252 +15:01:33 step 197880 loss=2.8150 lr=1.96e-06 tok=26.069B dt=6924 wait=252 +15:02:38 step 197900 loss=2.9740 lr=1.94e-06 tok=26.072B dt=6990 wait=252 +15:03:43 step 197920 loss=3.0896 lr=1.92e-06 tok=26.074B dt=7054 wait=252 +15:04:47 step 197940 loss=3.0836 lr=1.90e-06 tok=26.077B dt=7119 wait=253 +15:05:51 step 197960 loss=3.1671 lr=1.88e-06 tok=26.079B dt=7183 wait=253 +15:06:56 step 197980 loss=2.8428 lr=1.87e-06 tok=26.082B dt=7248 wait=253 +15:08:01 step 198000 loss=3.1506 lr=1.85e-06 tok=26.085B dt=7312 wait=253 +15:08:14 pushed ckpt_w199.pt,log_w199.txt +15:09:29 waited 76s for global r=867 +15:09:37 global r=867 load 4.280 -> 4.280 +15:10:42 step 198020 loss=3.1018 lr=1.83e-06 tok=26.087B dt=7474 wait=253 +15:11:46 step 198040 loss=3.2391 lr=1.81e-06 tok=26.090B dt=7538 wait=253 +15:12:50 step 198060 loss=3.0948 lr=1.79e-06 tok=26.093B dt=7602 wait=254 +15:13:54 step 198080 loss=3.1765 lr=1.77e-06 tok=26.095B dt=7666 wait=254 +15:14:58 step 198100 loss=3.1657 lr=1.76e-06 tok=26.098B dt=7729 wait=254 +15:16:02 step 198120 loss=3.2318 lr=1.74e-06 tok=26.100B dt=7793 wait=254 +15:17:06 step 198140 loss=3.0443 lr=1.72e-06 tok=26.103B dt=7857 wait=254 +15:17:49 pushed ckpt_w199.pt,log_w199.txt +15:19:05 waited 76s for global r=868 +15:19:13 global r=868 load 4.280 -> 4.280 +15:19:47 step 198160 loss=3.1171 lr=1.70e-06 tok=26.106B dt=8018 wait=254 +15:20:50 step 198180 loss=3.0381 lr=1.68e-06 tok=26.108B dt=8082 wait=254 +15:21:53 step 198200 loss=2.9775 lr=1.66e-06 tok=26.111B dt=8145 wait=254 +15:22:57 step 198220 loss=3.0594 lr=1.64e-06 tok=26.113B dt=8209 wait=255 +15:24:01 step 198240 loss=2.8550 lr=1.63e-06 tok=26.116B dt=8272 wait=255 +15:25:04 step 198260 loss=2.7991 lr=1.61e-06 tok=26.119B dt=8336 wait=255 +15:26:08 step 198280 loss=3.0065 lr=1.59e-06 tok=26.121B dt=8399 wait=255 +15:27:12 step 198300 loss=3.0452 lr=1.57e-06 tok=26.124B dt=8464 wait=255 +15:27:27 pushed ckpt_w199.pt,log_w199.txt +15:28:42 waited 76s for global r=869 +15:28:50 global r=869 load 4.280 -> 4.281 +15:29:56 step 198320 loss=2.9664 lr=1.55e-06 tok=26.127B dt=8627 wait=255 +15:31:00 step 198340 loss=3.0134 lr=1.53e-06 tok=26.129B dt=8692 wait=255 +15:32:04 step 198360 loss=2.9233 lr=1.52e-06 tok=26.132B dt=8755 wait=256 +15:33:07 step 198380 loss=2.9156 lr=1.50e-06 tok=26.134B dt=8819 wait=256 +15:34:11 step 198400 loss=2.8490 lr=1.48e-06 tok=26.137B dt=8883 wait=256 +15:35:15 step 198420 loss=2.9716 lr=1.46e-06 tok=26.140B dt=8947 wait=256 +15:36:18 step 198440 loss=2.8579 lr=1.44e-06 tok=26.142B dt=9010 wait=256 +15:37:02 pushed ckpt_w199.pt,log_w199.txt +15:38:18 waited 76s for global r=870 +15:38:24 global r=870 load 4.281 -> 4.281 +15:38:57 step 198460 loss=2.9583 lr=1.42e-06 tok=26.145B dt=9169 wait=256 +15:40:00 step 198480 loss=2.9243 lr=1.40e-06 tok=26.148B dt=9232 wait=256 +15:41:04 step 198500 loss=2.9363 lr=1.39e-06 tok=26.150B dt=9296 wait=257 +15:42:08 step 198520 loss=2.9911 lr=1.37e-06 tok=26.153B dt=9359 wait=257 +15:43:11 step 198540 loss=2.9083 lr=1.35e-06 tok=26.155B dt=9423 wait=257 +15:44:16 step 198560 loss=2.9687 lr=1.33e-06 tok=26.158B dt=9487 wait=257 +15:45:19 step 198580 loss=2.9619 lr=1.31e-06 tok=26.161B dt=9551 wait=257 +15:46:23 step 198600 loss=2.9669 lr=1.29e-06 tok=26.163B dt=9615 wait=257 diff --git a/log_w2.txt b/log_w2.txt new file mode 100644 index 0000000000000000000000000000000000000000..4ca07100588707003fb3046f3666ec913fb20b6d --- /dev/null +++ b/log_w2.txt @@ -0,0 +1,37 @@ +launch +13:22:49 name=w2 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:22:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:23:03 params 1.08B +13:23:05 resumed step=25200 seen=3.30B round=13 +13:24:59 start: accum=4, 131072 tokens per optimiser step +13:27:54 step 25220 loss=3.7965 lr=2.90e-04 tok=3.306B dt=175 +13:29:37 step 25240 loss=3.9621 lr=2.90e-04 tok=3.308B dt=278 +13:31:18 step 25260 loss=3.8722 lr=2.90e-04 tok=3.311B dt=379 +13:33:00 step 25280 loss=3.9282 lr=2.90e-04 tok=3.314B dt=481 +13:34:42 step 25300 loss=3.7706 lr=2.90e-04 tok=3.316B dt=583 +13:36:25 step 25320 loss=3.8383 lr=2.90e-04 tok=3.319B dt=686 +13:38:09 step 25340 loss=3.8428 lr=2.90e-04 tok=3.321B dt=790 +13:39:52 step 25360 loss=3.8209 lr=2.90e-04 tok=3.324B dt=893 +13:41:35 step 25380 loss=3.7460 lr=2.90e-04 tok=3.327B dt=996 +13:43:18 step 25400 loss=3.8449 lr=2.90e-04 tok=3.329B dt=1099 +13:45:01 step 25420 loss=3.8120 lr=2.90e-04 tok=3.332B dt=1202 +13:46:43 step 25440 loss=3.7848 lr=2.90e-04 tok=3.334B dt=1303 +13:48:25 step 25460 loss=3.8205 lr=2.90e-04 tok=3.337B dt=1406 +13:50:07 step 25480 loss=3.8054 lr=2.90e-04 tok=3.340B dt=1508 +13:51:49 step 25500 loss=3.8699 lr=2.90e-04 tok=3.342B dt=1610 +13:53:31 step 25520 loss=3.9570 lr=2.90e-04 tok=3.345B dt=1712 +13:55:13 step 25540 loss=3.8982 lr=2.90e-04 tok=3.348B dt=1814 +13:56:56 step 25560 loss=3.8311 lr=2.90e-04 tok=3.350B dt=1917 +13:58:39 step 25580 loss=3.8899 lr=2.90e-04 tok=3.353B dt=2020 +14:00:22 step 25600 loss=3.8393 lr=2.90e-04 tok=3.355B dt=2123 +14:02:05 step 25620 loss=3.8254 lr=2.90e-04 tok=3.358B dt=2225 +14:03:47 step 25640 loss=3.7458 lr=2.90e-04 tok=3.361B dt=2328 +14:05:30 step 25660 loss=3.7925 lr=2.90e-04 tok=3.363B dt=2431 +14:07:13 step 25680 loss=3.7266 lr=2.90e-04 tok=3.366B dt=2534 +14:08:56 step 25700 loss=3.8329 lr=2.90e-04 tok=3.369B dt=2637 +14:10:38 step 25720 loss=3.7875 lr=2.90e-04 tok=3.371B dt=2739 +14:12:21 step 25740 loss=3.7848 lr=2.90e-04 tok=3.374B dt=2842 +14:14:03 step 25760 loss=3.8740 lr=2.90e-04 tok=3.376B dt=2944 +14:15:46 step 25780 loss=3.7379 lr=2.89e-04 tok=3.379B dt=3047 +14:17:29 step 25800 loss=3.8626 lr=2.89e-04 tok=3.382B dt=3150 +14:17:46 pushed ckpt_w2.pt diff --git a/log_w200.txt b/log_w200.txt new file mode 100644 index 0000000000000000000000000000000000000000..009156b086771451152b5c895ac3b6079dd5e907 --- /dev/null +++ b/log_w200.txt @@ -0,0 +1,95 @@ +launch +16:01:47 name=w200 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 147/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:01:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:01:54 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:02:03 params 1.08B +16:02:05 resumed step=198750 seen=26.23B round=872 +16:02:05 stream salt=198750 +16:03:57 blocks compiled +16:03:57 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:08:24 step 198751 loss=2.8239 lr=1.15e-06 tok=26.230B dt=267 wait=237 +16:08:41 step 198752 loss=2.8763 lr=1.15e-06 tok=26.230B dt=284 wait=237 +16:09:07 step 198760 loss=3.2960 lr=1.15e-06 tok=26.231B dt=309 wait=237 +16:10:12 step 198780 loss=3.1882 lr=1.13e-06 tok=26.234B dt=374 wait=237 +16:11:17 step 198800 loss=3.1703 lr=1.11e-06 tok=26.237B dt=440 wait=237 +16:11:38 global r=873 load 4.281 -> 4.281 +16:12:42 step 198920 loss=3.2801 lr=9.98e-07 tok=26.239B dt=525 wait=237 +16:13:46 step 198940 loss=3.0928 lr=9.80e-07 tok=26.242B dt=589 wait=238 +16:15:01 step 198960 loss=3.2554 lr=9.61e-07 tok=26.245B dt=664 wait=238 +16:16:06 step 198980 loss=3.1693 lr=9.43e-07 tok=26.247B dt=729 wait=238 +16:17:10 step 199000 loss=3.2104 lr=9.24e-07 tok=26.250B dt=793 wait=238 +16:18:14 step 199020 loss=3.1310 lr=9.06e-07 tok=26.252B dt=857 wait=238 +16:19:19 step 199040 loss=2.3865 lr=8.87e-07 tok=26.255B dt=922 wait=238 +16:20:08 pushed ckpt_w200.pt,log_w200.txt +16:23:44 step 199060 loss=3.2712 lr=8.69e-07 tok=26.258B dt=1187 wait=239 +16:23:54 global r=874 load 4.281 -> 4.280 +16:24:27 step 199060 loss=3.2945 lr=8.69e-07 tok=26.259B dt=1229 wait=239 +16:25:31 step 199080 loss=3.0985 lr=8.51e-07 tok=26.262B dt=1294 wait=239 +16:26:35 step 199100 loss=3.0648 lr=8.32e-07 tok=26.264B dt=1358 wait=239 +16:27:40 step 199120 loss=3.2053 lr=8.14e-07 tok=26.267B dt=1423 wait=239 +16:28:44 step 199140 loss=3.0547 lr=7.95e-07 tok=26.269B dt=1487 wait=239 +16:29:48 step 199160 loss=3.1275 lr=7.77e-07 tok=26.272B dt=1551 wait=239 +16:30:52 step 199180 loss=3.0681 lr=7.58e-07 tok=26.275B dt=1615 wait=240 +16:31:56 step 199200 loss=3.1491 lr=7.40e-07 tok=26.277B dt=1679 wait=240 +16:32:08 pushed ckpt_w200.pt,log_w200.txt +16:33:39 waited 91s for global r=875 +16:33:48 global r=875 load 4.280 -> 4.281 +16:34:53 step 199220 loss=2.9999 lr=7.21e-07 tok=26.280B dt=1856 wait=240 +16:35:57 step 199240 loss=3.1627 lr=7.03e-07 tok=26.283B dt=1919 wait=240 +16:37:01 step 199260 loss=3.0746 lr=6.84e-07 tok=26.285B dt=1983 wait=240 +16:38:05 step 199280 loss=3.0119 lr=6.66e-07 tok=26.288B dt=2047 wait=240 +16:39:08 step 199300 loss=3.0280 lr=6.47e-07 tok=26.290B dt=2110 wait=240 +16:40:11 step 199320 loss=3.1584 lr=6.29e-07 tok=26.293B dt=2174 wait=241 +16:41:15 step 199340 loss=2.9532 lr=6.10e-07 tok=26.296B dt=2238 wait=241 +16:41:59 pushed ckpt_w200.pt,log_w200.txt +16:43:30 waited 91s for global r=876 +16:43:35 global r=876 load 4.280 -> 4.280 +16:44:09 step 199360 loss=3.0524 lr=5.92e-07 tok=26.298B dt=2411 wait=241 +16:45:12 step 199380 loss=2.9480 lr=5.73e-07 tok=26.301B dt=2475 wait=241 +16:46:16 step 199400 loss=2.9766 lr=5.55e-07 tok=26.304B dt=2539 wait=241 +16:47:20 step 199420 loss=3.0562 lr=5.37e-07 tok=26.306B dt=2602 wait=241 +16:48:23 step 199440 loss=2.9610 lr=5.18e-07 tok=26.309B dt=2666 wait=241 +16:49:27 step 199460 loss=3.0112 lr=5.00e-07 tok=26.311B dt=2730 wait=242 +16:50:30 step 199480 loss=2.9608 lr=4.81e-07 tok=26.314B dt=2793 wait=242 +16:51:33 step 199500 loss=3.0253 lr=4.63e-07 tok=26.317B dt=2856 wait=242 +16:51:47 pushed ckpt_w200.pt,log_w200.txt +16:55:29 global r=877 load 4.280 -> 4.281 +16:56:32 step 199520 loss=2.9522 lr=4.44e-07 tok=26.321B dt=3155 wait=242 +16:57:35 step 199540 loss=2.9868 lr=4.26e-07 tok=26.323B dt=3218 wait=242 +16:58:40 step 199560 loss=2.7526 lr=4.07e-07 tok=26.326B dt=3282 wait=242 +16:59:43 step 199580 loss=2.9472 lr=3.89e-07 tok=26.328B dt=3345 wait=242 +17:00:46 step 199600 loss=2.7660 lr=3.70e-07 tok=26.331B dt=3409 wait=243 +17:01:49 step 199620 loss=3.0071 lr=3.52e-07 tok=26.334B dt=3472 wait=243 +17:02:54 step 199640 loss=2.8614 lr=3.33e-07 tok=26.336B dt=3537 wait=243 +17:03:39 pushed ckpt_w200.pt,log_w200.txt +17:07:14 step 199660 loss=3.0310 lr=3.15e-07 tok=26.339B dt=3797 wait=243 +17:07:23 global r=878 load 4.280 -> 4.280 +17:07:55 step 199660 loss=3.0116 lr=3.15e-07 tok=26.340B dt=3837 wait=243 +17:08:58 step 199680 loss=2.9380 lr=2.96e-07 tok=26.343B dt=3901 wait=243 +17:10:02 step 199700 loss=2.9623 lr=2.78e-07 tok=26.345B dt=3965 wait=244 +17:11:06 step 199720 loss=2.9941 lr=2.59e-07 tok=26.348B dt=4028 wait=244 +17:12:09 step 199740 loss=2.9065 lr=2.41e-07 tok=26.351B dt=4092 wait=244 +17:13:13 step 199760 loss=3.0403 lr=2.23e-07 tok=26.353B dt=4155 wait=244 +17:14:16 step 199780 loss=2.9026 lr=2.04e-07 tok=26.356B dt=4219 wait=244 +17:15:20 step 199800 loss=2.9218 lr=1.86e-07 tok=26.359B dt=4282 wait=244 +17:15:32 pushed ckpt_w200.pt,log_w200.txt +17:18:33 waited 181s for global r=879 +17:18:43 global r=879 load 4.281 -> 4.280 +17:19:50 step 199820 loss=2.8788 lr=1.67e-07 tok=26.361B dt=4552 wait=244 +17:20:53 step 199840 loss=2.9029 lr=1.49e-07 tok=26.364B dt=4615 wait=245 +17:21:57 step 199860 loss=2.9352 lr=1.30e-07 tok=26.366B dt=4679 wait=245 +17:23:01 step 199880 loss=2.9600 lr=1.12e-07 tok=26.369B dt=4743 wait=245 +17:24:04 step 199900 loss=2.9937 lr=9.33e-08 tok=26.372B dt=4807 wait=245 +17:25:08 step 199920 loss=3.0051 lr=7.48e-08 tok=26.374B dt=4870 wait=245 +17:26:12 step 199940 loss=2.8054 lr=5.63e-08 tok=26.377B dt=4934 wait=245 +17:26:56 pushed ckpt_w200.pt,log_w200.txt +17:30:30 step 199960 loss=3.0906 lr=3.79e-08 tok=26.380B dt=5193 wait=245 +17:30:42 global r=880 load 4.280 -> 4.280 +17:31:14 step 199960 loss=3.0478 lr=3.79e-08 tok=26.381B dt=5237 wait=245 +17:32:18 step 199980 loss=2.8991 lr=1.94e-08 tok=26.383B dt=5301 wait=246 +17:33:22 step 200000 loss=2.8521 lr=9.23e-10 tok=26.386B dt=5364 wait=246 +17:34:26 step 200020 loss=2.9367 lr=0.00e+00 tok=26.389B dt=5428 wait=246 +17:35:29 step 200040 loss=2.8824 lr=0.00e+00 tok=26.391B dt=5491 wait=246 +17:36:32 step 200060 loss=2.9447 lr=0.00e+00 tok=26.394B dt=5555 wait=246 +17:37:36 step 200080 loss=2.8304 lr=0.00e+00 tok=26.397B dt=5619 wait=246 +17:38:40 step 200100 loss=2.9273 lr=0.00e+00 tok=26.399B dt=5683 wait=246 diff --git a/log_w201.txt b/log_w201.txt new file mode 100644 index 0000000000000000000000000000000000000000..e0fee0b00b5b9145058a7c78b4cb6949e9b917e3 --- /dev/null +++ b/log_w201.txt @@ -0,0 +1,94 @@ +launch +16:03:13 name=w201 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 148/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:03:19 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:03:19 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:03:29 params 1.08B +16:03:30 resumed step=198750 seen=26.23B round=872 +16:03:30 stream salt=198750 +16:05:18 blocks compiled +16:05:18 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:09:39 step 198751 loss=2.5988 lr=1.15e-06 tok=26.230B dt=261 wait=232 +16:09:54 step 198752 loss=2.8246 lr=1.15e-06 tok=26.230B dt=276 wait=232 +16:10:19 step 198760 loss=3.2609 lr=1.15e-06 tok=26.231B dt=302 wait=232 +16:11:24 step 198780 loss=3.0701 lr=1.13e-06 tok=26.234B dt=366 wait=232 +16:11:41 global r=873 load 4.281 -> 4.281 +16:12:45 step 198920 loss=3.1020 lr=9.98e-07 tok=26.237B dt=447 wait=232 +16:13:49 step 198940 loss=3.1368 lr=9.80e-07 tok=26.239B dt=511 wait=232 +16:15:03 step 198960 loss=3.1609 lr=9.61e-07 tok=26.242B dt=585 wait=232 +16:16:08 step 198980 loss=3.0458 lr=9.43e-07 tok=26.245B dt=650 wait=233 +16:17:13 step 199000 loss=3.0381 lr=9.24e-07 tok=26.247B dt=715 wait=233 +16:18:18 step 199020 loss=3.1943 lr=9.06e-07 tok=26.250B dt=780 wait=233 +16:19:22 step 199040 loss=3.0721 lr=8.87e-07 tok=26.252B dt=844 wait=233 +16:20:10 pushed ckpt_w201.pt,log_w201.txt +16:23:46 step 199060 loss=3.2486 lr=8.69e-07 tok=26.255B dt=1108 wait=233 +16:23:54 global r=874 load 4.280 -> 4.280 +16:24:26 step 199060 loss=3.1983 lr=8.69e-07 tok=26.256B dt=1148 wait=233 +16:25:31 step 199080 loss=3.0705 lr=8.51e-07 tok=26.259B dt=1213 wait=233 +16:26:36 step 199100 loss=3.0180 lr=8.32e-07 tok=26.262B dt=1278 wait=234 +16:27:41 step 199120 loss=3.1214 lr=8.14e-07 tok=26.264B dt=1343 wait=234 +16:28:46 step 199140 loss=2.9280 lr=7.95e-07 tok=26.267B dt=1408 wait=234 +16:29:50 step 199160 loss=3.0370 lr=7.77e-07 tok=26.269B dt=1472 wait=234 +16:30:53 step 199180 loss=2.9489 lr=7.58e-07 tok=26.272B dt=1535 wait=234 +16:31:57 step 199200 loss=3.0864 lr=7.40e-07 tok=26.275B dt=1599 wait=234 +16:32:08 pushed ckpt_w201.pt,log_w201.txt +16:33:38 waited 91s for global r=875 +16:33:44 global r=875 load 4.280 -> 4.281 +16:34:50 step 199220 loss=2.9673 lr=7.21e-07 tok=26.277B dt=1772 wait=234 +16:35:53 step 199240 loss=3.0378 lr=7.03e-07 tok=26.280B dt=1836 wait=235 +16:36:57 step 199260 loss=3.0617 lr=6.84e-07 tok=26.283B dt=1899 wait=235 +16:38:01 step 199280 loss=2.9870 lr=6.66e-07 tok=26.285B dt=1963 wait=235 +16:39:05 step 199300 loss=2.9645 lr=6.47e-07 tok=26.288B dt=2027 wait=235 +16:40:09 step 199320 loss=3.2209 lr=6.29e-07 tok=26.290B dt=2091 wait=235 +16:41:14 step 199340 loss=2.9253 lr=6.10e-07 tok=26.293B dt=2156 wait=235 +16:41:57 pushed ckpt_w201.pt,log_w201.txt +16:43:28 waited 91s for global r=876 +16:43:36 global r=876 load 4.280 -> 4.280 +16:44:09 step 199360 loss=3.1164 lr=5.92e-07 tok=26.296B dt=2331 wait=235 +16:45:13 step 199380 loss=3.0270 lr=5.73e-07 tok=26.298B dt=2395 wait=236 +16:46:18 step 199400 loss=2.9293 lr=5.55e-07 tok=26.301B dt=2460 wait=236 +16:47:23 step 199420 loss=3.0783 lr=5.37e-07 tok=26.304B dt=2525 wait=236 +16:48:27 step 199440 loss=2.9019 lr=5.18e-07 tok=26.306B dt=2589 wait=236 +16:49:31 step 199460 loss=3.1049 lr=5.00e-07 tok=26.309B dt=2653 wait=236 +16:50:35 step 199480 loss=3.0212 lr=4.81e-07 tok=26.311B dt=2717 wait=236 +16:51:40 step 199500 loss=3.0286 lr=4.63e-07 tok=26.314B dt=2782 wait=236 +16:51:50 pushed ckpt_w201.pt,log_w201.txt +16:55:30 global r=877 load 4.280 -> 4.281 +16:56:33 step 199520 loss=3.0011 lr=4.44e-07 tok=26.318B dt=3076 wait=237 +16:57:37 step 199540 loss=3.0659 lr=4.26e-07 tok=26.321B dt=3139 wait=237 +16:58:41 step 199560 loss=3.0181 lr=4.07e-07 tok=26.323B dt=3203 wait=237 +16:59:45 step 199580 loss=2.9130 lr=3.89e-07 tok=26.326B dt=3267 wait=237 +17:00:48 step 199600 loss=3.0446 lr=3.70e-07 tok=26.328B dt=3330 wait=237 +17:01:51 step 199620 loss=3.0532 lr=3.52e-07 tok=26.331B dt=3394 wait=237 +17:02:55 step 199640 loss=3.0157 lr=3.33e-07 tok=26.334B dt=3457 wait=237 +17:03:38 pushed ckpt_w201.pt,log_w201.txt +17:07:13 step 199660 loss=3.0332 lr=3.15e-07 tok=26.336B dt=3715 wait=238 +17:07:20 global r=878 load 4.280 -> 4.280 +17:07:52 step 199660 loss=3.0216 lr=3.15e-07 tok=26.338B dt=3754 wait=238 +17:08:55 step 199680 loss=2.9768 lr=2.96e-07 tok=26.340B dt=3817 wait=238 +17:09:59 step 199700 loss=3.0299 lr=2.78e-07 tok=26.343B dt=3881 wait=238 +17:11:03 step 199720 loss=3.0993 lr=2.59e-07 tok=26.345B dt=3945 wait=238 +17:12:07 step 199740 loss=2.9485 lr=2.41e-07 tok=26.348B dt=4009 wait=238 +17:13:11 step 199760 loss=3.0866 lr=2.23e-07 tok=26.351B dt=4073 wait=238 +17:14:15 step 199780 loss=3.0794 lr=2.04e-07 tok=26.353B dt=4137 wait=238 +17:15:20 step 199800 loss=2.9603 lr=1.86e-07 tok=26.356B dt=4202 wait=239 +17:15:31 pushed ckpt_w201.pt,log_w201.txt +17:18:33 waited 181s for global r=879 +17:18:44 global r=879 load 4.280 -> 4.280 +17:19:49 step 199820 loss=3.0137 lr=1.67e-07 tok=26.359B dt=4471 wait=239 +17:20:53 step 199840 loss=2.9551 lr=1.49e-07 tok=26.361B dt=4535 wait=239 +17:21:57 step 199860 loss=3.0688 lr=1.30e-07 tok=26.364B dt=4600 wait=239 +17:23:02 step 199880 loss=2.9095 lr=1.12e-07 tok=26.366B dt=4664 wait=239 +17:24:07 step 199900 loss=2.8953 lr=9.33e-08 tok=26.369B dt=4729 wait=239 +17:25:11 step 199920 loss=3.0365 lr=7.48e-08 tok=26.372B dt=4793 wait=240 +17:26:14 step 199940 loss=2.9396 lr=5.63e-08 tok=26.374B dt=4856 wait=240 +17:26:58 pushed ckpt_w201.pt,log_w201.txt +17:30:33 step 199960 loss=3.0686 lr=3.79e-08 tok=26.377B dt=5115 wait=240 +17:30:39 global r=880 load 4.281 -> 4.280 +17:31:10 step 199960 loss=3.0771 lr=3.79e-08 tok=26.378B dt=5152 wait=240 +17:32:14 step 199980 loss=2.9081 lr=1.94e-08 tok=26.381B dt=5217 wait=240 +17:33:18 step 200000 loss=2.9379 lr=9.23e-10 tok=26.383B dt=5280 wait=240 +17:34:23 step 200020 loss=3.0636 lr=0.00e+00 tok=26.386B dt=5345 wait=240 +17:35:27 step 200040 loss=2.9404 lr=0.00e+00 tok=26.389B dt=5409 wait=241 +17:36:32 step 200060 loss=3.0374 lr=0.00e+00 tok=26.391B dt=5474 wait=241 +17:37:36 step 200080 loss=3.0477 lr=0.00e+00 tok=26.394B dt=5538 wait=241 +17:38:40 step 200100 loss=2.9286 lr=0.00e+00 tok=26.397B dt=5602 wait=241 diff --git a/log_w202.txt b/log_w202.txt new file mode 100644 index 0000000000000000000000000000000000000000..36295b55e70123d230abe44277433eb79ff5816b --- /dev/null +++ b/log_w202.txt @@ -0,0 +1,93 @@ +launch +16:04:37 name=w202 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 149/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:04:43 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:04:44 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:04:53 params 1.08B +16:04:55 resumed step=198750 seen=26.23B round=872 +16:04:55 stream salt=198750 +16:06:44 blocks compiled +16:06:44 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:11:11 step 198751 loss=2.8694 lr=1.15e-06 tok=26.230B dt=267 wait=237 +16:11:27 step 198752 loss=2.9459 lr=1.15e-06 tok=26.230B dt=283 wait=237 +16:11:52 step 198760 loss=3.1997 lr=1.15e-06 tok=26.231B dt=309 wait=237 +16:12:08 global r=873 load 4.280 -> 4.281 +16:13:13 step 198920 loss=3.2676 lr=9.98e-07 tok=26.234B dt=389 wait=237 +16:14:18 step 198940 loss=3.2688 lr=9.80e-07 tok=26.237B dt=454 wait=238 +16:15:31 step 198960 loss=3.3277 lr=9.61e-07 tok=26.239B dt=527 wait=241 +16:16:35 step 198980 loss=3.2047 lr=9.43e-07 tok=26.242B dt=591 wait=241 +16:17:41 step 199000 loss=3.2445 lr=9.24e-07 tok=26.245B dt=657 wait=241 +16:18:45 step 199020 loss=3.2611 lr=9.06e-07 tok=26.247B dt=722 wait=241 +16:19:50 step 199040 loss=3.1270 lr=8.87e-07 tok=26.250B dt=786 wait=241 +16:20:38 pushed ckpt_w202.pt,log_w202.txt +16:23:40 waited 181s for global r=874 +16:23:48 global r=874 load 4.281 -> 4.280 +16:24:23 step 199060 loss=3.2289 lr=8.69e-07 tok=26.252B dt=1059 wait=241 +16:25:27 step 199080 loss=3.2175 lr=8.51e-07 tok=26.255B dt=1124 wait=242 +16:26:33 step 199100 loss=2.6083 lr=8.32e-07 tok=26.258B dt=1189 wait=242 +16:27:38 step 199120 loss=3.2654 lr=8.14e-07 tok=26.260B dt=1254 wait=242 +16:28:43 step 199140 loss=3.0927 lr=7.95e-07 tok=26.263B dt=1319 wait=242 +16:29:48 step 199160 loss=3.1091 lr=7.77e-07 tok=26.266B dt=1384 wait=242 +16:30:53 step 199180 loss=3.1271 lr=7.58e-07 tok=26.268B dt=1449 wait=242 +16:31:58 step 199200 loss=3.1643 lr=7.40e-07 tok=26.271B dt=1514 wait=242 +16:32:14 pushed ckpt_w202.pt,log_w202.txt +16:33:29 waited 76s for global r=875 +16:33:39 global r=875 load 4.281 -> 4.281 +16:34:46 step 199220 loss=3.1213 lr=7.21e-07 tok=26.273B dt=1682 wait=243 +16:35:50 step 199240 loss=3.0720 lr=7.03e-07 tok=26.276B dt=1746 wait=243 +16:36:54 step 199260 loss=3.1166 lr=6.84e-07 tok=26.279B dt=1810 wait=243 +16:37:58 step 199280 loss=3.0729 lr=6.66e-07 tok=26.281B dt=1874 wait=243 +16:39:02 step 199300 loss=3.0381 lr=6.47e-07 tok=26.284B dt=1938 wait=243 +16:40:06 step 199320 loss=3.1234 lr=6.29e-07 tok=26.286B dt=2002 wait=243 +16:41:10 step 199340 loss=2.9890 lr=6.10e-07 tok=26.289B dt=2066 wait=244 +16:41:54 pushed ckpt_w202.pt,log_w202.txt +16:43:25 waited 91s for global r=876 +16:43:34 global r=876 load 4.280 -> 4.280 +16:44:07 step 199360 loss=3.1169 lr=5.92e-07 tok=26.292B dt=2243 wait=244 +16:45:11 step 199380 loss=2.9472 lr=5.73e-07 tok=26.294B dt=2307 wait=244 +16:46:15 step 199400 loss=2.9355 lr=5.55e-07 tok=26.297B dt=2371 wait=244 +16:47:19 step 199420 loss=2.9730 lr=5.37e-07 tok=26.300B dt=2435 wait=244 +16:48:23 step 199440 loss=2.8505 lr=5.18e-07 tok=26.302B dt=2499 wait=244 +16:49:27 step 199460 loss=2.9882 lr=5.00e-07 tok=26.305B dt=2563 wait=244 +16:50:30 step 199480 loss=2.9488 lr=4.81e-07 tok=26.307B dt=2626 wait=245 +16:51:33 step 199500 loss=2.9299 lr=4.63e-07 tok=26.310B dt=2690 wait=245 +16:51:45 pushed ckpt_w202.pt,log_w202.txt +16:55:27 global r=877 load 4.281 -> 4.281 +16:56:30 step 199520 loss=2.9264 lr=4.44e-07 tok=26.314B dt=2986 wait=245 +16:57:33 step 199540 loss=3.0201 lr=4.26e-07 tok=26.317B dt=3049 wait=245 +16:58:37 step 199560 loss=2.9007 lr=4.07e-07 tok=26.319B dt=3113 wait=245 +16:59:40 step 199580 loss=2.8946 lr=3.89e-07 tok=26.322B dt=3176 wait=245 +17:00:44 step 199600 loss=2.8818 lr=3.70e-07 tok=26.325B dt=3240 wait=245 +17:01:47 step 199620 loss=2.9816 lr=3.52e-07 tok=26.327B dt=3303 wait=246 +17:02:50 step 199640 loss=2.7741 lr=3.33e-07 tok=26.330B dt=3367 wait=246 +17:03:34 pushed ckpt_w202.pt,log_w202.txt +17:07:09 step 199660 loss=2.8976 lr=3.15e-07 tok=26.332B dt=3625 wait=246 +17:07:17 global r=878 load 4.280 -> 4.280 +17:07:48 step 199660 loss=2.8947 lr=3.15e-07 tok=26.334B dt=3665 wait=246 +17:08:52 step 199680 loss=2.7471 lr=2.96e-07 tok=26.336B dt=3728 wait=246 +17:09:55 step 199700 loss=2.8372 lr=2.78e-07 tok=26.339B dt=3791 wait=246 +17:10:58 step 199720 loss=2.8996 lr=2.59e-07 tok=26.342B dt=3854 wait=246 +17:12:01 step 199740 loss=2.8874 lr=2.41e-07 tok=26.344B dt=3917 wait=247 +17:13:06 step 199760 loss=3.0159 lr=2.23e-07 tok=26.347B dt=3982 wait=247 +17:14:10 step 199780 loss=3.0417 lr=2.04e-07 tok=26.349B dt=4046 wait=247 +17:15:15 step 199800 loss=2.9817 lr=1.86e-07 tok=26.352B dt=4111 wait=247 +17:15:29 pushed ckpt_w202.pt,log_w202.txt +17:18:30 waited 181s for global r=879 +17:18:37 global r=879 load 4.280 -> 4.280 +17:19:43 step 199820 loss=2.9983 lr=1.67e-07 tok=26.355B dt=4380 wait=247 +17:20:49 step 199840 loss=3.0389 lr=1.49e-07 tok=26.357B dt=4445 wait=247 +17:21:53 step 199860 loss=3.0575 lr=1.30e-07 tok=26.360B dt=4509 wait=247 +17:22:57 step 199880 loss=2.9672 lr=1.12e-07 tok=26.363B dt=4573 wait=248 +17:24:01 step 199900 loss=2.9825 lr=9.33e-08 tok=26.365B dt=4637 wait=248 +17:25:04 step 199920 loss=2.9699 lr=7.48e-08 tok=26.368B dt=4701 wait=248 +17:26:08 step 199940 loss=2.8984 lr=5.63e-08 tok=26.370B dt=4764 wait=248 +17:26:52 pushed ckpt_w202.pt,log_w202.txt +17:30:26 step 199960 loss=2.9642 lr=3.79e-08 tok=26.373B dt=5023 wait=248 +17:30:33 global r=880 load 4.280 -> 4.280 +17:31:04 step 199960 loss=3.0072 lr=3.79e-08 tok=26.374B dt=5060 wait=248 +17:32:08 step 199980 loss=2.8897 lr=1.94e-08 tok=26.377B dt=5124 wait=248 +17:33:12 step 200000 loss=2.8610 lr=9.23e-10 tok=26.380B dt=5188 wait=249 +17:34:15 step 200020 loss=3.0328 lr=0.00e+00 tok=26.382B dt=5252 wait=249 +17:35:19 step 200040 loss=2.9454 lr=0.00e+00 tok=26.385B dt=5315 wait=249 +17:36:22 step 200060 loss=3.0309 lr=0.00e+00 tok=26.387B dt=5378 wait=249 +17:37:25 step 200080 loss=2.8775 lr=0.00e+00 tok=26.390B dt=5442 wait=249 +17:38:28 step 200100 loss=3.0528 lr=0.00e+00 tok=26.393B dt=5504 wait=249 diff --git a/log_w203.txt b/log_w203.txt new file mode 100644 index 0000000000000000000000000000000000000000..08161e3163981c9db6aefa23b67cee86fcdacb0c --- /dev/null +++ b/log_w203.txt @@ -0,0 +1,93 @@ +launch +16:06:04 name=w203 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 150/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:06:10 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:06:10 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:06:20 params 1.08B +16:06:22 resumed step=198750 seen=26.23B round=872 +16:06:22 stream salt=198750 +16:08:07 blocks compiled +16:08:07 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:12:59 step 198751 loss=2.8739 lr=1.15e-06 tok=26.230B dt=291 wait=261 +16:13:15 step 198752 loss=2.8434 lr=1.15e-06 tok=26.230B dt=308 wait=261 +16:13:41 step 198760 loss=3.2964 lr=1.15e-06 tok=26.231B dt=333 wait=262 +16:14:04 global r=873 load 4.280 -> 4.281 +16:15:35 step 198920 loss=2.9798 lr=9.98e-07 tok=26.234B dt=448 wait=288 +16:16:40 step 198940 loss=3.0019 lr=9.80e-07 tok=26.237B dt=513 wait=288 +16:17:43 step 198960 loss=3.0375 lr=9.61e-07 tok=26.239B dt=576 wait=289 +16:18:47 step 198980 loss=2.9847 lr=9.43e-07 tok=26.242B dt=640 wait=289 +16:19:52 step 199000 loss=2.9417 lr=9.24e-07 tok=26.245B dt=704 wait=289 +16:20:56 step 199020 loss=3.0373 lr=9.06e-07 tok=26.247B dt=769 wait=289 +16:22:01 step 199040 loss=2.8817 lr=8.87e-07 tok=26.250B dt=834 wait=289 +16:22:47 pushed ckpt_w203.pt,log_w203.txt +16:23:33 waited 46s for global r=874 +16:23:43 global r=874 load 4.281 -> 4.280 +16:24:18 step 199060 loss=3.0637 lr=8.69e-07 tok=26.252B dt=971 wait=289 +16:25:23 step 199080 loss=2.9134 lr=8.51e-07 tok=26.255B dt=1036 wait=289 +16:26:28 step 199100 loss=2.9977 lr=8.32e-07 tok=26.258B dt=1100 wait=290 +16:27:33 step 199120 loss=3.0757 lr=8.14e-07 tok=26.260B dt=1166 wait=290 +16:28:39 step 199140 loss=2.9505 lr=7.95e-07 tok=26.263B dt=1231 wait=290 +16:29:44 step 199160 loss=3.1485 lr=7.77e-07 tok=26.266B dt=1297 wait=290 +16:30:49 step 199180 loss=3.1900 lr=7.58e-07 tok=26.268B dt=1362 wait=290 +16:31:54 step 199200 loss=3.0769 lr=7.40e-07 tok=26.271B dt=1427 wait=290 +16:32:06 pushed ckpt_w203.pt,log_w203.txt +16:33:37 waited 91s for global r=875 +16:33:48 global r=875 load 4.280 -> 4.281 +16:34:54 step 199220 loss=3.0930 lr=7.21e-07 tok=26.273B dt=1607 wait=291 +16:35:59 step 199240 loss=3.1431 lr=7.03e-07 tok=26.276B dt=1672 wait=291 +16:37:03 step 199260 loss=3.1379 lr=6.84e-07 tok=26.279B dt=1736 wait=291 +16:38:08 step 199280 loss=3.0997 lr=6.66e-07 tok=26.281B dt=1801 wait=291 +16:39:12 step 199300 loss=3.0092 lr=6.47e-07 tok=26.284B dt=1864 wait=291 +16:40:16 step 199320 loss=3.1110 lr=6.29e-07 tok=26.286B dt=1929 wait=291 +16:41:20 step 199340 loss=2.9931 lr=6.10e-07 tok=26.289B dt=1993 wait=292 +16:42:06 pushed ckpt_w203.pt,log_w203.txt +16:43:22 waited 76s for global r=876 +16:43:30 global r=876 load 4.280 -> 4.280 +16:44:04 step 199360 loss=3.1111 lr=5.92e-07 tok=26.292B dt=2157 wait=292 +16:45:08 step 199380 loss=2.9843 lr=5.73e-07 tok=26.294B dt=2221 wait=292 +16:46:13 step 199400 loss=2.9923 lr=5.55e-07 tok=26.297B dt=2286 wait=292 +16:47:18 step 199420 loss=3.0378 lr=5.37e-07 tok=26.300B dt=2350 wait=292 +16:48:22 step 199440 loss=2.9244 lr=5.18e-07 tok=26.302B dt=2414 wait=292 +16:49:25 step 199460 loss=3.0550 lr=5.00e-07 tok=26.305B dt=2478 wait=292 +16:50:29 step 199480 loss=2.9961 lr=4.81e-07 tok=26.307B dt=2542 wait=292 +16:51:33 step 199500 loss=3.0689 lr=4.63e-07 tok=26.310B dt=2605 wait=293 +16:51:45 pushed ckpt_w203.pt,log_w203.txt +16:55:28 global r=877 load 4.280 -> 4.281 +16:56:32 step 199520 loss=2.9639 lr=4.44e-07 tok=26.314B dt=2905 wait=293 +16:57:36 step 199540 loss=3.1321 lr=4.26e-07 tok=26.317B dt=2969 wait=293 +16:58:40 step 199560 loss=3.0516 lr=4.07e-07 tok=26.319B dt=3032 wait=293 +16:59:43 step 199580 loss=3.0543 lr=3.89e-07 tok=26.322B dt=3096 wait=293 +17:00:48 step 199600 loss=3.1151 lr=3.70e-07 tok=26.325B dt=3161 wait=294 +17:01:52 step 199620 loss=3.1753 lr=3.52e-07 tok=26.327B dt=3225 wait=294 +17:02:56 step 199640 loss=2.9131 lr=3.33e-07 tok=26.330B dt=3289 wait=294 +17:03:40 pushed ckpt_w203.pt,log_w203.txt +17:07:16 step 199660 loss=3.1317 lr=3.15e-07 tok=26.332B dt=3549 wait=294 +17:07:25 global r=878 load 4.281 -> 4.280 +17:07:57 step 199660 loss=3.1086 lr=3.15e-07 tok=26.334B dt=3590 wait=294 +17:09:01 step 199680 loss=2.8736 lr=2.96e-07 tok=26.336B dt=3654 wait=294 +17:10:05 step 199700 loss=3.0364 lr=2.78e-07 tok=26.339B dt=3718 wait=294 +17:11:10 step 199720 loss=3.0156 lr=2.59e-07 tok=26.342B dt=3782 wait=295 +17:12:14 step 199740 loss=2.8134 lr=2.41e-07 tok=26.344B dt=3847 wait=295 +17:13:19 step 199760 loss=2.9722 lr=2.23e-07 tok=26.347B dt=3912 wait=295 +17:14:25 step 199780 loss=2.7707 lr=2.04e-07 tok=26.349B dt=3977 wait=295 +17:15:30 step 199800 loss=2.8598 lr=1.86e-07 tok=26.352B dt=4042 wait=295 +17:15:42 pushed ckpt_w203.pt,log_w203.txt +17:18:28 waited 166s for global r=879 +17:18:41 global r=879 load 4.280 -> 4.280 +17:19:48 step 199820 loss=2.9353 lr=1.67e-07 tok=26.355B dt=4301 wait=295 +17:20:54 step 199840 loss=2.8389 lr=1.49e-07 tok=26.357B dt=4367 wait=296 +17:22:00 step 199860 loss=2.8818 lr=1.30e-07 tok=26.360B dt=4433 wait=296 +17:23:05 step 199880 loss=2.8142 lr=1.12e-07 tok=26.363B dt=4498 wait=296 +17:24:11 step 199900 loss=2.8014 lr=9.33e-08 tok=26.365B dt=4564 wait=296 +17:25:17 step 199920 loss=3.0464 lr=7.48e-08 tok=26.368B dt=4630 wait=296 +17:26:24 step 199940 loss=2.8113 lr=5.63e-08 tok=26.370B dt=4696 wait=296 +17:27:09 pushed ckpt_w203.pt,log_w203.txt +17:30:10 waited 181s for global r=880 +17:30:20 global r=880 load 4.280 -> 4.280 +17:30:54 step 199960 loss=3.0004 lr=3.79e-08 tok=26.373B dt=4967 wait=297 +17:32:00 step 199980 loss=2.9220 lr=1.94e-08 tok=26.376B dt=5032 wait=297 +17:33:05 step 200000 loss=2.9030 lr=9.23e-10 tok=26.378B dt=5098 wait=297 +17:34:12 step 200020 loss=2.9502 lr=0.00e+00 tok=26.381B dt=5165 wait=297 +17:35:19 step 200040 loss=2.8449 lr=0.00e+00 tok=26.383B dt=5232 wait=297 +17:36:25 step 200060 loss=2.9105 lr=0.00e+00 tok=26.386B dt=5297 wait=297 +17:37:31 step 200080 loss=2.8451 lr=0.00e+00 tok=26.389B dt=5363 wait=297 +17:38:36 step 200100 loss=2.9550 lr=0.00e+00 tok=26.391B dt=5429 wait=298 diff --git a/log_w204.txt b/log_w204.txt new file mode 100644 index 0000000000000000000000000000000000000000..446115adf7404163d96dbca24fc4d5a870015d49 --- /dev/null +++ b/log_w204.txt @@ -0,0 +1,92 @@ +launch +16:07:42 name=w204 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 151/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:07:48 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:07:48 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:07:58 params 1.08B +16:08:00 resumed step=198750 seen=26.23B round=872 +16:08:00 stream salt=198750 +16:09:48 blocks compiled +16:09:48 start: accum=4, 131072 tokens per optimiser step, short rows 10% x 128 +16:13:17 step 198751 loss=2.9485 lr=1.15e-06 tok=26.230B dt=210 wait=168 +16:13:41 step 198752 loss=2.8536 lr=1.15e-06 tok=26.230B dt=233 wait=168 +16:14:08 step 198760 loss=3.4322 lr=1.15e-06 tok=26.231B dt=261 wait=169 +16:15:16 global r=873 load 4.281 -> 4.281 +16:16:22 step 198920 loss=3.0548 lr=9.98e-07 tok=26.235B dt=394 wait=174 +16:17:27 step 198940 loss=3.1122 lr=9.80e-07 tok=26.238B dt=460 wait=174 +16:18:33 step 198960 loss=3.1314 lr=9.61e-07 tok=26.241B dt=525 wait=174 +16:19:38 step 198980 loss=3.0396 lr=9.43e-07 tok=26.243B dt=590 wait=174 +16:20:44 step 199000 loss=2.9792 lr=9.24e-07 tok=26.246B dt=656 wait=174 +16:21:50 step 199020 loss=3.0684 lr=9.06e-07 tok=26.248B dt=722 wait=174 +16:22:54 step 199040 loss=3.1426 lr=8.87e-07 tok=26.251B dt=787 wait=174 +16:23:43 pushed ckpt_w204.pt,log_w204.txt +16:23:43 waited 0s for global r=874 +16:23:54 global r=874 load 4.281 -> 4.280 +16:24:29 step 199060 loss=3.0243 lr=8.69e-07 tok=26.254B dt=881 wait=175 +16:25:34 step 199080 loss=2.9336 lr=8.51e-07 tok=26.256B dt=946 wait=175 +16:26:39 step 199100 loss=3.0431 lr=8.32e-07 tok=26.259B dt=1012 wait=175 +16:27:45 step 199120 loss=3.1337 lr=8.14e-07 tok=26.262B dt=1077 wait=175 +16:28:50 step 199140 loss=2.9502 lr=7.95e-07 tok=26.264B dt=1143 wait=175 +16:29:56 step 199160 loss=3.0690 lr=7.77e-07 tok=26.267B dt=1209 wait=175 +16:31:01 step 199180 loss=3.1093 lr=7.58e-07 tok=26.269B dt=1273 wait=176 +16:32:06 step 199200 loss=3.0963 lr=7.40e-07 tok=26.272B dt=1338 wait=176 +16:32:19 pushed ckpt_w204.pt,log_w204.txt +16:33:34 waited 76s for global r=875 +16:33:41 global r=875 load 4.280 -> 4.281 +16:34:49 step 199220 loss=3.1002 lr=7.21e-07 tok=26.275B dt=1501 wait=176 +16:35:54 step 199240 loss=3.1122 lr=7.03e-07 tok=26.277B dt=1566 wait=176 +16:36:59 step 199260 loss=3.1263 lr=6.84e-07 tok=26.280B dt=1632 wait=176 +16:38:03 step 199280 loss=3.1062 lr=6.66e-07 tok=26.283B dt=1696 wait=176 +16:39:09 step 199300 loss=2.9925 lr=6.47e-07 tok=26.285B dt=1761 wait=176 +16:40:14 step 199320 loss=2.8531 lr=6.29e-07 tok=26.288B dt=1826 wait=177 +16:41:18 step 199340 loss=3.0956 lr=6.10e-07 tok=26.290B dt=1891 wait=177 +16:42:06 pushed ckpt_w204.pt,log_w204.txt +16:43:22 waited 76s for global r=876 +16:43:31 global r=876 load 4.280 -> 4.280 +16:44:05 step 199360 loss=3.1668 lr=5.92e-07 tok=26.293B dt=2057 wait=177 +16:45:11 step 199380 loss=2.8847 lr=5.73e-07 tok=26.296B dt=2123 wait=177 +16:46:16 step 199400 loss=3.1454 lr=5.55e-07 tok=26.298B dt=2189 wait=177 +16:47:22 step 199420 loss=3.1640 lr=5.37e-07 tok=26.301B dt=2254 wait=177 +16:48:27 step 199440 loss=3.0297 lr=5.18e-07 tok=26.304B dt=2319 wait=178 +16:49:32 step 199460 loss=3.1959 lr=5.00e-07 tok=26.306B dt=2385 wait=178 +16:50:37 step 199480 loss=3.0663 lr=4.81e-07 tok=26.309B dt=2449 wait=178 +16:51:41 step 199500 loss=3.0821 lr=4.63e-07 tok=26.311B dt=2514 wait=178 +16:51:53 pushed ckpt_w204.pt,log_w204.txt +16:55:44 global r=877 load 4.280 -> 4.281 +16:56:48 step 199520 loss=3.0146 lr=4.44e-07 tok=26.315B dt=2820 wait=178 +16:57:52 step 199540 loss=3.0456 lr=4.26e-07 tok=26.318B dt=2885 wait=178 +16:58:57 step 199560 loss=3.0613 lr=4.07e-07 tok=26.321B dt=2949 wait=179 +17:00:02 step 199580 loss=2.9867 lr=3.89e-07 tok=26.323B dt=3014 wait=179 +17:01:07 step 199600 loss=2.8623 lr=3.70e-07 tok=26.326B dt=3080 wait=179 +17:02:12 step 199620 loss=3.0815 lr=3.52e-07 tok=26.328B dt=3144 wait=179 +17:03:16 step 199640 loss=2.8883 lr=3.33e-07 tok=26.331B dt=3208 wait=179 +17:04:01 pushed ckpt_w204.pt,log_w204.txt +17:06:47 waited 166s for global r=878 +17:06:57 global r=878 load 4.280 -> 4.280 +17:07:31 step 199660 loss=2.9577 lr=3.15e-07 tok=26.334B dt=3463 wait=179 +17:08:36 step 199680 loss=2.7130 lr=2.96e-07 tok=26.336B dt=3529 wait=180 +17:09:40 step 199700 loss=2.8409 lr=2.78e-07 tok=26.339B dt=3593 wait=180 +17:10:45 step 199720 loss=2.9921 lr=2.59e-07 tok=26.342B dt=3657 wait=180 +17:11:49 step 199740 loss=2.7407 lr=2.41e-07 tok=26.344B dt=3721 wait=180 +17:12:53 step 199760 loss=2.8807 lr=2.23e-07 tok=26.347B dt=3785 wait=180 +17:13:57 step 199780 loss=2.8081 lr=2.04e-07 tok=26.349B dt=3849 wait=180 +17:15:01 step 199800 loss=2.9250 lr=1.86e-07 tok=26.352B dt=3914 wait=180 +17:15:15 pushed ckpt_w204.pt,log_w204.txt +17:19:01 global r=879 load 4.280 -> 4.280 +17:20:06 step 199820 loss=2.8245 lr=1.67e-07 tok=26.356B dt=4219 wait=181 +17:21:11 step 199840 loss=2.9345 lr=1.49e-07 tok=26.359B dt=4283 wait=181 +17:22:16 step 199860 loss=2.8683 lr=1.30e-07 tok=26.361B dt=4348 wait=181 +17:23:20 step 199880 loss=3.0160 lr=1.12e-07 tok=26.364B dt=4412 wait=181 +17:24:24 step 199900 loss=2.9568 lr=9.33e-08 tok=26.366B dt=4477 wait=181 +17:25:30 step 199920 loss=3.0323 lr=7.48e-08 tok=26.369B dt=4542 wait=182 +17:26:34 step 199940 loss=2.8870 lr=5.63e-08 tok=26.372B dt=4606 wait=182 +17:27:20 pushed ckpt_w204.pt,log_w204.txt +17:30:07 waited 166s for global r=880 +17:30:18 global r=880 load 4.281 -> 4.280 +17:30:51 step 199960 loss=2.9996 lr=3.79e-08 tok=26.374B dt=4863 wait=182 +17:31:56 step 199980 loss=2.9452 lr=1.94e-08 tok=26.377B dt=4929 wait=182 +17:33:02 step 200000 loss=3.0585 lr=9.23e-10 tok=26.380B dt=4994 wait=182 +17:34:07 step 200020 loss=2.9613 lr=0.00e+00 tok=26.382B dt=5060 wait=182 +17:35:13 step 200040 loss=2.9201 lr=0.00e+00 tok=26.385B dt=5125 wait=182 +17:36:17 step 200060 loss=3.0928 lr=0.00e+00 tok=26.387B dt=5190 wait=183 +17:37:22 step 200080 loss=2.9815 lr=0.00e+00 tok=26.390B dt=5255 wait=183 +17:38:27 step 200100 loss=3.0373 lr=0.00e+00 tok=26.393B dt=5319 wait=183 diff --git a/log_w21.txt b/log_w21.txt new file mode 100644 index 0000000000000000000000000000000000000000..02a333f20f4fe40d4e70cf45d2c9a17981ba1cfb --- /dev/null +++ b/log_w21.txt @@ -0,0 +1,155 @@ +launch +22:05:20 name=w21 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:05:26 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:05:35 params 1.08B +22:05:37 resumed step=71700 seen=9.40B round=134 +22:05:37 stream salt=71700 +22:08:05 start: accum=4, 131072 tokens per optimiser step +22:10:06 step 71701 loss=3.2098 lr=1.12e-04 tok=9.398B dt=121 wait=93 +22:10:11 step 71702 loss=3.0832 lr=1.12e-04 tok=9.398B dt=126 wait=93 +22:11:37 step 71720 loss=3.3079 lr=1.12e-04 tok=9.400B dt=212 wait=93 +22:13:12 step 71740 loss=3.3351 lr=1.12e-04 tok=9.403B dt=307 wait=94 +22:14:27 step 71760 loss=3.2998 lr=1.12e-04 tok=9.406B dt=382 wait=94 +22:15:42 step 71780 loss=3.1922 lr=1.12e-04 tok=9.408B dt=457 wait=94 +22:16:57 step 71800 loss=3.2215 lr=1.12e-04 tok=9.411B dt=532 wait=95 +22:18:12 step 71820 loss=3.1404 lr=1.12e-04 tok=9.414B dt=607 wait=95 +22:19:26 step 71840 loss=3.1460 lr=1.12e-04 tok=9.416B dt=682 wait=96 +22:20:18 pushed ckpt_w21.pt,log_w21.txt +22:20:18 waited 0s for global r=135 +22:20:33 global r=135 load 4.284 -> 4.267 +22:21:16 step 71860 loss=3.2071 lr=1.12e-04 tok=9.419B dt=791 wait=96 +22:22:31 step 71880 loss=3.1217 lr=1.12e-04 tok=9.421B dt=866 wait=96 +22:23:46 step 71900 loss=3.1908 lr=1.12e-04 tok=9.424B dt=941 wait=97 +22:25:01 step 71920 loss=3.1982 lr=1.12e-04 tok=9.427B dt=1016 wait=97 +22:26:16 step 71940 loss=3.1630 lr=1.12e-04 tok=9.429B dt=1091 wait=98 +22:27:31 step 71960 loss=3.2604 lr=1.12e-04 tok=9.432B dt=1166 wait=98 +22:28:46 step 71980 loss=3.1286 lr=1.12e-04 tok=9.435B dt=1241 wait=98 +22:30:01 step 72000 loss=3.1756 lr=1.12e-04 tok=9.437B dt=1316 wait=99 +22:30:13 pushed ckpt_w21.pt,log_w21.txt +22:31:59 waited 106s for global r=136 +22:32:11 global r=136 load 4.282 -> 4.264 +22:33:31 step 72020 loss=3.1223 lr=1.12e-04 tok=9.440B dt=1527 wait=99 +22:34:46 step 72040 loss=3.1736 lr=1.12e-04 tok=9.442B dt=1601 wait=100 +22:36:01 step 72060 loss=3.1513 lr=1.12e-04 tok=9.445B dt=1676 wait=100 +22:37:16 step 72080 loss=3.2153 lr=1.12e-04 tok=9.448B dt=1751 wait=100 +22:38:31 step 72100 loss=3.2116 lr=1.12e-04 tok=9.450B dt=1827 wait=101 +22:39:47 step 72120 loss=3.1556 lr=1.12e-04 tok=9.453B dt=1902 wait=101 +22:41:02 step 72140 loss=3.2005 lr=1.12e-04 tok=9.456B dt=1977 wait=102 +22:41:51 pushed ckpt_w21.pt,log_w21.txt +22:45:38 step 72160 loss=3.1504 lr=1.12e-04 tok=9.458B dt=2253 wait=102 +22:46:52 step 72180 loss=3.1550 lr=1.12e-04 tok=9.461B dt=2328 wait=102 +22:48:07 step 72200 loss=3.0885 lr=1.12e-04 tok=9.463B dt=2402 wait=103 +22:49:22 step 72220 loss=3.1099 lr=1.12e-04 tok=9.466B dt=2477 wait=103 +22:50:37 step 72240 loss=3.1513 lr=1.11e-04 tok=9.469B dt=2552 wait=104 +22:51:52 step 72260 loss=3.1839 lr=1.11e-04 tok=9.471B dt=2627 wait=104 +22:53:07 step 72280 loss=3.2263 lr=1.11e-04 tok=9.474B dt=2702 wait=104 +22:54:22 step 72300 loss=3.1557 lr=1.11e-04 tok=9.477B dt=2777 wait=105 +22:54:34 pushed ckpt_w21.pt,log_w21.txt +22:54:34 waited 0s for global r=137 +22:54:47 global r=137 load 4.282 -> 4.258 +22:56:05 step 72320 loss=3.1516 lr=1.11e-04 tok=9.479B dt=2881 wait=105 +22:57:20 step 72340 loss=3.1816 lr=1.11e-04 tok=9.482B dt=2956 wait=106 +22:58:35 step 72360 loss=3.1458 lr=1.11e-04 tok=9.484B dt=3031 wait=106 +22:59:50 step 72380 loss=3.1727 lr=1.11e-04 tok=9.487B dt=3105 wait=106 +23:01:05 step 72400 loss=3.1954 lr=1.11e-04 tok=9.490B dt=3181 wait=107 +23:02:20 step 72420 loss=3.1680 lr=1.11e-04 tok=9.492B dt=3255 wait=107 +23:03:35 step 72440 loss=3.2085 lr=1.11e-04 tok=9.495B dt=3331 wait=108 +23:04:28 pushed ckpt_w21.pt,log_w21.txt +23:05:44 waited 76s for global r=138 +23:05:53 global r=138 load 4.279 -> 4.256 +23:06:36 step 72460 loss=3.1696 lr=1.11e-04 tok=9.497B dt=3511 wait=108 +23:07:51 step 72480 loss=3.1239 lr=1.11e-04 tok=9.500B dt=3586 wait=108 +23:09:06 step 72500 loss=3.1983 lr=1.11e-04 tok=9.503B dt=3661 wait=109 +23:10:21 step 72520 loss=3.2193 lr=1.11e-04 tok=9.505B dt=3736 wait=109 +23:11:36 step 72540 loss=3.1328 lr=1.11e-04 tok=9.508B dt=3811 wait=109 +23:12:51 step 72560 loss=3.2417 lr=1.11e-04 tok=9.511B dt=3886 wait=110 +23:14:06 step 72580 loss=3.1250 lr=1.11e-04 tok=9.513B dt=3961 wait=110 +23:15:21 step 72600 loss=3.2024 lr=1.11e-04 tok=9.516B dt=4036 wait=111 +23:15:38 pushed ckpt_w21.pt,log_w21.txt +23:20:00 step 72620 loss=3.2291 lr=1.11e-04 tok=9.518B dt=4315 wait=111 +23:21:15 step 72640 loss=3.1270 lr=1.11e-04 tok=9.521B dt=4390 wait=111 +23:22:29 step 72660 loss=3.1409 lr=1.11e-04 tok=9.524B dt=4465 wait=112 +23:23:44 step 72680 loss=3.0996 lr=1.11e-04 tok=9.526B dt=4539 wait=112 +23:24:59 step 72700 loss=3.0354 lr=1.11e-04 tok=9.529B dt=4614 wait=113 +23:26:14 step 72720 loss=3.1844 lr=1.11e-04 tok=9.532B dt=4689 wait=113 +23:27:29 step 72740 loss=3.1647 lr=1.11e-04 tok=9.534B dt=4764 wait=113 +23:28:19 pushed ckpt_w21.pt,log_w21.txt +23:28:19 waited 0s for global r=139 +23:28:28 global r=139 load 4.289 -> 4.260 +23:29:11 step 72760 loss=3.1409 lr=1.11e-04 tok=9.537B dt=4866 wait=114 +23:30:25 step 72780 loss=3.1635 lr=1.11e-04 tok=9.539B dt=4941 wait=114 +23:31:40 step 72800 loss=3.1950 lr=1.11e-04 tok=9.542B dt=5016 wait=115 +23:32:55 step 72820 loss=3.2031 lr=1.11e-04 tok=9.545B dt=5091 wait=115 +23:34:10 step 72840 loss=3.1494 lr=1.11e-04 tok=9.547B dt=5165 wait=115 +23:35:25 step 72860 loss=3.1968 lr=1.11e-04 tok=9.550B dt=5240 wait=116 +23:36:40 step 72880 loss=3.1311 lr=1.11e-04 tok=9.553B dt=5315 wait=116 +23:37:55 step 72900 loss=3.1826 lr=1.11e-04 tok=9.555B dt=5390 wait=117 +23:38:08 pushed ckpt_w21.pt,log_w21.txt +23:39:24 waited 76s for global r=140 +23:39:31 global r=140 load 4.273 -> 4.257 +23:40:50 step 72920 loss=3.1468 lr=1.11e-04 tok=9.558B dt=5565 wait=117 +23:42:05 step 72940 loss=3.1916 lr=1.11e-04 tok=9.560B dt=5640 wait=117 +23:43:20 step 72960 loss=3.2015 lr=1.11e-04 tok=9.563B dt=5715 wait=118 +23:44:35 step 72980 loss=3.2046 lr=1.11e-04 tok=9.566B dt=5790 wait=118 +23:45:50 step 73000 loss=3.1933 lr=1.11e-04 tok=9.568B dt=5865 wait=119 +23:47:05 step 73020 loss=3.1350 lr=1.11e-04 tok=9.571B dt=5940 wait=119 +23:48:19 step 73040 loss=3.1480 lr=1.11e-04 tok=9.573B dt=6015 wait=119 +23:49:10 pushed ckpt_w21.pt,log_w21.txt +23:52:58 step 73060 loss=3.1418 lr=1.11e-04 tok=9.576B dt=6293 wait=120 +23:54:13 step 73080 loss=3.2061 lr=1.11e-04 tok=9.579B dt=6368 wait=120 +23:55:28 step 73100 loss=3.1300 lr=1.11e-04 tok=9.581B dt=6443 wait=121 +23:56:43 step 73120 loss=3.1052 lr=1.11e-04 tok=9.584B dt=6518 wait=121 +23:57:58 step 73140 loss=3.1813 lr=1.11e-04 tok=9.587B dt=6594 wait=121 +23:59:13 step 73160 loss=3.2099 lr=1.11e-04 tok=9.589B dt=6669 wait=122 +00:00:28 step 73180 loss=3.2025 lr=1.11e-04 tok=9.592B dt=6744 wait=122 +00:01:43 step 73200 loss=3.1802 lr=1.11e-04 tok=9.594B dt=6819 wait=123 +00:01:56 pushed ckpt_w21.pt,log_w21.txt +00:01:56 waited 0s for global r=141 +00:02:04 global r=141 load 4.272 -> 4.253 +00:03:23 step 73220 loss=3.1253 lr=1.11e-04 tok=9.597B dt=6918 wait=123 +00:04:38 step 73240 loss=3.1168 lr=1.11e-04 tok=9.600B dt=6993 wait=123 +00:05:52 step 73260 loss=3.1502 lr=1.11e-04 tok=9.602B dt=7068 wait=124 +00:07:07 step 73280 loss=3.2198 lr=1.10e-04 tok=9.605B dt=7143 wait=124 +00:08:23 step 73300 loss=3.2266 lr=1.10e-04 tok=9.608B dt=7218 wait=125 +00:09:38 step 73320 loss=3.1408 lr=1.10e-04 tok=9.610B dt=7293 wait=125 +00:10:53 step 73340 loss=3.1986 lr=1.10e-04 tok=9.613B dt=7368 wait=125 +00:11:48 pushed ckpt_w21.pt,log_w21.txt +00:13:04 waited 76s for global r=142 +00:13:12 global r=142 load 4.284 -> 4.263 +00:13:53 step 73360 loss=3.1117 lr=1.10e-04 tok=9.615B dt=7548 wait=126 +00:15:08 step 73380 loss=3.0999 lr=1.10e-04 tok=9.618B dt=7623 wait=126 +00:16:23 step 73400 loss=3.2048 lr=1.10e-04 tok=9.621B dt=7698 wait=127 +00:17:39 step 73420 loss=3.2260 lr=1.10e-04 tok=9.623B dt=7774 wait=127 +00:18:54 step 73440 loss=3.1688 lr=1.10e-04 tok=9.626B dt=7849 wait=127 +00:20:09 step 73460 loss=3.2497 lr=1.10e-04 tok=9.629B dt=7924 wait=128 +00:21:24 step 73480 loss=3.1294 lr=1.10e-04 tok=9.631B dt=7999 wait=128 +00:22:39 step 73500 loss=3.1778 lr=1.10e-04 tok=9.634B dt=8074 wait=129 +00:22:54 pushed ckpt_w21.pt,log_w21.txt +00:27:20 step 73520 loss=3.1947 lr=1.10e-04 tok=9.636B dt=8356 wait=129 +00:28:35 step 73540 loss=3.0787 lr=1.10e-04 tok=9.639B dt=8430 wait=129 +00:29:50 step 73560 loss=3.1518 lr=1.10e-04 tok=9.642B dt=8505 wait=130 +00:31:05 step 73580 loss=3.1584 lr=1.10e-04 tok=9.644B dt=8580 wait=130 +00:32:20 step 73600 loss=3.1169 lr=1.10e-04 tok=9.647B dt=8655 wait=131 +00:33:35 step 73620 loss=3.1954 lr=1.10e-04 tok=9.650B dt=8730 wait=131 +00:34:50 step 73640 loss=3.1874 lr=1.10e-04 tok=9.652B dt=8805 wait=131 +00:35:41 pushed ckpt_w21.pt,log_w21.txt +00:35:41 waited 0s for global r=143 +00:35:52 global r=143 load 4.275 -> 4.254 +00:36:37 step 73660 loss=3.1743 lr=1.10e-04 tok=9.655B dt=8912 wait=132 +00:37:52 step 73680 loss=3.1338 lr=1.10e-04 tok=9.657B dt=8987 wait=132 +00:39:07 step 73700 loss=3.1663 lr=1.10e-04 tok=9.660B dt=9062 wait=133 +00:40:22 step 73720 loss=3.2145 lr=1.10e-04 tok=9.663B dt=9137 wait=133 +00:41:37 step 73740 loss=3.1338 lr=1.10e-04 tok=9.665B dt=9212 wait=133 +00:42:52 step 73760 loss=3.2248 lr=1.10e-04 tok=9.668B dt=9287 wait=134 +00:44:07 step 73780 loss=3.1008 lr=1.10e-04 tok=9.670B dt=9362 wait=134 +00:45:22 step 73800 loss=3.1382 lr=1.10e-04 tok=9.673B dt=9437 wait=135 +00:45:35 pushed ckpt_w21.pt,log_w21.txt +00:46:20 waited 45s for global r=144 +00:46:33 global r=144 load 4.276 -> 4.261 +00:47:52 step 73820 loss=3.1486 lr=1.10e-04 tok=9.676B dt=9587 wait=135 +00:49:07 step 73840 loss=3.1766 lr=1.10e-04 tok=9.678B dt=9662 wait=135 +00:50:22 step 73860 loss=3.1437 lr=1.10e-04 tok=9.681B dt=9737 wait=136 +00:51:37 step 73880 loss=3.2053 lr=1.10e-04 tok=9.684B dt=9812 wait=136 +00:52:52 step 73900 loss=3.1889 lr=1.10e-04 tok=9.686B dt=9887 wait=137 +00:54:07 step 73920 loss=3.1480 lr=1.10e-04 tok=9.689B dt=9962 wait=137 +00:55:22 step 73940 loss=3.1852 lr=1.10e-04 tok=9.691B dt=10037 wait=137 diff --git a/log_w22.txt b/log_w22.txt new file mode 100644 index 0000000000000000000000000000000000000000..0bdac895f737c95a99c768972a432259b09fa52a --- /dev/null +++ b/log_w22.txt @@ -0,0 +1,453 @@ +launch +15:55:53 name=w22 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:55:59 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:56:08 params 1.08B +15:56:10 resumed step=67050 seen=8.79B round=123 +15:56:10 stream salt=67050 +15:58:10 start: accum=4, 131072 tokens per optimiser step +15:59:53 step 67051 loss=3.4140 lr=2.33e-04 tok=8.789B dt=103 wait=73 +15:59:58 step 67052 loss=3.3105 lr=2.33e-04 tok=8.789B dt=108 wait=73 +16:00:36 step 67060 loss=3.3688 lr=2.33e-04 tok=8.790B dt=146 wait=73 +16:02:09 step 67080 loss=3.4215 lr=2.33e-04 tok=8.792B dt=239 wait=74 +16:03:24 step 67100 loss=3.3935 lr=2.33e-04 tok=8.795B dt=314 wait=74 +launch +16:04:21 name=w22 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:04:25 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:04:35 params 1.08B +16:04:36 resumed step=67050 seen=8.79B round=123 +16:04:36 stream salt=67050 +16:06:51 start: accum=4, 131072 tokens per optimiser step +16:08:12 step 67051 loss=3.4140 lr=1.16e-04 tok=8.789B dt=81 wait=62 +16:08:17 step 67052 loss=3.3000 lr=1.16e-04 tok=8.789B dt=86 wait=62 +16:08:57 step 67060 loss=3.3283 lr=1.16e-04 tok=8.790B dt=126 wait=63 +16:10:32 step 67080 loss=3.3530 lr=1.16e-04 tok=8.792B dt=222 wait=63 +16:11:48 step 67100 loss=3.3177 lr=1.16e-04 tok=8.795B dt=297 wait=64 +16:13:03 step 67120 loss=3.3303 lr=1.16e-04 tok=8.798B dt=372 wait=64 +16:14:18 step 67140 loss=3.2704 lr=1.16e-04 tok=8.800B dt=447 wait=65 +16:15:33 step 67160 loss=3.1681 lr=1.16e-04 tok=8.803B dt=522 wait=65 +16:16:48 step 67180 loss=3.2098 lr=1.16e-04 tok=8.805B dt=598 wait=66 +16:18:03 step 67200 loss=3.2924 lr=1.16e-04 tok=8.808B dt=673 wait=66 +16:18:20 pushed ckpt_w22.pt,log_w22.txt +16:18:20 waited 0s for global r=124 +16:18:39 global r=124 load 4.332 -> 4.332 +16:19:59 step 67220 loss=3.2074 lr=1.16e-04 tok=8.811B dt=788 wait=66 +16:21:14 step 67240 loss=3.2203 lr=1.16e-04 tok=8.813B dt=864 wait=67 +16:22:29 step 67260 loss=3.2122 lr=1.16e-04 tok=8.816B dt=939 wait=67 +16:23:45 step 67280 loss=3.2879 lr=1.16e-04 tok=8.819B dt=1014 wait=68 +16:25:00 step 67300 loss=3.2865 lr=1.16e-04 tok=8.821B dt=1089 wait=68 +16:26:17 step 67320 loss=3.2464 lr=1.16e-04 tok=8.824B dt=1166 wait=69 +16:27:36 step 67340 loss=3.2419 lr=1.16e-04 tok=8.826B dt=1245 wait=69 +16:28:26 pushed ckpt_w22.pt,log_w22.txt +16:31:28 waited 182s for global r=125 +16:31:36 global r=125 load 4.324 -> 4.296 +16:32:22 step 67360 loss=3.1960 lr=1.16e-04 tok=8.829B dt=1532 wait=69 +16:33:37 step 67380 loss=3.2111 lr=1.16e-04 tok=8.832B dt=1607 wait=70 +16:34:52 step 67400 loss=3.2459 lr=1.16e-04 tok=8.834B dt=1682 wait=70 +16:36:08 step 67420 loss=3.2551 lr=1.16e-04 tok=8.837B dt=1757 wait=71 +16:37:22 step 67440 loss=3.1840 lr=1.16e-04 tok=8.839B dt=1832 wait=71 +16:38:37 step 67460 loss=3.3125 lr=1.16e-04 tok=8.842B dt=1907 wait=71 +16:39:52 step 67480 loss=3.2092 lr=1.16e-04 tok=8.845B dt=1982 wait=72 +16:41:07 step 67500 loss=3.2557 lr=1.16e-04 tok=8.847B dt=2057 wait=72 +16:41:20 pushed ckpt_w22.pt,log_w22.txt +16:45:40 step 67520 loss=3.2509 lr=1.16e-04 tok=8.850B dt=2330 wait=73 +16:46:55 step 67540 loss=3.1859 lr=1.16e-04 tok=8.853B dt=2405 wait=73 +16:48:10 step 67560 loss=3.1858 lr=1.16e-04 tok=8.855B dt=2479 wait=73 +16:49:25 step 67580 loss=3.1892 lr=1.16e-04 tok=8.858B dt=2554 wait=74 +16:50:39 step 67600 loss=3.1188 lr=1.16e-04 tok=8.860B dt=2629 wait=74 +16:51:54 step 67620 loss=3.2589 lr=1.16e-04 tok=8.863B dt=2704 wait=75 +16:53:09 step 67640 loss=3.1838 lr=1.16e-04 tok=8.866B dt=2779 wait=75 +16:54:00 pushed ckpt_w22.pt,log_w22.txt +16:54:00 waited 0s for global r=126 +16:54:09 global r=126 load 4.316 -> 4.295 +16:54:54 step 67660 loss=3.1785 lr=1.16e-04 tok=8.868B dt=2883 wait=75 +16:56:09 step 67680 loss=3.1969 lr=1.16e-04 tok=8.871B dt=2958 wait=76 +16:57:24 step 67700 loss=3.2441 lr=1.16e-04 tok=8.874B dt=3033 wait=76 +16:58:39 step 67720 loss=3.2606 lr=1.16e-04 tok=8.876B dt=3108 wait=76 +16:59:53 step 67740 loss=3.1648 lr=1.16e-04 tok=8.879B dt=3183 wait=77 +17:01:08 step 67760 loss=3.2596 lr=1.16e-04 tok=8.881B dt=3258 wait=77 +17:02:23 step 67780 loss=3.1598 lr=1.16e-04 tok=8.884B dt=3332 wait=78 +17:03:38 step 67800 loss=3.1885 lr=1.16e-04 tok=8.887B dt=3407 wait=78 +17:03:50 pushed ckpt_w22.pt,log_w22.txt +17:05:36 waited 106s for global r=127 +17:05:44 global r=127 load 4.312 -> 4.287 +17:07:07 step 67820 loss=3.1963 lr=1.16e-04 tok=8.889B dt=3616 wait=78 +17:08:22 step 67840 loss=3.2762 lr=1.16e-04 tok=8.892B dt=3691 wait=79 +17:09:37 step 67860 loss=3.2425 lr=1.16e-04 tok=8.895B dt=3767 wait=79 +17:10:52 step 67880 loss=3.2825 lr=1.16e-04 tok=8.897B dt=3842 wait=80 +17:12:07 step 67900 loss=3.2450 lr=1.16e-04 tok=8.900B dt=3917 wait=80 +17:13:22 step 67920 loss=3.1965 lr=1.16e-04 tok=8.902B dt=3991 wait=80 +17:14:37 step 67940 loss=3.2221 lr=1.16e-04 tok=8.905B dt=4066 wait=81 +17:15:28 pushed ckpt_w22.pt,log_w22.txt +17:19:16 step 67960 loss=3.2177 lr=1.16e-04 tok=8.908B dt=4345 wait=81 +17:20:31 step 67980 loss=3.2613 lr=1.16e-04 tok=8.910B dt=4420 wait=82 +17:21:46 step 68000 loss=3.2106 lr=1.15e-04 tok=8.913B dt=4495 wait=82 +17:23:00 step 68020 loss=3.1970 lr=1.15e-04 tok=8.916B dt=4570 wait=82 +17:24:15 step 68040 loss=3.1963 lr=1.15e-04 tok=8.918B dt=4645 wait=83 +17:25:30 step 68060 loss=3.1683 lr=1.15e-04 tok=8.921B dt=4719 wait=83 +17:26:45 step 68080 loss=3.2291 lr=1.15e-04 tok=8.923B dt=4794 wait=84 +17:28:00 step 68100 loss=3.1979 lr=1.15e-04 tok=8.926B dt=4869 wait=84 +17:28:11 pushed ckpt_w22.pt,log_w22.txt +17:28:12 waited 0s for global r=128 +17:28:19 global r=128 load 4.313 -> 4.293 +17:29:37 step 68120 loss=3.1802 lr=1.15e-04 tok=8.929B dt=4967 wait=84 +17:30:52 step 68140 loss=3.2296 lr=1.15e-04 tok=8.931B dt=5042 wait=85 +17:32:07 step 68160 loss=3.2156 lr=1.15e-04 tok=8.934B dt=5117 wait=85 +17:33:22 step 68180 loss=3.2866 lr=1.15e-04 tok=8.936B dt=5191 wait=86 +17:34:37 step 68200 loss=3.2583 lr=1.15e-04 tok=8.939B dt=5266 wait=86 +17:35:52 step 68220 loss=3.1549 lr=1.15e-04 tok=8.942B dt=5341 wait=86 +17:37:07 step 68240 loss=3.2306 lr=1.15e-04 tok=8.944B dt=5416 wait=87 +17:37:59 pushed ckpt_w22.pt,log_w22.txt +17:39:14 waited 76s for global r=129 +17:39:23 global r=129 load 4.303 -> 4.283 +17:40:03 step 68260 loss=3.2134 lr=1.15e-04 tok=8.947B dt=5593 wait=87 +17:41:18 step 68280 loss=3.1627 lr=1.15e-04 tok=8.950B dt=5668 wait=88 +17:42:33 step 68300 loss=3.2469 lr=1.15e-04 tok=8.952B dt=5743 wait=88 +17:43:48 step 68320 loss=3.2579 lr=1.15e-04 tok=8.955B dt=5818 wait=88 +17:45:03 step 68340 loss=3.1909 lr=1.15e-04 tok=8.957B dt=5893 wait=89 +17:46:18 step 68360 loss=3.2691 lr=1.15e-04 tok=8.960B dt=5967 wait=89 +17:47:33 step 68380 loss=3.1268 lr=1.15e-04 tok=8.963B dt=6042 wait=90 +17:48:48 step 68400 loss=3.1790 lr=1.15e-04 tok=8.965B dt=6117 wait=90 +17:49:01 pushed ckpt_w22.pt,log_w22.txt +17:53:23 step 68420 loss=3.2210 lr=1.15e-04 tok=8.968B dt=6393 wait=90 +17:54:38 step 68440 loss=3.1670 lr=1.15e-04 tok=8.971B dt=6467 wait=91 +17:55:53 step 68460 loss=3.1632 lr=1.15e-04 tok=8.973B dt=6543 wait=91 +17:57:08 step 68480 loss=3.1546 lr=1.15e-04 tok=8.976B dt=6618 wait=92 +17:58:23 step 68500 loss=3.1271 lr=1.15e-04 tok=8.978B dt=6692 wait=92 +17:59:38 step 68520 loss=3.2524 lr=1.15e-04 tok=8.981B dt=6767 wait=92 +18:00:53 step 68540 loss=3.1969 lr=1.15e-04 tok=8.984B dt=6842 wait=93 +18:01:42 pushed ckpt_w22.pt,log_w22.txt +18:01:42 waited 0s for global r=130 +18:01:50 global r=130 load 4.300 -> 4.288 +18:02:34 step 68560 loss=3.1592 lr=1.15e-04 tok=8.986B dt=6943 wait=93 +18:03:49 step 68580 loss=3.1987 lr=1.15e-04 tok=8.989B dt=7018 wait=94 +18:05:04 step 68600 loss=3.2147 lr=1.15e-04 tok=8.992B dt=7093 wait=94 +18:06:19 step 68620 loss=3.2691 lr=1.15e-04 tok=8.994B dt=7168 wait=95 +18:07:34 step 68640 loss=3.2037 lr=1.15e-04 tok=8.997B dt=7243 wait=95 +18:08:49 step 68660 loss=3.2900 lr=1.15e-04 tok=8.999B dt=7318 wait=95 +18:10:04 step 68680 loss=3.1562 lr=1.15e-04 tok=9.002B dt=7393 wait=96 +18:11:19 step 68700 loss=3.2076 lr=1.15e-04 tok=9.005B dt=7468 wait=96 +18:11:35 pushed ckpt_w22.pt,log_w22.txt +18:11:35 waited 0s for global r=131 +18:11:45 global r=131 load 4.298 -> 4.290 +18:13:09 step 68720 loss=3.1528 lr=1.15e-04 tok=9.007B dt=7578 wait=97 +18:14:23 step 68740 loss=3.1985 lr=1.15e-04 tok=9.010B dt=7653 wait=97 +18:15:38 step 68760 loss=3.1923 lr=1.15e-04 tok=9.013B dt=7728 wait=97 +18:16:54 step 68780 loss=3.2624 lr=1.15e-04 tok=9.015B dt=7803 wait=98 +18:18:09 step 68800 loss=3.2277 lr=1.15e-04 tok=9.018B dt=7878 wait=98 +18:19:24 step 68820 loss=3.2272 lr=1.15e-04 tok=9.020B dt=7953 wait=99 +18:20:39 step 68840 loss=3.2248 lr=1.15e-04 tok=9.023B dt=8028 wait=99 +18:21:28 pushed ckpt_w22.pt,log_w22.txt +18:25:15 step 68860 loss=3.1853 lr=1.15e-04 tok=9.026B dt=8305 wait=99 +18:26:30 step 68880 loss=3.2446 lr=1.15e-04 tok=9.028B dt=8379 wait=100 +18:27:45 step 68900 loss=3.1248 lr=1.15e-04 tok=9.031B dt=8454 wait=100 +18:29:00 step 68920 loss=3.1147 lr=1.15e-04 tok=9.033B dt=8529 wait=101 +18:30:15 step 68940 loss=3.1779 lr=1.15e-04 tok=9.036B dt=8604 wait=101 +18:31:29 step 68960 loss=3.1811 lr=1.15e-04 tok=9.039B dt=8679 wait=101 +18:32:44 step 68980 loss=3.2407 lr=1.15e-04 tok=9.041B dt=8754 wait=102 +18:33:59 step 69000 loss=3.2019 lr=1.15e-04 tok=9.044B dt=8829 wait=102 +18:34:11 pushed ckpt_w22.pt,log_w22.txt +18:34:11 waited 0s for global r=132 +18:34:17 global r=132 load 4.303 -> 4.285 +18:35:39 step 69020 loss=3.1824 lr=1.15e-04 tok=9.047B dt=8929 wait=103 +18:36:54 step 69040 loss=3.1921 lr=1.15e-04 tok=9.049B dt=9004 wait=103 +18:38:09 step 69060 loss=3.1634 lr=1.14e-04 tok=9.052B dt=9079 wait=103 +18:39:24 step 69080 loss=3.2420 lr=1.14e-04 tok=9.054B dt=9154 wait=104 +18:40:39 step 69100 loss=3.2329 lr=1.14e-04 tok=9.057B dt=9229 wait=104 +18:41:54 step 69120 loss=3.2078 lr=1.14e-04 tok=9.060B dt=9304 wait=105 +18:43:09 step 69140 loss=3.2217 lr=1.14e-04 tok=9.062B dt=9379 wait=105 +18:43:58 pushed ckpt_w22.pt,log_w22.txt +18:43:59 waited 0s for global r=133 +18:44:09 global r=133 load 4.298 -> 4.282 +18:44:50 step 69160 loss=3.1558 lr=1.14e-04 tok=9.065B dt=9479 wait=105 +18:46:05 step 69180 loss=3.1635 lr=1.14e-04 tok=9.068B dt=9554 wait=106 +18:47:20 step 69200 loss=3.2361 lr=1.14e-04 tok=9.070B dt=9629 wait=106 +18:48:35 step 69220 loss=3.2713 lr=1.14e-04 tok=9.073B dt=9704 wait=107 +18:49:50 step 69240 loss=3.2016 lr=1.14e-04 tok=9.075B dt=9779 wait=107 +18:51:05 step 69260 loss=3.3069 lr=1.14e-04 tok=9.078B dt=9854 wait=107 +18:52:20 step 69280 loss=3.1911 lr=1.14e-04 tok=9.081B dt=9929 wait=108 +18:53:35 step 69300 loss=3.2164 lr=1.14e-04 tok=9.083B dt=10005 wait=108 +18:53:49 pushed ckpt_w22.pt,log_w22.txt +18:58:17 step 69320 loss=3.2303 lr=1.14e-04 tok=9.086B dt=10287 wait=109 +18:59:32 step 69340 loss=3.1232 lr=1.14e-04 tok=9.089B dt=10361 wait=109 +19:00:47 step 69360 loss=3.1746 lr=1.14e-04 tok=9.091B dt=10436 wait=109 +19:02:02 step 69380 loss=3.1642 lr=1.14e-04 tok=9.094B dt=10511 wait=110 +19:03:17 step 69400 loss=3.1300 lr=1.14e-04 tok=9.096B dt=10586 wait=110 +19:04:31 step 69420 loss=3.2518 lr=1.14e-04 tok=9.099B dt=10661 wait=111 +19:05:47 step 69440 loss=3.2299 lr=1.14e-04 tok=9.102B dt=10736 wait=111 +19:06:39 pushed ckpt_w22.pt,log_w22.txt +19:10:25 step 69460 loss=3.2166 lr=1.14e-04 tok=9.104B dt=11014 wait=111 +19:11:40 step 69480 loss=3.1635 lr=1.14e-04 tok=9.107B dt=11089 wait=112 +19:12:54 step 69500 loss=3.1883 lr=1.14e-04 tok=9.110B dt=11164 wait=112 +19:14:09 step 69520 loss=3.1723 lr=1.14e-04 tok=9.112B dt=11239 wait=113 +19:15:24 step 69540 loss=3.2016 lr=1.14e-04 tok=9.115B dt=11314 wait=113 +19:16:39 step 69560 loss=3.1684 lr=1.14e-04 tok=9.117B dt=11389 wait=113 +19:17:54 step 69580 loss=3.2511 lr=1.14e-04 tok=9.120B dt=11464 wait=114 +19:19:09 step 69600 loss=3.1436 lr=1.14e-04 tok=9.123B dt=11538 wait=114 +19:19:24 pushed ckpt_w22.pt,log_w22.txt +19:23:46 step 69620 loss=3.1929 lr=1.14e-04 tok=9.125B dt=11815 wait=114 +19:25:01 step 69640 loss=3.2212 lr=1.14e-04 tok=9.128B dt=11890 wait=115 +19:26:16 step 69660 loss=3.1719 lr=1.14e-04 tok=9.130B dt=11965 wait=115 +19:27:30 step 69680 loss=3.3079 lr=1.14e-04 tok=9.133B dt=12040 wait=116 +19:28:45 step 69700 loss=3.2526 lr=1.14e-04 tok=9.136B dt=12115 wait=116 +19:30:00 step 69720 loss=3.3161 lr=1.14e-04 tok=9.138B dt=12190 wait=116 +19:31:15 step 69740 loss=3.2519 lr=1.14e-04 tok=9.141B dt=12265 wait=117 +19:32:05 pushed ckpt_w22.pt,log_w22.txt +19:35:52 step 69760 loss=3.2626 lr=1.14e-04 tok=9.144B dt=12541 wait=117 +19:37:07 step 69780 loss=3.1643 lr=1.14e-04 tok=9.146B dt=12616 wait=118 +19:38:22 step 69800 loss=3.1826 lr=1.14e-04 tok=9.149B dt=12691 wait=118 +19:39:37 step 69820 loss=3.1906 lr=1.14e-04 tok=9.151B dt=12766 wait=118 +19:40:52 step 69840 loss=3.1839 lr=1.14e-04 tok=9.154B dt=12841 wait=119 +19:42:06 step 69860 loss=3.1512 lr=1.14e-04 tok=9.157B dt=12916 wait=119 +19:43:21 step 69880 loss=3.1569 lr=1.14e-04 tok=9.159B dt=12991 wait=120 +19:44:36 step 69900 loss=3.1798 lr=1.14e-04 tok=9.162B dt=13066 wait=120 +19:44:50 pushed ckpt_w22.pt,log_w22.txt +19:49:10 step 69920 loss=3.1873 lr=1.14e-04 tok=9.165B dt=13340 wait=120 +19:50:25 step 69940 loss=3.1408 lr=1.14e-04 tok=9.167B dt=13415 wait=121 +19:51:40 step 69960 loss=3.3108 lr=1.14e-04 tok=9.170B dt=13490 wait=121 +19:52:55 step 69980 loss=3.1992 lr=1.14e-04 tok=9.172B dt=13565 wait=122 +19:54:11 step 70000 loss=3.2774 lr=1.14e-04 tok=9.175B dt=13640 wait=122 +19:55:25 step 70020 loss=3.2258 lr=1.14e-04 tok=9.178B dt=13715 wait=122 +19:56:40 step 70040 loss=3.1853 lr=1.14e-04 tok=9.180B dt=13790 wait=123 +19:57:29 pushed ckpt_w22.pt,log_w22.txt +20:01:14 step 70060 loss=3.2999 lr=1.14e-04 tok=9.183B dt=14063 wait=123 +20:02:28 step 70080 loss=3.1664 lr=1.14e-04 tok=9.186B dt=14138 wait=124 +20:03:43 step 70100 loss=3.2988 lr=1.14e-04 tok=9.188B dt=14213 wait=124 +20:04:58 step 70120 loss=3.1556 lr=1.14e-04 tok=9.191B dt=14288 wait=124 +20:06:13 step 70140 loss=3.1671 lr=1.13e-04 tok=9.193B dt=14362 wait=125 +20:07:28 step 70160 loss=3.1614 lr=1.13e-04 tok=9.196B dt=14437 wait=125 +20:08:43 step 70180 loss=3.2201 lr=1.13e-04 tok=9.199B dt=14512 wait=125 +20:09:58 step 70200 loss=3.2202 lr=1.13e-04 tok=9.201B dt=14587 wait=126 +20:10:10 pushed ckpt_w22.pt,log_w22.txt +20:14:32 step 70220 loss=3.2406 lr=1.13e-04 tok=9.204B dt=14862 wait=126 +20:15:47 step 70240 loss=3.1699 lr=1.13e-04 tok=9.206B dt=14937 wait=127 +20:17:02 step 70260 loss=3.3245 lr=1.13e-04 tok=9.209B dt=15012 wait=127 +20:18:17 step 70280 loss=3.2133 lr=1.13e-04 tok=9.212B dt=15087 wait=127 +20:19:32 step 70300 loss=3.1859 lr=1.13e-04 tok=9.214B dt=15162 wait=128 +20:20:47 step 70320 loss=3.1033 lr=1.13e-04 tok=9.217B dt=15236 wait=128 +20:22:02 step 70340 loss=3.1808 lr=1.13e-04 tok=9.220B dt=15311 wait=129 +20:22:51 pushed ckpt_w22.pt,log_w22.txt +20:26:38 step 70360 loss=3.2517 lr=1.13e-04 tok=9.222B dt=15587 wait=129 +20:27:53 step 70380 loss=3.2897 lr=1.13e-04 tok=9.225B dt=15663 wait=129 +20:29:08 step 70400 loss=3.3021 lr=1.13e-04 tok=9.227B dt=15738 wait=130 +20:30:23 step 70420 loss=3.2419 lr=1.13e-04 tok=9.230B dt=15813 wait=130 +20:31:38 step 70440 loss=3.1392 lr=1.13e-04 tok=9.233B dt=15888 wait=131 +20:32:53 step 70460 loss=3.1381 lr=1.13e-04 tok=9.235B dt=15962 wait=131 +20:34:08 step 70480 loss=3.2852 lr=1.13e-04 tok=9.238B dt=16038 wait=131 +20:35:23 step 70500 loss=3.1981 lr=1.13e-04 tok=9.241B dt=16112 wait=132 +20:35:36 pushed ckpt_w22.pt,log_w22.txt +20:39:59 step 70520 loss=3.2736 lr=1.13e-04 tok=9.243B dt=16388 wait=132 +20:41:13 step 70540 loss=3.2885 lr=1.13e-04 tok=9.246B dt=16463 wait=133 +20:42:28 step 70560 loss=3.2268 lr=1.13e-04 tok=9.248B dt=16538 wait=133 +20:43:43 step 70580 loss=3.2682 lr=1.13e-04 tok=9.251B dt=16613 wait=133 +20:44:58 step 70600 loss=3.1687 lr=1.13e-04 tok=9.254B dt=16688 wait=134 +20:46:13 step 70620 loss=3.2121 lr=1.13e-04 tok=9.256B dt=16763 wait=134 +20:47:28 step 70640 loss=3.2509 lr=1.13e-04 tok=9.259B dt=16837 wait=134 +20:48:18 pushed ckpt_w22.pt,log_w22.txt +20:52:02 step 70660 loss=3.2753 lr=1.13e-04 tok=9.262B dt=17112 wait=135 +20:53:17 step 70680 loss=3.2304 lr=1.13e-04 tok=9.264B dt=17186 wait=135 +20:54:32 step 70700 loss=3.1816 lr=1.13e-04 tok=9.267B dt=17261 wait=136 +20:55:47 step 70720 loss=3.2177 lr=1.13e-04 tok=9.269B dt=17336 wait=136 +20:57:02 step 70740 loss=3.1714 lr=1.13e-04 tok=9.272B dt=17411 wait=136 +20:58:17 step 70760 loss=3.3074 lr=1.13e-04 tok=9.275B dt=17486 wait=137 +20:59:32 step 70780 loss=3.2368 lr=1.13e-04 tok=9.277B dt=17561 wait=137 +21:00:47 step 70800 loss=3.1398 lr=1.13e-04 tok=9.280B dt=17636 wait=138 +21:01:01 pushed ckpt_w22.pt,log_w22.txt +21:05:24 step 70820 loss=3.1610 lr=1.13e-04 tok=9.283B dt=17913 wait=138 +21:06:39 step 70840 loss=3.2345 lr=1.13e-04 tok=9.285B dt=17988 wait=138 +21:07:54 step 70860 loss=3.1669 lr=1.13e-04 tok=9.288B dt=18063 wait=139 +21:09:09 step 70880 loss=3.1482 lr=1.13e-04 tok=9.290B dt=18138 wait=139 +21:10:23 step 70900 loss=3.2002 lr=1.13e-04 tok=9.293B dt=18213 wait=139 +21:11:38 step 70920 loss=3.1659 lr=1.13e-04 tok=9.296B dt=18288 wait=140 +21:12:53 step 70940 loss=3.2166 lr=1.13e-04 tok=9.298B dt=18362 wait=140 +21:13:43 pushed ckpt_w22.pt,log_w22.txt +21:17:27 step 70960 loss=3.1902 lr=1.13e-04 tok=9.301B dt=18637 wait=141 +21:18:43 step 70980 loss=3.2951 lr=1.13e-04 tok=9.303B dt=18712 wait=141 +21:19:58 step 71000 loss=3.2025 lr=1.13e-04 tok=9.306B dt=18787 wait=141 +21:21:13 step 71020 loss=3.2339 lr=1.13e-04 tok=9.309B dt=18862 wait=142 +21:22:27 step 71040 loss=3.0566 lr=1.13e-04 tok=9.311B dt=18937 wait=142 +21:23:42 step 71060 loss=3.1901 lr=1.13e-04 tok=9.314B dt=19012 wait=143 +21:24:57 step 71080 loss=3.3230 lr=1.13e-04 tok=9.317B dt=19087 wait=143 +21:26:12 step 71100 loss=3.2927 lr=1.13e-04 tok=9.319B dt=19162 wait=143 +21:26:25 pushed ckpt_w22.pt,log_w22.txt +21:30:50 step 71120 loss=3.1655 lr=1.13e-04 tok=9.322B dt=19439 wait=144 +21:32:04 step 71140 loss=3.2449 lr=1.13e-04 tok=9.324B dt=19514 wait=144 +21:33:19 step 71160 loss=3.1502 lr=1.13e-04 tok=9.327B dt=19589 wait=144 +21:34:34 step 71180 loss=3.2330 lr=1.13e-04 tok=9.330B dt=19664 wait=145 +21:35:49 step 71200 loss=3.1535 lr=1.12e-04 tok=9.332B dt=19739 wait=145 +21:37:04 step 71220 loss=3.2227 lr=1.12e-04 tok=9.335B dt=19813 wait=146 +21:38:19 step 71240 loss=3.2360 lr=1.12e-04 tok=9.338B dt=19889 wait=146 +21:39:09 pushed ckpt_w22.pt,log_w22.txt +21:42:58 step 71260 loss=3.2819 lr=1.12e-04 tok=9.340B dt=20168 wait=146 +21:44:13 step 71280 loss=3.1218 lr=1.12e-04 tok=9.343B dt=20243 wait=147 +21:45:28 step 71300 loss=3.2293 lr=1.12e-04 tok=9.345B dt=20317 wait=147 +21:46:43 step 71320 loss=3.2017 lr=1.12e-04 tok=9.348B dt=20392 wait=148 +21:47:58 step 71340 loss=3.2499 lr=1.12e-04 tok=9.351B dt=20467 wait=148 +21:49:13 step 71360 loss=3.2353 lr=1.12e-04 tok=9.353B dt=20542 wait=148 +21:50:28 step 71380 loss=3.2733 lr=1.12e-04 tok=9.356B dt=20617 wait=149 +21:51:43 step 71400 loss=3.1726 lr=1.12e-04 tok=9.359B dt=20692 wait=149 +21:51:59 pushed ckpt_w22.pt,log_w22.txt +21:56:22 step 71420 loss=3.1558 lr=1.12e-04 tok=9.361B dt=20971 wait=150 +21:57:37 step 71440 loss=3.2941 lr=1.12e-04 tok=9.364B dt=21046 wait=150 +21:58:52 step 71460 loss=3.2208 lr=1.12e-04 tok=9.366B dt=21121 wait=150 +22:00:07 step 71480 loss=3.1781 lr=1.12e-04 tok=9.369B dt=21196 wait=151 +22:01:22 step 71500 loss=3.1842 lr=1.12e-04 tok=9.372B dt=21271 wait=151 +22:02:36 step 71520 loss=3.2128 lr=1.12e-04 tok=9.374B dt=21346 wait=152 +22:03:51 step 71540 loss=3.1561 lr=1.12e-04 tok=9.377B dt=21420 wait=152 +22:04:41 pushed ckpt_w22.pt,log_w22.txt +22:04:41 waited 0s for global r=134 +22:04:53 global r=134 load 4.317 -> 4.266 +22:05:37 step 71560 loss=3.1473 lr=1.12e-04 tok=9.380B dt=21527 wait=152 +22:06:53 step 71580 loss=3.1526 lr=1.12e-04 tok=9.382B dt=21602 wait=153 +22:08:08 step 71600 loss=3.2342 lr=1.12e-04 tok=9.385B dt=21677 wait=153 +22:09:23 step 71620 loss=3.2486 lr=1.12e-04 tok=9.387B dt=21752 wait=154 +22:10:38 step 71640 loss=3.1424 lr=1.12e-04 tok=9.390B dt=21827 wait=154 +22:11:53 step 71660 loss=3.2696 lr=1.12e-04 tok=9.393B dt=21902 wait=154 +22:13:08 step 71680 loss=3.1188 lr=1.12e-04 tok=9.395B dt=21977 wait=155 +22:14:23 step 71700 loss=3.1686 lr=1.12e-04 tok=9.398B dt=22052 wait=155 +22:14:36 pushed ckpt_w22.pt,log_w22.txt +22:15:37 waited 61s for global r=135 +22:15:53 global r=135 load 4.285 -> 4.267 +22:17:14 step 71720 loss=3.1745 lr=1.12e-04 tok=9.400B dt=22223 wait=155 +22:18:29 step 71740 loss=3.1961 lr=1.12e-04 tok=9.403B dt=22298 wait=156 +22:19:44 step 71760 loss=3.1528 lr=1.12e-04 tok=9.406B dt=22373 wait=156 +22:20:59 step 71780 loss=3.2169 lr=1.12e-04 tok=9.408B dt=22448 wait=157 +22:22:14 step 71800 loss=3.2508 lr=1.12e-04 tok=9.411B dt=22523 wait=157 +22:23:29 step 71820 loss=3.1876 lr=1.12e-04 tok=9.414B dt=22598 wait=157 +22:24:44 step 71840 loss=3.2019 lr=1.12e-04 tok=9.416B dt=22673 wait=158 +22:25:34 pushed ckpt_w22.pt,log_w22.txt +22:29:21 step 71860 loss=3.1264 lr=1.12e-04 tok=9.419B dt=22951 wait=158 +22:30:37 step 71880 loss=3.2140 lr=1.12e-04 tok=9.421B dt=23026 wait=159 +22:31:52 step 71900 loss=3.1347 lr=1.12e-04 tok=9.424B dt=23102 wait=159 +22:33:07 step 71920 loss=3.1099 lr=1.12e-04 tok=9.427B dt=23177 wait=159 +22:34:23 step 71940 loss=3.1305 lr=1.12e-04 tok=9.429B dt=23252 wait=160 +22:35:37 step 71960 loss=3.1652 lr=1.12e-04 tok=9.432B dt=23327 wait=160 +22:36:53 step 71980 loss=3.2271 lr=1.12e-04 tok=9.435B dt=23402 wait=161 +22:38:08 step 72000 loss=3.1714 lr=1.12e-04 tok=9.437B dt=23477 wait=161 +22:38:22 pushed ckpt_w22.pt,log_w22.txt +22:38:22 waited 0s for global r=136 +22:38:30 global r=136 load 4.287 -> 4.264 +22:39:51 step 72020 loss=3.1148 lr=1.12e-04 tok=9.440B dt=23580 wait=161 +22:41:06 step 72040 loss=3.1807 lr=1.12e-04 tok=9.442B dt=23655 wait=162 +22:42:21 step 72060 loss=3.1983 lr=1.12e-04 tok=9.445B dt=23730 wait=162 +22:43:36 step 72080 loss=3.1935 lr=1.12e-04 tok=9.448B dt=23806 wait=162 +22:44:51 step 72100 loss=3.2024 lr=1.12e-04 tok=9.450B dt=23881 wait=163 +22:46:06 step 72120 loss=3.1623 lr=1.12e-04 tok=9.453B dt=23955 wait=163 +22:47:21 step 72140 loss=3.2086 lr=1.12e-04 tok=9.456B dt=24030 wait=164 +22:48:16 pushed ckpt_w22.pt,log_w22.txt +22:48:16 waited 0s for global r=137 +22:48:25 global r=137 load 4.279 -> 4.258 +22:49:07 step 72160 loss=3.1430 lr=1.12e-04 tok=9.458B dt=24136 wait=164 +22:50:22 step 72180 loss=3.1118 lr=1.12e-04 tok=9.461B dt=24211 wait=164 +22:51:37 step 72200 loss=3.2048 lr=1.12e-04 tok=9.463B dt=24286 wait=165 +22:52:52 step 72220 loss=3.2692 lr=1.12e-04 tok=9.466B dt=24361 wait=165 +22:54:07 step 72240 loss=3.1818 lr=1.11e-04 tok=9.469B dt=24436 wait=166 +22:55:22 step 72260 loss=3.2541 lr=1.11e-04 tok=9.471B dt=24512 wait=166 +22:56:37 step 72280 loss=3.0681 lr=1.11e-04 tok=9.474B dt=24586 wait=166 +22:57:53 step 72300 loss=3.1772 lr=1.11e-04 tok=9.477B dt=24662 wait=167 +22:58:05 pushed ckpt_w22.pt,log_w22.txt +23:02:25 step 72320 loss=3.2199 lr=1.11e-04 tok=9.479B dt=24934 wait=167 +23:03:40 step 72340 loss=3.1135 lr=1.11e-04 tok=9.482B dt=25009 wait=168 +23:04:54 step 72360 loss=3.1092 lr=1.11e-04 tok=9.484B dt=25084 wait=168 +23:06:10 step 72380 loss=3.1617 lr=1.11e-04 tok=9.487B dt=25160 wait=168 +23:07:25 step 72400 loss=3.1317 lr=1.11e-04 tok=9.490B dt=25235 wait=169 +23:08:40 step 72420 loss=3.2307 lr=1.11e-04 tok=9.492B dt=25310 wait=169 +23:09:55 step 72440 loss=3.1335 lr=1.11e-04 tok=9.495B dt=25384 wait=170 +23:10:45 pushed ckpt_w22.pt,log_w22.txt +23:10:45 waited 0s for global r=138 +23:10:56 global r=138 load 4.279 -> 4.256 +23:11:39 step 72460 loss=3.1274 lr=1.11e-04 tok=9.497B dt=25488 wait=170 +23:12:54 step 72480 loss=3.1464 lr=1.11e-04 tok=9.500B dt=25563 wait=170 +23:14:09 step 72500 loss=3.1756 lr=1.11e-04 tok=9.503B dt=25638 wait=171 +23:15:24 step 72520 loss=3.2508 lr=1.11e-04 tok=9.505B dt=25713 wait=171 +23:16:39 step 72540 loss=3.1821 lr=1.11e-04 tok=9.508B dt=25788 wait=172 +23:17:54 step 72560 loss=3.2716 lr=1.11e-04 tok=9.511B dt=25863 wait=172 +23:19:09 step 72580 loss=3.1295 lr=1.11e-04 tok=9.513B dt=25938 wait=172 +23:20:24 step 72600 loss=3.1462 lr=1.11e-04 tok=9.516B dt=26013 wait=173 +23:20:36 pushed ckpt_w22.pt,log_w22.txt +23:21:51 waited 76s for global r=139 +23:22:00 global r=139 load 4.281 -> 4.260 +23:23:26 step 72620 loss=3.1709 lr=1.11e-04 tok=9.518B dt=26195 wait=173 +23:24:41 step 72640 loss=3.2020 lr=1.11e-04 tok=9.521B dt=26270 wait=174 +23:25:56 step 72660 loss=3.2130 lr=1.11e-04 tok=9.524B dt=26345 wait=174 +23:27:10 step 72680 loss=3.1872 lr=1.11e-04 tok=9.526B dt=26420 wait=174 +23:28:25 step 72700 loss=3.1639 lr=1.11e-04 tok=9.529B dt=26495 wait=175 +23:29:40 step 72720 loss=3.1712 lr=1.11e-04 tok=9.532B dt=26570 wait=175 +23:30:56 step 72740 loss=3.1886 lr=1.11e-04 tok=9.534B dt=26645 wait=176 +23:31:48 pushed ckpt_w22.pt,log_w22.txt +23:35:33 step 72760 loss=3.1670 lr=1.11e-04 tok=9.537B dt=26922 wait=176 +23:36:48 step 72780 loss=3.2152 lr=1.11e-04 tok=9.539B dt=26997 wait=176 +23:38:03 step 72800 loss=3.1560 lr=1.11e-04 tok=9.542B dt=27072 wait=177 +23:39:18 step 72820 loss=3.1290 lr=1.11e-04 tok=9.545B dt=27148 wait=177 +23:40:34 step 72840 loss=3.1050 lr=1.11e-04 tok=9.547B dt=27223 wait=178 +23:41:48 step 72860 loss=3.1226 lr=1.11e-04 tok=9.550B dt=27298 wait=178 +23:43:03 step 72880 loss=3.1613 lr=1.11e-04 tok=9.553B dt=27373 wait=178 +23:44:18 step 72900 loss=3.1619 lr=1.11e-04 tok=9.555B dt=27448 wait=179 +23:44:30 pushed ckpt_w22.pt,log_w22.txt +23:44:30 waited 0s for global r=140 +23:44:41 global r=140 load 4.288 -> 4.257 +23:46:02 step 72920 loss=3.1635 lr=1.11e-04 tok=9.558B dt=27551 wait=179 +23:47:17 step 72940 loss=3.2004 lr=1.11e-04 tok=9.560B dt=27626 wait=180 +23:48:32 step 72960 loss=3.1658 lr=1.11e-04 tok=9.563B dt=27701 wait=180 +23:49:47 step 72980 loss=3.2169 lr=1.11e-04 tok=9.566B dt=27776 wait=180 +23:51:02 step 73000 loss=3.1784 lr=1.11e-04 tok=9.568B dt=27852 wait=181 +23:52:17 step 73020 loss=3.1494 lr=1.11e-04 tok=9.571B dt=27927 wait=181 +23:53:32 step 73040 loss=3.1768 lr=1.11e-04 tok=9.573B dt=28002 wait=182 +23:54:22 pushed ckpt_w22.pt,log_w22.txt +23:55:24 waited 62s for global r=141 +23:55:33 global r=141 load 4.271 -> 4.253 +23:56:17 step 73060 loss=3.1031 lr=1.11e-04 tok=9.576B dt=28166 wait=182 +23:57:32 step 73080 loss=3.0926 lr=1.11e-04 tok=9.579B dt=28241 wait=182 +23:58:47 step 73100 loss=3.1827 lr=1.11e-04 tok=9.581B dt=28316 wait=183 +00:00:02 step 73120 loss=3.1992 lr=1.11e-04 tok=9.584B dt=28391 wait=183 +00:01:17 step 73140 loss=3.1394 lr=1.11e-04 tok=9.587B dt=28466 wait=184 +00:02:32 step 73160 loss=3.2489 lr=1.11e-04 tok=9.589B dt=28541 wait=184 +00:03:47 step 73180 loss=3.1570 lr=1.11e-04 tok=9.592B dt=28617 wait=184 +00:05:02 step 73200 loss=3.1979 lr=1.11e-04 tok=9.594B dt=28692 wait=185 +00:05:15 pushed ckpt_w22.pt,log_w22.txt +00:09:42 step 73220 loss=3.2205 lr=1.11e-04 tok=9.597B dt=28971 wait=185 +00:10:57 step 73240 loss=3.0949 lr=1.11e-04 tok=9.600B dt=29046 wait=186 +00:12:12 step 73260 loss=3.1390 lr=1.11e-04 tok=9.602B dt=29121 wait=186 +00:13:27 step 73280 loss=3.0942 lr=1.10e-04 tok=9.605B dt=29196 wait=186 +00:14:42 step 73300 loss=3.0870 lr=1.10e-04 tok=9.608B dt=29271 wait=187 +00:15:57 step 73320 loss=3.2155 lr=1.10e-04 tok=9.610B dt=29346 wait=187 +00:17:12 step 73340 loss=3.1835 lr=1.10e-04 tok=9.613B dt=29421 wait=188 +00:18:02 pushed ckpt_w22.pt,log_w22.txt +00:18:02 waited 0s for global r=142 +00:18:14 global r=142 load 4.280 -> 4.263 +00:18:57 step 73360 loss=3.1595 lr=1.10e-04 tok=9.615B dt=29526 wait=188 +00:20:12 step 73380 loss=3.1662 lr=1.10e-04 tok=9.618B dt=29602 wait=188 +00:21:27 step 73400 loss=3.2339 lr=1.10e-04 tok=9.621B dt=29677 wait=189 +00:22:42 step 73420 loss=3.2342 lr=1.10e-04 tok=9.623B dt=29752 wait=189 +00:23:58 step 73440 loss=3.1348 lr=1.10e-04 tok=9.626B dt=29827 wait=190 +00:25:13 step 73460 loss=3.2193 lr=1.10e-04 tok=9.629B dt=29902 wait=190 +00:26:28 step 73480 loss=3.0875 lr=1.10e-04 tok=9.631B dt=29977 wait=190 +00:27:43 step 73500 loss=3.1576 lr=1.10e-04 tok=9.634B dt=30052 wait=191 +00:27:56 pushed ckpt_w22.pt,log_w22.txt +00:29:26 waited 91s for global r=143 +00:29:36 global r=143 load 4.270 -> 4.254 +00:31:00 step 73520 loss=3.1264 lr=1.10e-04 tok=9.636B dt=30249 wait=191 +00:32:15 step 73540 loss=3.1761 lr=1.10e-04 tok=9.639B dt=30324 wait=192 +00:33:30 step 73560 loss=3.1735 lr=1.10e-04 tok=9.642B dt=30399 wait=192 +00:34:45 step 73580 loss=3.2160 lr=1.10e-04 tok=9.644B dt=30474 wait=192 +00:36:00 step 73600 loss=3.2139 lr=1.10e-04 tok=9.647B dt=30550 wait=193 +00:37:16 step 73620 loss=3.1654 lr=1.10e-04 tok=9.650B dt=30625 wait=193 +00:38:31 step 73640 loss=3.1392 lr=1.10e-04 tok=9.652B dt=30700 wait=194 +00:39:22 pushed ckpt_w22.pt,log_w22.txt +00:43:08 step 73660 loss=3.1394 lr=1.10e-04 tok=9.655B dt=30977 wait=194 +00:44:23 step 73680 loss=3.1759 lr=1.10e-04 tok=9.657B dt=31052 wait=194 +00:45:38 step 73700 loss=3.0946 lr=1.10e-04 tok=9.660B dt=31127 wait=195 +00:46:53 step 73720 loss=3.0950 lr=1.10e-04 tok=9.663B dt=31203 wait=195 +00:48:08 step 73740 loss=3.1347 lr=1.10e-04 tok=9.665B dt=31278 wait=196 +00:49:23 step 73760 loss=3.1901 lr=1.10e-04 tok=9.668B dt=31353 wait=196 +00:50:38 step 73780 loss=3.2421 lr=1.10e-04 tok=9.670B dt=31428 wait=196 +00:51:53 step 73800 loss=3.1398 lr=1.10e-04 tok=9.673B dt=31503 wait=197 +00:52:05 pushed ckpt_w22.pt,log_w22.txt +00:52:05 waited 0s for global r=144 +00:52:14 global r=144 load 4.286 -> 4.261 +00:53:35 step 73820 loss=3.1487 lr=1.10e-04 tok=9.676B dt=31604 wait=197 +00:54:50 step 73840 loss=3.1618 lr=1.10e-04 tok=9.678B dt=31680 wait=198 +00:56:05 step 73860 loss=3.1593 lr=1.10e-04 tok=9.681B dt=31755 wait=198 +00:57:20 step 73880 loss=3.2180 lr=1.10e-04 tok=9.684B dt=31830 wait=198 +00:58:36 step 73900 loss=3.1872 lr=1.10e-04 tok=9.686B dt=31905 wait=199 +00:59:50 step 73920 loss=3.1318 lr=1.10e-04 tok=9.689B dt=31980 wait=199 +01:01:06 step 73940 loss=3.1838 lr=1.10e-04 tok=9.691B dt=32055 wait=200 diff --git a/log_w23.txt b/log_w23.txt new file mode 100644 index 0000000000000000000000000000000000000000..4dbdaab03bebbdf95134fdf01ca305f3b5f83f19 --- /dev/null +++ b/log_w23.txt @@ -0,0 +1,600 @@ +launch +11:11:35 name=w23 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +11:11:42 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +11:11:51 params 1.08B +11:11:52 resumed step=63600 seen=8.34B round=108 +11:11:52 stream salt=63600 +11:13:37 start: accum=4, 131072 tokens per optimiser step +11:15:01 step 63601 loss=3.5404 lr=2.39e-04 tok=8.336B dt=83 wait=49 +11:15:05 step 63602 loss=3.4030 lr=2.39e-04 tok=8.336B dt=88 wait=49 +11:16:29 step 63620 loss=3.5524 lr=2.39e-04 tok=8.339B dt=171 wait=50 +11:17:44 step 63640 loss=3.4736 lr=2.39e-04 tok=8.341B dt=246 wait=50 +11:18:59 step 63660 loss=3.5239 lr=2.39e-04 tok=8.344B dt=322 wait=51 +11:20:14 step 63680 loss=3.4082 lr=2.39e-04 tok=8.347B dt=397 wait=51 +11:21:29 step 63700 loss=3.4294 lr=2.39e-04 tok=8.349B dt=472 wait=51 +11:22:44 step 63720 loss=3.3121 lr=2.39e-04 tok=8.352B dt=547 wait=52 +11:23:59 step 63740 loss=3.3536 lr=2.39e-04 tok=8.355B dt=622 wait=52 +11:24:52 pushed ckpt_w23.pt,log_w23.txt +11:26:08 waited 76s for global r=109 +11:26:18 global r=109 load 4.449 -> 4.368 +11:27:03 step 63760 loss=3.4185 lr=2.39e-04 tok=8.357B dt=806 wait=53 +11:28:18 step 63780 loss=3.3076 lr=2.39e-04 tok=8.360B dt=881 wait=53 +11:29:33 step 63800 loss=3.4097 lr=2.39e-04 tok=8.362B dt=956 wait=53 +11:30:48 step 63820 loss=3.4480 lr=2.39e-04 tok=8.365B dt=1031 wait=54 +11:32:04 step 63840 loss=3.3524 lr=2.39e-04 tok=8.368B dt=1106 wait=54 +11:33:19 step 63860 loss=3.4596 lr=2.38e-04 tok=8.370B dt=1182 wait=55 +11:34:34 step 63880 loss=3.3557 lr=2.38e-04 tok=8.373B dt=1257 wait=55 +11:35:49 step 63900 loss=3.3865 lr=2.38e-04 tok=8.376B dt=1332 wait=55 +11:36:01 pushed ckpt_w23.pt,log_w23.txt +11:40:23 step 63920 loss=3.3964 lr=2.38e-04 tok=8.378B dt=1605 wait=56 +11:41:38 step 63940 loss=3.2966 lr=2.38e-04 tok=8.381B dt=1680 wait=56 +11:42:53 step 63960 loss=3.3255 lr=2.38e-04 tok=8.383B dt=1755 wait=57 +11:44:08 step 63980 loss=3.3487 lr=2.38e-04 tok=8.386B dt=1830 wait=57 +11:45:23 step 64000 loss=3.2754 lr=2.38e-04 tok=8.389B dt=1905 wait=57 +11:46:37 step 64020 loss=3.3909 lr=2.38e-04 tok=8.391B dt=1980 wait=58 +11:47:53 step 64040 loss=3.3627 lr=2.38e-04 tok=8.394B dt=2055 wait=58 +11:48:44 pushed ckpt_w23.pt,log_w23.txt +11:48:44 waited 0s for global r=110 +11:48:49 global r=110 load 4.436 -> 4.349 +11:49:33 step 64060 loss=3.3463 lr=2.38e-04 tok=8.396B dt=2156 wait=59 +11:50:48 step 64080 loss=3.3131 lr=2.38e-04 tok=8.399B dt=2231 wait=59 +11:52:03 step 64100 loss=3.3813 lr=2.38e-04 tok=8.402B dt=2306 wait=59 +11:53:19 step 64120 loss=3.4056 lr=2.38e-04 tok=8.404B dt=2381 wait=60 +11:54:34 step 64140 loss=3.3404 lr=2.38e-04 tok=8.407B dt=2456 wait=60 +11:55:49 step 64160 loss=3.4298 lr=2.38e-04 tok=8.410B dt=2532 wait=61 +11:57:04 step 64180 loss=3.2818 lr=2.38e-04 tok=8.412B dt=2607 wait=61 +11:58:19 step 64200 loss=3.3566 lr=2.38e-04 tok=8.415B dt=2682 wait=61 +11:58:33 pushed ckpt_w23.pt,log_w23.txt +12:00:33 waited 121s for global r=111 +12:00:40 global r=111 load 4.428 -> 4.339 +12:02:01 step 64220 loss=3.2524 lr=2.38e-04 tok=8.417B dt=2904 wait=62 +12:03:16 step 64240 loss=3.3241 lr=2.38e-04 tok=8.420B dt=2979 wait=62 +12:04:31 step 64260 loss=3.3396 lr=2.38e-04 tok=8.423B dt=3054 wait=63 +12:05:47 step 64280 loss=3.4035 lr=2.38e-04 tok=8.425B dt=3129 wait=63 +12:07:02 step 64300 loss=3.4018 lr=2.38e-04 tok=8.428B dt=3205 wait=63 +12:08:17 step 64320 loss=3.3299 lr=2.38e-04 tok=8.431B dt=3280 wait=64 +12:09:32 step 64340 loss=3.3855 lr=2.38e-04 tok=8.433B dt=3355 wait=64 +12:10:24 pushed ckpt_w23.pt,log_w23.txt +12:14:09 step 64360 loss=3.3513 lr=2.38e-04 tok=8.436B dt=3632 wait=65 +12:15:24 step 64380 loss=3.3881 lr=2.38e-04 tok=8.438B dt=3707 wait=65 +12:16:39 step 64400 loss=3.3456 lr=2.38e-04 tok=8.441B dt=3782 wait=65 +12:17:54 step 64420 loss=3.2754 lr=2.37e-04 tok=8.444B dt=3857 wait=66 +12:19:09 step 64440 loss=3.3124 lr=2.37e-04 tok=8.446B dt=3932 wait=66 +12:20:24 step 64460 loss=3.3187 lr=2.37e-04 tok=8.449B dt=4007 wait=67 +12:21:39 step 64480 loss=3.3993 lr=2.37e-04 tok=8.452B dt=4082 wait=67 +12:22:54 step 64500 loss=3.3392 lr=2.37e-04 tok=8.454B dt=4157 wait=67 +12:23:06 pushed ckpt_w23.pt,log_w23.txt +12:23:06 waited 0s for global r=112 +12:23:16 global r=112 load 4.430 -> 4.333 +12:24:38 step 64520 loss=3.2608 lr=2.37e-04 tok=8.457B dt=4261 wait=68 +12:25:53 step 64540 loss=3.3535 lr=2.37e-04 tok=8.459B dt=4336 wait=68 +12:27:09 step 64560 loss=3.3182 lr=2.37e-04 tok=8.462B dt=4411 wait=69 +12:28:24 step 64580 loss=3.3674 lr=2.37e-04 tok=8.465B dt=4486 wait=69 +12:29:39 step 64600 loss=3.3701 lr=2.37e-04 tok=8.467B dt=4562 wait=69 +12:30:54 step 64620 loss=3.2902 lr=2.37e-04 tok=8.470B dt=4636 wait=70 +12:32:09 step 64640 loss=3.3431 lr=2.37e-04 tok=8.472B dt=4711 wait=70 +12:32:58 pushed ckpt_w23.pt,log_w23.txt +12:33:43 waited 45s for global r=113 +12:33:50 global r=113 load 4.408 -> 4.328 +12:34:37 step 64660 loss=3.2562 lr=2.37e-04 tok=8.475B dt=4860 wait=71 +12:35:52 step 64680 loss=3.2848 lr=2.37e-04 tok=8.478B dt=4935 wait=71 +12:37:07 step 64700 loss=3.3797 lr=2.37e-04 tok=8.480B dt=5010 wait=71 +12:38:23 step 64720 loss=3.4212 lr=2.37e-04 tok=8.483B dt=5086 wait=72 +12:39:38 step 64740 loss=3.3472 lr=2.37e-04 tok=8.486B dt=5161 wait=72 +12:40:53 step 64760 loss=3.4191 lr=2.37e-04 tok=8.488B dt=5236 wait=73 +12:42:08 step 64780 loss=3.2852 lr=2.37e-04 tok=8.491B dt=5311 wait=73 +12:43:23 step 64800 loss=3.3313 lr=2.37e-04 tok=8.493B dt=5386 wait=73 +12:43:39 pushed ckpt_w23.pt,log_w23.txt +12:48:01 step 64820 loss=3.3706 lr=2.37e-04 tok=8.496B dt=5664 wait=74 +12:49:16 step 64840 loss=3.2784 lr=2.37e-04 tok=8.499B dt=5739 wait=74 +12:50:31 step 64860 loss=3.3157 lr=2.37e-04 tok=8.501B dt=5814 wait=75 +12:51:46 step 64880 loss=3.3042 lr=2.37e-04 tok=8.504B dt=5889 wait=75 +12:53:01 step 64900 loss=3.3231 lr=2.37e-04 tok=8.507B dt=5964 wait=75 +12:54:16 step 64920 loss=3.3781 lr=2.37e-04 tok=8.509B dt=6039 wait=76 +12:55:31 step 64940 loss=3.3364 lr=2.37e-04 tok=8.512B dt=6114 wait=76 +12:56:23 pushed ckpt_w23.pt,log_w23.txt +12:56:24 waited 0s for global r=114 +12:56:33 global r=114 load 4.424 -> 4.341 +12:57:16 step 64960 loss=3.2737 lr=2.37e-04 tok=8.514B dt=6219 wait=77 +12:58:31 step 64980 loss=3.2723 lr=2.36e-04 tok=8.517B dt=6294 wait=77 +12:59:47 step 65000 loss=3.3366 lr=2.36e-04 tok=8.520B dt=6369 wait=77 +13:01:02 step 65020 loss=3.3941 lr=2.36e-04 tok=8.522B dt=6445 wait=78 +13:02:17 step 65040 loss=3.2976 lr=2.36e-04 tok=8.525B dt=6520 wait=78 +13:03:32 step 65060 loss=3.4026 lr=2.36e-04 tok=8.528B dt=6595 wait=78 +13:04:48 step 65080 loss=3.2678 lr=2.36e-04 tok=8.530B dt=6670 wait=79 +13:06:03 step 65100 loss=3.3803 lr=2.36e-04 tok=8.533B dt=6746 wait=79 +13:06:16 pushed ckpt_w23.pt,log_w23.txt +13:07:17 waited 61s for global r=115 +13:07:25 global r=115 load 4.398 -> 4.322 +13:08:45 step 65120 loss=3.2630 lr=2.36e-04 tok=8.535B dt=6908 wait=80 +13:10:00 step 65140 loss=3.3069 lr=2.36e-04 tok=8.538B dt=6983 wait=80 +13:11:16 step 65160 loss=3.2930 lr=2.36e-04 tok=8.541B dt=7058 wait=80 +13:12:31 step 65180 loss=3.3351 lr=2.36e-04 tok=8.543B dt=7133 wait=81 +13:13:46 step 65200 loss=3.3312 lr=2.36e-04 tok=8.546B dt=7209 wait=81 +13:15:01 step 65220 loss=3.3223 lr=2.36e-04 tok=8.549B dt=7284 wait=82 +13:16:16 step 65240 loss=3.3543 lr=2.36e-04 tok=8.551B dt=7359 wait=82 +13:17:06 pushed ckpt_w23.pt,log_w23.txt +13:20:52 step 65260 loss=3.3349 lr=2.36e-04 tok=8.554B dt=7635 wait=83 +13:22:07 step 65280 loss=3.3787 lr=2.36e-04 tok=8.556B dt=7710 wait=83 +13:23:23 step 65300 loss=3.3231 lr=2.36e-04 tok=8.559B dt=7785 wait=83 +13:24:37 step 65320 loss=3.2457 lr=2.36e-04 tok=8.562B dt=7860 wait=84 +13:25:52 step 65340 loss=3.3090 lr=2.36e-04 tok=8.564B dt=7935 wait=84 +13:27:07 step 65360 loss=3.3130 lr=2.36e-04 tok=8.567B dt=8010 wait=84 +13:28:22 step 65380 loss=3.3124 lr=2.36e-04 tok=8.569B dt=8085 wait=85 +13:29:38 step 65400 loss=3.3191 lr=2.36e-04 tok=8.572B dt=8160 wait=85 +13:29:50 pushed ckpt_w23.pt,log_w23.txt +13:29:50 waited 0s for global r=116 +13:29:59 global r=116 load 4.403 -> 4.325 +13:31:24 step 65420 loss=3.2774 lr=2.36e-04 tok=8.575B dt=8267 wait=86 +13:32:39 step 65440 loss=3.3203 lr=2.36e-04 tok=8.577B dt=8342 wait=86 +13:33:54 step 65460 loss=3.2698 lr=2.36e-04 tok=8.580B dt=8417 wait=86 +13:35:09 step 65480 loss=3.3283 lr=2.36e-04 tok=8.583B dt=8492 wait=87 +13:36:24 step 65500 loss=3.3279 lr=2.36e-04 tok=8.585B dt=8567 wait=87 +13:37:39 step 65520 loss=3.3112 lr=2.36e-04 tok=8.588B dt=8642 wait=88 +13:38:54 step 65540 loss=3.3558 lr=2.35e-04 tok=8.590B dt=8717 wait=88 +13:39:46 pushed ckpt_w23.pt,log_w23.txt +13:39:46 waited 0s for global r=117 +13:39:54 global r=117 load 4.389 -> 4.330 +13:40:40 step 65560 loss=3.2852 lr=2.35e-04 tok=8.593B dt=8823 wait=89 +13:41:55 step 65580 loss=3.2876 lr=2.35e-04 tok=8.596B dt=8898 wait=89 +13:43:10 step 65600 loss=3.3686 lr=2.35e-04 tok=8.598B dt=8973 wait=89 +13:44:25 step 65620 loss=3.3533 lr=2.35e-04 tok=8.601B dt=9048 wait=90 +13:45:40 step 65640 loss=3.2901 lr=2.35e-04 tok=8.604B dt=9123 wait=90 +13:46:56 step 65660 loss=3.3818 lr=2.35e-04 tok=8.606B dt=9198 wait=91 +13:48:11 step 65680 loss=3.2580 lr=2.35e-04 tok=8.609B dt=9273 wait=91 +13:49:26 step 65700 loss=3.3285 lr=2.35e-04 tok=8.611B dt=9348 wait=91 +13:49:38 pushed ckpt_w23.pt,log_w23.txt +13:54:01 step 65720 loss=3.3242 lr=2.35e-04 tok=8.614B dt=9624 wait=92 +13:55:16 step 65740 loss=3.2532 lr=2.35e-04 tok=8.617B dt=9699 wait=92 +13:56:31 step 65760 loss=3.3161 lr=2.35e-04 tok=8.619B dt=9774 wait=92 +13:57:46 step 65780 loss=3.3326 lr=2.35e-04 tok=8.622B dt=9849 wait=93 +13:59:01 step 65800 loss=3.3094 lr=2.35e-04 tok=8.625B dt=9924 wait=93 +14:00:17 step 65820 loss=3.3929 lr=2.35e-04 tok=8.627B dt=9999 wait=94 +14:01:32 step 65840 loss=3.2916 lr=2.35e-04 tok=8.630B dt=10075 wait=94 +14:02:21 pushed ckpt_w23.pt,log_w23.txt +14:02:21 waited 0s for global r=118 +14:02:32 global r=118 load 4.399 -> 4.343 +14:03:18 step 65860 loss=3.2385 lr=2.35e-04 tok=8.632B dt=10181 wait=94 +14:04:33 step 65880 loss=3.2181 lr=2.35e-04 tok=8.635B dt=10256 wait=95 +14:05:48 step 65900 loss=3.3253 lr=2.35e-04 tok=8.638B dt=10331 wait=95 +14:07:03 step 65920 loss=3.3529 lr=2.35e-04 tok=8.640B dt=10406 wait=96 +14:08:19 step 65940 loss=3.3223 lr=2.35e-04 tok=8.643B dt=10481 wait=96 +14:09:34 step 65960 loss=3.4705 lr=2.35e-04 tok=8.646B dt=10557 wait=96 +14:10:49 step 65980 loss=3.2964 lr=2.35e-04 tok=8.648B dt=10632 wait=97 +14:12:05 step 66000 loss=3.3298 lr=2.35e-04 tok=8.651B dt=10708 wait=97 +14:12:19 pushed ckpt_w23.pt,log_w23.txt +14:12:19 waited 0s for global r=119 +14:12:32 global r=119 load 4.387 -> 4.344 +14:13:52 step 66020 loss=3.2632 lr=2.35e-04 tok=8.653B dt=10814 wait=98 +14:15:07 step 66040 loss=3.2717 lr=2.35e-04 tok=8.656B dt=10890 wait=98 +14:16:22 step 66060 loss=3.2929 lr=2.35e-04 tok=8.659B dt=10965 wait=99 +14:17:37 step 66080 loss=3.3900 lr=2.34e-04 tok=8.661B dt=11040 wait=99 +14:18:53 step 66100 loss=3.3691 lr=2.34e-04 tok=8.664B dt=11115 wait=99 +14:20:08 step 66120 loss=3.3442 lr=2.34e-04 tok=8.666B dt=11191 wait=100 +14:21:23 step 66140 loss=3.3787 lr=2.34e-04 tok=8.669B dt=11266 wait=100 +14:22:13 pushed ckpt_w23.pt,log_w23.txt +14:25:57 step 66160 loss=3.2890 lr=2.34e-04 tok=8.672B dt=11540 wait=101 +14:27:12 step 66180 loss=3.3201 lr=2.34e-04 tok=8.674B dt=11615 wait=101 +14:28:27 step 66200 loss=3.2593 lr=2.34e-04 tok=8.677B dt=11690 wait=101 +14:29:42 step 66220 loss=3.2083 lr=2.34e-04 tok=8.680B dt=11765 wait=102 +14:30:57 step 66240 loss=3.3383 lr=2.34e-04 tok=8.682B dt=11840 wait=102 +14:32:12 step 66260 loss=3.3350 lr=2.34e-04 tok=8.685B dt=11915 wait=103 +14:33:27 step 66280 loss=3.3752 lr=2.34e-04 tok=8.687B dt=11990 wait=103 +14:34:42 step 66300 loss=3.3155 lr=2.34e-04 tok=8.690B dt=12065 wait=103 +14:34:56 pushed ckpt_w23.pt,log_w23.txt +14:34:56 waited 0s for global r=120 +14:35:08 global r=120 load 4.408 -> 4.338 +14:36:33 step 66320 loss=3.2548 lr=2.34e-04 tok=8.693B dt=12175 wait=104 +14:37:48 step 66340 loss=3.2789 lr=2.34e-04 tok=8.695B dt=12250 wait=104 +14:39:03 step 66360 loss=3.2989 lr=2.34e-04 tok=8.698B dt=12326 wait=104 +14:40:18 step 66380 loss=3.3646 lr=2.34e-04 tok=8.701B dt=12401 wait=105 +14:41:33 step 66400 loss=3.3220 lr=2.34e-04 tok=8.703B dt=12476 wait=105 +14:42:48 step 66420 loss=3.2665 lr=2.34e-04 tok=8.706B dt=12551 wait=106 +14:44:04 step 66440 loss=3.3342 lr=2.34e-04 tok=8.708B dt=12626 wait=106 +14:44:54 pushed ckpt_w23.pt,log_w23.txt +14:48:38 step 66460 loss=3.3183 lr=2.34e-04 tok=8.711B dt=12900 wait=106 +14:49:53 step 66480 loss=3.3773 lr=2.34e-04 tok=8.714B dt=12976 wait=107 +14:51:08 step 66500 loss=3.2936 lr=2.34e-04 tok=8.716B dt=13051 wait=107 +14:52:23 step 66520 loss=3.2320 lr=2.34e-04 tok=8.719B dt=13125 wait=108 +14:53:38 step 66540 loss=3.2653 lr=2.34e-04 tok=8.722B dt=13201 wait=108 +14:54:53 step 66560 loss=3.3011 lr=2.34e-04 tok=8.724B dt=13276 wait=108 +14:56:08 step 66580 loss=3.3324 lr=2.34e-04 tok=8.727B dt=13351 wait=109 +14:57:23 step 66600 loss=3.3191 lr=2.34e-04 tok=8.729B dt=13426 wait=109 +14:57:35 pushed ckpt_w23.pt,log_w23.txt +14:57:35 waited 0s for global r=121 +14:57:49 global r=121 load 4.409 -> 4.329 +14:59:11 step 66620 loss=3.2999 lr=2.34e-04 tok=8.732B dt=13534 wait=110 +15:00:26 step 66640 loss=3.3477 lr=2.33e-04 tok=8.735B dt=13609 wait=110 +15:01:41 step 66660 loss=3.3761 lr=2.33e-04 tok=8.737B dt=13684 wait=110 +15:02:57 step 66680 loss=3.3820 lr=2.33e-04 tok=8.740B dt=13759 wait=111 +15:04:12 step 66700 loss=3.3213 lr=2.33e-04 tok=8.743B dt=13835 wait=111 +15:05:27 step 66720 loss=3.2660 lr=2.33e-04 tok=8.745B dt=13910 wait=112 +15:06:42 step 66740 loss=3.3237 lr=2.33e-04 tok=8.748B dt=13985 wait=112 +15:07:32 pushed ckpt_w23.pt,log_w23.txt +15:11:13 step 66760 loss=3.2833 lr=2.33e-04 tok=8.750B dt=14256 wait=112 +15:12:28 step 66780 loss=3.3676 lr=2.33e-04 tok=8.753B dt=14331 wait=113 +15:13:43 step 66800 loss=3.2940 lr=2.33e-04 tok=8.756B dt=14406 wait=113 +15:14:58 step 66820 loss=3.2383 lr=2.33e-04 tok=8.758B dt=14481 wait=114 +15:16:13 step 66840 loss=3.2889 lr=2.33e-04 tok=8.761B dt=14556 wait=114 +15:17:28 step 66860 loss=3.3009 lr=2.33e-04 tok=8.763B dt=14631 wait=114 +15:18:43 step 66880 loss=3.3380 lr=2.33e-04 tok=8.766B dt=14706 wait=115 +15:19:59 step 66900 loss=3.3200 lr=2.33e-04 tok=8.769B dt=14781 wait=115 +15:20:12 pushed ckpt_w23.pt,log_w23.txt +15:20:12 waited 0s for global r=122 +15:20:21 global r=122 load 4.405 -> 4.335 +15:21:41 step 66920 loss=3.2352 lr=2.33e-04 tok=8.771B dt=14884 wait=116 +15:22:56 step 66940 loss=3.3148 lr=2.33e-04 tok=8.774B dt=14959 wait=116 +15:24:11 step 66960 loss=3.3125 lr=2.33e-04 tok=8.777B dt=15034 wait=116 +15:25:27 step 66980 loss=3.3457 lr=2.33e-04 tok=8.779B dt=15109 wait=117 +15:26:42 step 67000 loss=3.3377 lr=2.33e-04 tok=8.782B dt=15184 wait=117 +15:27:57 step 67020 loss=3.2846 lr=2.33e-04 tok=8.784B dt=15260 wait=118 +15:29:12 step 67040 loss=3.3149 lr=2.33e-04 tok=8.787B dt=15334 wait=118 +15:30:01 pushed ckpt_w23.pt,log_w23.txt +15:33:45 step 67060 loss=3.2776 lr=2.33e-04 tok=8.790B dt=15608 wait=118 +15:35:00 step 67080 loss=3.3280 lr=2.33e-04 tok=8.792B dt=15683 wait=119 +15:36:15 step 67100 loss=3.2666 lr=2.33e-04 tok=8.795B dt=15758 wait=119 +15:37:30 step 67120 loss=3.2877 lr=2.33e-04 tok=8.798B dt=15833 wait=120 +15:38:45 step 67140 loss=3.2995 lr=2.33e-04 tok=8.800B dt=15908 wait=120 +15:40:00 step 67160 loss=3.3179 lr=2.33e-04 tok=8.803B dt=15983 wait=120 +15:41:15 step 67180 loss=3.3134 lr=2.32e-04 tok=8.805B dt=16058 wait=121 +15:42:30 step 67200 loss=3.3230 lr=2.32e-04 tok=8.808B dt=16133 wait=121 +15:42:42 pushed ckpt_w23.pt,log_w23.txt +15:42:42 waited 0s for global r=123 +15:42:50 global r=123 load 4.435 -> 4.333 +15:44:16 step 67220 loss=3.2395 lr=2.32e-04 tok=8.811B dt=16238 wait=122 +15:45:31 step 67240 loss=3.2786 lr=2.32e-04 tok=8.813B dt=16313 wait=122 +15:46:46 step 67260 loss=3.2810 lr=2.32e-04 tok=8.816B dt=16389 wait=122 +15:48:01 step 67280 loss=3.3642 lr=2.32e-04 tok=8.819B dt=16464 wait=123 +15:49:16 step 67300 loss=3.3453 lr=2.32e-04 tok=8.821B dt=16539 wait=123 +15:50:31 step 67320 loss=3.3270 lr=2.32e-04 tok=8.824B dt=16614 wait=124 +15:51:46 step 67340 loss=3.3943 lr=2.32e-04 tok=8.826B dt=16689 wait=124 +15:52:36 pushed ckpt_w23.pt,log_w23.txt +15:56:21 step 67360 loss=3.3120 lr=2.32e-04 tok=8.829B dt=16964 wait=124 +15:57:36 step 67380 loss=3.3257 lr=2.32e-04 tok=8.832B dt=17039 wait=125 +15:58:51 step 67400 loss=3.2523 lr=2.32e-04 tok=8.834B dt=17114 wait=125 +16:00:06 step 67420 loss=3.2339 lr=2.32e-04 tok=8.837B dt=17189 wait=126 +16:01:21 step 67440 loss=3.3131 lr=2.32e-04 tok=8.839B dt=17264 wait=126 +16:02:36 step 67460 loss=3.3091 lr=2.32e-04 tok=8.842B dt=17339 wait=126 +16:03:51 step 67480 loss=3.3374 lr=2.32e-04 tok=8.845B dt=17414 wait=127 +launch +16:04:22 name=w23 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:04:27 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:04:36 params 1.08B +16:04:40 resumed step=67350 seen=8.83B round=123 +16:04:40 stream salt=67350 +16:06:55 start: accum=4, 131072 tokens per optimiser step +16:08:25 step 67351 loss=3.3144 lr=1.16e-04 tok=8.828B dt=90 wait=74 +16:08:30 step 67352 loss=3.1991 lr=1.16e-04 tok=8.828B dt=95 wait=74 +16:09:08 step 67360 loss=3.3351 lr=1.16e-04 tok=8.829B dt=133 wait=74 +16:10:41 step 67380 loss=3.3720 lr=1.16e-04 tok=8.832B dt=226 wait=74 +16:11:57 step 67400 loss=3.3314 lr=1.16e-04 tok=8.834B dt=302 wait=75 +16:13:12 step 67420 loss=3.3352 lr=1.16e-04 tok=8.837B dt=377 wait=75 +16:14:27 step 67440 loss=3.3035 lr=1.16e-04 tok=8.839B dt=452 wait=76 +16:15:42 step 67460 loss=3.2148 lr=1.16e-04 tok=8.842B dt=527 wait=76 +16:16:57 step 67480 loss=3.2726 lr=1.16e-04 tok=8.845B dt=602 wait=76 +16:18:12 step 67500 loss=3.3491 lr=1.16e-04 tok=8.847B dt=677 wait=77 +16:18:26 pushed ckpt_w23.pt,log_w23.txt +16:18:26 waited 0s for global r=124 +16:18:41 global r=124 load 4.343 -> 4.332 +16:20:02 step 67520 loss=3.2230 lr=1.16e-04 tok=8.850B dt=787 wait=77 +16:21:17 step 67540 loss=3.2302 lr=1.16e-04 tok=8.853B dt=862 wait=78 +16:22:32 step 67560 loss=3.2285 lr=1.16e-04 tok=8.855B dt=937 wait=78 +16:23:47 step 67580 loss=3.3105 lr=1.16e-04 tok=8.858B dt=1012 wait=78 +16:25:02 step 67600 loss=3.2954 lr=1.16e-04 tok=8.860B dt=1088 wait=79 +16:26:18 step 67620 loss=3.2446 lr=1.16e-04 tok=8.863B dt=1163 wait=79 +16:27:33 step 67640 loss=3.2504 lr=1.16e-04 tok=8.866B dt=1238 wait=80 +16:28:22 pushed ckpt_w23.pt,log_w23.txt +16:32:04 step 67660 loss=3.2000 lr=1.16e-04 tok=8.868B dt=1509 wait=80 +16:33:19 step 67680 loss=3.2286 lr=1.16e-04 tok=8.871B dt=1584 wait=80 +16:34:34 step 67700 loss=3.1609 lr=1.16e-04 tok=8.874B dt=1659 wait=81 +16:35:49 step 67720 loss=3.1547 lr=1.16e-04 tok=8.876B dt=1734 wait=81 +16:37:04 step 67740 loss=3.2039 lr=1.16e-04 tok=8.879B dt=1809 wait=81 +16:38:19 step 67760 loss=3.2303 lr=1.16e-04 tok=8.881B dt=1884 wait=82 +16:39:34 step 67780 loss=3.2373 lr=1.16e-04 tok=8.884B dt=1959 wait=82 +16:40:49 step 67800 loss=3.2459 lr=1.16e-04 tok=8.887B dt=2035 wait=83 +16:41:02 pushed ckpt_w23.pt,log_w23.txt +16:41:02 waited 0s for global r=125 +16:41:10 global r=125 load 4.318 -> 4.296 +16:42:30 step 67820 loss=3.2275 lr=1.16e-04 tok=8.889B dt=2135 wait=83 +16:43:45 step 67840 loss=3.2249 lr=1.16e-04 tok=8.892B dt=2210 wait=84 +16:45:00 step 67860 loss=3.2154 lr=1.16e-04 tok=8.895B dt=2285 wait=84 +16:46:15 step 67880 loss=3.2614 lr=1.16e-04 tok=8.897B dt=2360 wait=84 +16:47:30 step 67900 loss=3.2654 lr=1.16e-04 tok=8.900B dt=2435 wait=85 +16:48:45 step 67920 loss=3.2023 lr=1.16e-04 tok=8.902B dt=2510 wait=85 +16:50:01 step 67940 loss=3.2125 lr=1.16e-04 tok=8.905B dt=2586 wait=85 +16:50:50 pushed ckpt_w23.pt,log_w23.txt +16:50:50 waited 0s for global r=126 +16:50:58 global r=126 load 4.309 -> 4.295 +16:51:40 step 67960 loss=3.1981 lr=1.16e-04 tok=8.908B dt=2685 wait=86 +16:52:55 step 67980 loss=3.1902 lr=1.16e-04 tok=8.910B dt=2760 wait=86 +16:54:10 step 68000 loss=3.2457 lr=1.15e-04 tok=8.913B dt=2835 wait=87 +16:55:25 step 68020 loss=3.2811 lr=1.15e-04 tok=8.916B dt=2911 wait=87 +16:56:41 step 68040 loss=3.2278 lr=1.15e-04 tok=8.918B dt=2986 wait=87 +16:57:56 step 68060 loss=3.3071 lr=1.15e-04 tok=8.921B dt=3061 wait=88 +16:59:11 step 68080 loss=3.2018 lr=1.15e-04 tok=8.923B dt=3136 wait=88 +17:00:26 step 68100 loss=3.2479 lr=1.15e-04 tok=8.926B dt=3211 wait=89 +17:00:40 pushed ckpt_w23.pt,log_w23.txt +17:05:06 step 68120 loss=3.2452 lr=1.15e-04 tok=8.929B dt=3491 wait=89 +17:06:21 step 68140 loss=3.1521 lr=1.15e-04 tok=8.931B dt=3566 wait=89 +17:07:36 step 68160 loss=3.1438 lr=1.15e-04 tok=8.934B dt=3641 wait=90 +17:08:51 step 68180 loss=3.1848 lr=1.15e-04 tok=8.936B dt=3716 wait=90 +17:10:06 step 68200 loss=3.1641 lr=1.15e-04 tok=8.939B dt=3791 wait=91 +17:11:21 step 68220 loss=3.2348 lr=1.15e-04 tok=8.942B dt=3866 wait=91 +17:12:36 step 68240 loss=3.1967 lr=1.15e-04 tok=8.944B dt=3941 wait=91 +17:13:26 pushed ckpt_w23.pt,log_w23.txt +17:13:26 waited 0s for global r=127 +17:13:38 global r=127 load 4.310 -> 4.287 +17:14:22 step 68260 loss=3.2237 lr=1.15e-04 tok=8.947B dt=4047 wait=92 +17:15:37 step 68280 loss=3.1951 lr=1.15e-04 tok=8.950B dt=4122 wait=92 +17:16:52 step 68300 loss=3.2865 lr=1.15e-04 tok=8.952B dt=4197 wait=92 +17:18:07 step 68320 loss=3.2645 lr=1.15e-04 tok=8.955B dt=4272 wait=93 +17:19:22 step 68340 loss=3.1666 lr=1.15e-04 tok=8.957B dt=4347 wait=93 +17:20:37 step 68360 loss=3.2560 lr=1.15e-04 tok=8.960B dt=4422 wait=94 +17:21:52 step 68380 loss=3.1406 lr=1.15e-04 tok=8.963B dt=4497 wait=94 +17:23:07 step 68400 loss=3.1892 lr=1.15e-04 tok=8.965B dt=4572 wait=94 +17:23:20 pushed ckpt_w23.pt,log_w23.txt +17:23:20 waited 0s for global r=128 +17:23:32 global r=128 load 4.308 -> 4.293 +17:24:52 step 68420 loss=3.1956 lr=1.15e-04 tok=8.968B dt=4677 wait=95 +17:26:07 step 68440 loss=3.2034 lr=1.15e-04 tok=8.971B dt=4752 wait=95 +17:27:22 step 68460 loss=3.2123 lr=1.15e-04 tok=8.973B dt=4827 wait=96 +17:28:37 step 68480 loss=3.2481 lr=1.15e-04 tok=8.976B dt=4902 wait=96 +17:29:52 step 68500 loss=3.2679 lr=1.15e-04 tok=8.978B dt=4977 wait=96 +17:31:07 step 68520 loss=3.1938 lr=1.15e-04 tok=8.981B dt=5052 wait=97 +17:32:23 step 68540 loss=3.2468 lr=1.15e-04 tok=8.984B dt=5128 wait=97 +17:33:12 pushed ckpt_w23.pt,log_w23.txt +17:36:56 step 68560 loss=3.2248 lr=1.15e-04 tok=8.986B dt=5401 wait=98 +17:38:10 step 68580 loss=3.2681 lr=1.15e-04 tok=8.989B dt=5476 wait=98 +17:39:26 step 68600 loss=3.1542 lr=1.15e-04 tok=8.992B dt=5551 wait=98 +17:40:40 step 68620 loss=3.1092 lr=1.15e-04 tok=8.994B dt=5626 wait=99 +17:41:56 step 68640 loss=3.1940 lr=1.15e-04 tok=8.997B dt=5701 wait=99 +17:43:11 step 68660 loss=3.2044 lr=1.15e-04 tok=8.999B dt=5776 wait=100 +17:44:26 step 68680 loss=3.2274 lr=1.15e-04 tok=9.002B dt=5851 wait=100 +17:45:41 step 68700 loss=3.1623 lr=1.15e-04 tok=9.005B dt=5926 wait=100 +17:45:52 pushed ckpt_w23.pt,log_w23.txt +17:45:52 waited 0s for global r=129 +17:46:01 global r=129 load 4.313 -> 4.283 +17:47:25 step 68720 loss=3.2029 lr=1.15e-04 tok=9.007B dt=6030 wait=101 +17:48:40 step 68740 loss=3.2256 lr=1.15e-04 tok=9.010B dt=6105 wait=101 +17:49:55 step 68760 loss=3.2332 lr=1.15e-04 tok=9.013B dt=6180 wait=102 +17:51:10 step 68780 loss=3.2764 lr=1.15e-04 tok=9.015B dt=6256 wait=102 +17:52:26 step 68800 loss=3.2195 lr=1.15e-04 tok=9.018B dt=6331 wait=102 +17:53:41 step 68820 loss=3.1529 lr=1.15e-04 tok=9.020B dt=6406 wait=103 +17:54:56 step 68840 loss=3.2055 lr=1.15e-04 tok=9.023B dt=6481 wait=103 +17:55:48 pushed ckpt_w23.pt,log_w23.txt +17:55:48 waited 0s for global r=130 +17:55:59 global r=130 load 4.298 -> 4.288 +17:56:40 step 68860 loss=3.1580 lr=1.15e-04 tok=9.026B dt=6585 wait=104 +17:57:55 step 68880 loss=3.1660 lr=1.15e-04 tok=9.028B dt=6660 wait=104 +17:59:10 step 68900 loss=3.2472 lr=1.15e-04 tok=9.031B dt=6735 wait=104 +18:00:25 step 68920 loss=3.2961 lr=1.15e-04 tok=9.033B dt=6811 wait=105 +18:01:41 step 68940 loss=3.1994 lr=1.15e-04 tok=9.036B dt=6886 wait=105 +18:02:56 step 68960 loss=3.2599 lr=1.15e-04 tok=9.039B dt=6961 wait=105 +18:04:11 step 68980 loss=3.0986 lr=1.15e-04 tok=9.041B dt=7036 wait=106 +18:05:26 step 69000 loss=3.1532 lr=1.15e-04 tok=9.044B dt=7111 wait=106 +18:05:38 pushed ckpt_w23.pt,log_w23.txt +18:10:00 step 69020 loss=3.2133 lr=1.15e-04 tok=9.047B dt=7385 wait=107 +18:11:15 step 69040 loss=3.1415 lr=1.15e-04 tok=9.049B dt=7460 wait=107 +18:12:30 step 69060 loss=3.1599 lr=1.14e-04 tok=9.052B dt=7535 wait=107 +18:13:45 step 69080 loss=3.2106 lr=1.14e-04 tok=9.054B dt=7610 wait=108 +18:15:00 step 69100 loss=3.1474 lr=1.14e-04 tok=9.057B dt=7685 wait=108 +18:16:15 step 69120 loss=3.2503 lr=1.14e-04 tok=9.060B dt=7761 wait=109 +18:17:31 step 69140 loss=3.1764 lr=1.14e-04 tok=9.062B dt=7836 wait=109 +18:18:20 pushed ckpt_w23.pt,log_w23.txt +18:18:21 waited 0s for global r=131 +18:18:33 global r=131 load 4.299 -> 4.290 +18:19:15 step 69160 loss=3.1362 lr=1.14e-04 tok=9.065B dt=7940 wait=109 +18:20:29 step 69180 loss=3.1452 lr=1.14e-04 tok=9.068B dt=8015 wait=110 +18:21:44 step 69200 loss=3.2102 lr=1.14e-04 tok=9.070B dt=8090 wait=110 +18:23:00 step 69220 loss=3.2433 lr=1.14e-04 tok=9.073B dt=8165 wait=111 +18:24:15 step 69240 loss=3.2043 lr=1.14e-04 tok=9.075B dt=8240 wait=111 +18:25:30 step 69260 loss=3.2862 lr=1.14e-04 tok=9.078B dt=8315 wait=111 +18:26:45 step 69280 loss=3.1461 lr=1.14e-04 tok=9.081B dt=8390 wait=112 +18:28:00 step 69300 loss=3.1732 lr=1.14e-04 tok=9.083B dt=8466 wait=112 +18:28:14 pushed ckpt_w23.pt,log_w23.txt +18:28:14 waited 0s for global r=132 +18:28:27 global r=132 load 4.311 -> 4.285 +18:29:46 step 69320 loss=3.1780 lr=1.14e-04 tok=9.086B dt=8571 wait=113 +18:31:01 step 69340 loss=3.1735 lr=1.14e-04 tok=9.089B dt=8646 wait=113 +18:32:16 step 69360 loss=3.1628 lr=1.14e-04 tok=9.091B dt=8721 wait=113 +18:33:31 step 69380 loss=3.2329 lr=1.14e-04 tok=9.094B dt=8796 wait=114 +18:34:46 step 69400 loss=3.2206 lr=1.14e-04 tok=9.096B dt=8871 wait=114 +18:36:01 step 69420 loss=3.1963 lr=1.14e-04 tok=9.099B dt=8946 wait=115 +18:37:17 step 69440 loss=3.2328 lr=1.14e-04 tok=9.102B dt=9022 wait=115 +18:38:07 pushed ckpt_w23.pt,log_w23.txt +18:41:49 step 69460 loss=3.2292 lr=1.14e-04 tok=9.104B dt=9294 wait=115 +18:43:04 step 69480 loss=3.2312 lr=1.14e-04 tok=9.107B dt=9369 wait=116 +18:44:19 step 69500 loss=3.1569 lr=1.14e-04 tok=9.110B dt=9444 wait=116 +18:45:34 step 69520 loss=3.1228 lr=1.14e-04 tok=9.112B dt=9519 wait=116 +18:46:49 step 69540 loss=3.1685 lr=1.14e-04 tok=9.115B dt=9594 wait=117 +18:48:04 step 69560 loss=3.1698 lr=1.14e-04 tok=9.117B dt=9669 wait=117 +18:49:19 step 69580 loss=3.2077 lr=1.14e-04 tok=9.120B dt=9744 wait=118 +18:50:34 step 69600 loss=3.1707 lr=1.14e-04 tok=9.123B dt=9819 wait=118 +18:50:46 pushed ckpt_w23.pt,log_w23.txt +18:50:46 waited 0s for global r=133 +18:50:55 global r=133 load 4.297 -> 4.282 +18:52:16 step 69620 loss=3.1474 lr=1.14e-04 tok=9.125B dt=9921 wait=118 +18:53:31 step 69640 loss=3.1988 lr=1.14e-04 tok=9.128B dt=9996 wait=119 +18:54:46 step 69660 loss=3.1833 lr=1.14e-04 tok=9.130B dt=10071 wait=119 +18:56:01 step 69680 loss=3.2648 lr=1.14e-04 tok=9.133B dt=10146 wait=120 +18:57:17 step 69700 loss=3.2115 lr=1.14e-04 tok=9.136B dt=10222 wait=120 +18:58:32 step 69720 loss=3.1994 lr=1.14e-04 tok=9.138B dt=10297 wait=120 +18:59:47 step 69740 loss=3.2154 lr=1.14e-04 tok=9.141B dt=10372 wait=121 +19:00:39 pushed ckpt_w23.pt,log_w23.txt +19:04:24 step 69760 loss=3.1793 lr=1.14e-04 tok=9.144B dt=10649 wait=121 +19:05:39 step 69780 loss=3.2374 lr=1.14e-04 tok=9.146B dt=10724 wait=122 +19:06:54 step 69800 loss=3.1754 lr=1.14e-04 tok=9.149B dt=10799 wait=122 +19:08:09 step 69820 loss=3.0956 lr=1.14e-04 tok=9.151B dt=10874 wait=122 +19:09:25 step 69840 loss=3.1527 lr=1.14e-04 tok=9.154B dt=10950 wait=123 +19:10:40 step 69860 loss=3.1879 lr=1.14e-04 tok=9.157B dt=11025 wait=123 +19:11:55 step 69880 loss=3.2200 lr=1.14e-04 tok=9.159B dt=11100 wait=124 +19:13:10 step 69900 loss=3.2251 lr=1.14e-04 tok=9.162B dt=11175 wait=124 +19:13:22 pushed ckpt_w23.pt,log_w23.txt +19:17:43 step 69920 loss=3.1786 lr=1.14e-04 tok=9.165B dt=11448 wait=124 +19:18:58 step 69940 loss=3.1909 lr=1.14e-04 tok=9.167B dt=11523 wait=125 +19:20:13 step 69960 loss=3.2278 lr=1.14e-04 tok=9.170B dt=11598 wait=125 +19:21:28 step 69980 loss=3.2242 lr=1.14e-04 tok=9.172B dt=11673 wait=126 +19:22:43 step 70000 loss=3.1894 lr=1.14e-04 tok=9.175B dt=11748 wait=126 +19:23:58 step 70020 loss=3.2272 lr=1.14e-04 tok=9.178B dt=11823 wait=126 +19:25:13 step 70040 loss=3.1711 lr=1.14e-04 tok=9.180B dt=11898 wait=127 +19:26:03 pushed ckpt_w23.pt,log_w23.txt +19:29:51 step 70060 loss=3.2429 lr=1.14e-04 tok=9.183B dt=12176 wait=127 +19:31:06 step 70080 loss=3.1800 lr=1.14e-04 tok=9.186B dt=12251 wait=128 +19:32:21 step 70100 loss=3.1954 lr=1.14e-04 tok=9.188B dt=12326 wait=128 +19:33:36 step 70120 loss=3.1346 lr=1.14e-04 tok=9.191B dt=12401 wait=128 +19:34:51 step 70140 loss=3.1726 lr=1.13e-04 tok=9.193B dt=12476 wait=129 +19:36:07 step 70160 loss=3.2943 lr=1.13e-04 tok=9.196B dt=12552 wait=129 +19:37:22 step 70180 loss=3.2691 lr=1.13e-04 tok=9.199B dt=12627 wait=130 +19:38:38 step 70200 loss=3.2552 lr=1.13e-04 tok=9.201B dt=12703 wait=130 +19:38:50 pushed ckpt_w23.pt,log_w23.txt +19:43:11 step 70220 loss=3.2640 lr=1.13e-04 tok=9.204B dt=12976 wait=131 +19:44:26 step 70240 loss=3.2269 lr=1.13e-04 tok=9.206B dt=13051 wait=131 +19:45:41 step 70260 loss=3.1499 lr=1.13e-04 tok=9.209B dt=13126 wait=131 +19:46:57 step 70280 loss=3.2145 lr=1.13e-04 tok=9.212B dt=13202 wait=132 +19:48:12 step 70300 loss=3.1415 lr=1.13e-04 tok=9.214B dt=13277 wait=132 +19:49:27 step 70320 loss=3.1453 lr=1.13e-04 tok=9.217B dt=13352 wait=133 +19:50:43 step 70340 loss=3.2124 lr=1.13e-04 tok=9.220B dt=13428 wait=133 +19:51:32 pushed ckpt_w23.pt,log_w23.txt +19:55:18 step 70360 loss=3.1897 lr=1.13e-04 tok=9.222B dt=13703 wait=134 +19:56:33 step 70380 loss=3.1457 lr=1.13e-04 tok=9.225B dt=13778 wait=134 +19:57:49 step 70400 loss=3.1542 lr=1.13e-04 tok=9.227B dt=13854 wait=135 +19:59:04 step 70420 loss=3.2091 lr=1.13e-04 tok=9.230B dt=13929 wait=135 +20:00:20 step 70440 loss=3.2864 lr=1.13e-04 tok=9.233B dt=14005 wait=136 +20:01:35 step 70460 loss=3.1793 lr=1.13e-04 tok=9.235B dt=14080 wait=136 +20:02:50 step 70480 loss=3.2010 lr=1.13e-04 tok=9.238B dt=14155 wait=136 +20:04:06 step 70500 loss=3.3161 lr=1.13e-04 tok=9.241B dt=14231 wait=137 +20:04:21 pushed ckpt_w23.pt,log_w23.txt +20:08:44 step 70520 loss=3.2740 lr=1.13e-04 tok=9.243B dt=14509 wait=137 +20:09:59 step 70540 loss=3.2434 lr=1.13e-04 tok=9.246B dt=14584 wait=138 +20:11:14 step 70560 loss=3.2421 lr=1.13e-04 tok=9.248B dt=14659 wait=138 +20:12:29 step 70580 loss=3.0961 lr=1.13e-04 tok=9.251B dt=14735 wait=139 +20:13:44 step 70600 loss=3.1389 lr=1.13e-04 tok=9.254B dt=14810 wait=139 +20:15:00 step 70620 loss=3.1734 lr=1.13e-04 tok=9.256B dt=14885 wait=140 +20:16:15 step 70640 loss=3.2056 lr=1.13e-04 tok=9.259B dt=14960 wait=140 +20:17:04 pushed ckpt_w23.pt,log_w23.txt +20:20:47 step 70660 loss=3.2236 lr=1.13e-04 tok=9.262B dt=15232 wait=140 +20:22:02 step 70680 loss=3.2488 lr=1.13e-04 tok=9.264B dt=15307 wait=141 +20:23:18 step 70700 loss=3.2542 lr=1.13e-04 tok=9.267B dt=15383 wait=141 +20:24:33 step 70720 loss=3.2920 lr=1.13e-04 tok=9.269B dt=15458 wait=142 +20:25:49 step 70740 loss=3.2782 lr=1.13e-04 tok=9.272B dt=15534 wait=142 +20:27:04 step 70760 loss=3.1981 lr=1.13e-04 tok=9.275B dt=15609 wait=143 +20:28:19 step 70780 loss=3.1365 lr=1.13e-04 tok=9.277B dt=15684 wait=143 +20:29:34 step 70800 loss=3.1918 lr=1.13e-04 tok=9.280B dt=15759 wait=143 +20:29:46 pushed ckpt_w23.pt,log_w23.txt +20:34:07 step 70820 loss=3.2613 lr=1.13e-04 tok=9.283B dt=16032 wait=144 +20:35:22 step 70840 loss=3.3046 lr=1.13e-04 tok=9.285B dt=16107 wait=144 +20:36:38 step 70860 loss=3.3277 lr=1.13e-04 tok=9.288B dt=16183 wait=145 +20:37:53 step 70880 loss=3.1201 lr=1.13e-04 tok=9.290B dt=16258 wait=145 +20:39:08 step 70900 loss=3.1495 lr=1.13e-04 tok=9.293B dt=16333 wait=146 +20:40:23 step 70920 loss=3.2393 lr=1.13e-04 tok=9.296B dt=16408 wait=146 +20:41:39 step 70940 loss=3.2550 lr=1.13e-04 tok=9.298B dt=16484 wait=146 +20:42:28 pushed ckpt_w23.pt,log_w23.txt +20:46:14 step 70960 loss=3.2576 lr=1.13e-04 tok=9.301B dt=16759 wait=147 +20:47:29 step 70980 loss=3.2916 lr=1.13e-04 tok=9.303B dt=16834 wait=147 +20:48:44 step 71000 loss=3.3166 lr=1.13e-04 tok=9.306B dt=16909 wait=148 +20:50:00 step 71020 loss=3.2359 lr=1.13e-04 tok=9.309B dt=16985 wait=148 +20:51:15 step 71040 loss=3.1980 lr=1.13e-04 tok=9.311B dt=17060 wait=148 +20:52:30 step 71060 loss=3.1923 lr=1.13e-04 tok=9.314B dt=17135 wait=149 +20:53:45 step 71080 loss=3.2352 lr=1.13e-04 tok=9.317B dt=17210 wait=149 +20:55:00 step 71100 loss=3.2602 lr=1.13e-04 tok=9.319B dt=17286 wait=150 +20:55:14 pushed ckpt_w23.pt,log_w23.txt +20:59:36 step 71120 loss=3.2159 lr=1.13e-04 tok=9.322B dt=17561 wait=150 +21:00:51 step 71140 loss=3.2366 lr=1.13e-04 tok=9.324B dt=17636 wait=151 +21:02:06 step 71160 loss=3.2442 lr=1.13e-04 tok=9.327B dt=17711 wait=151 +21:03:22 step 71180 loss=3.1837 lr=1.13e-04 tok=9.330B dt=17787 wait=151 +21:04:37 step 71200 loss=3.2400 lr=1.12e-04 tok=9.332B dt=17862 wait=152 +21:05:52 step 71220 loss=3.2271 lr=1.12e-04 tok=9.335B dt=17937 wait=152 +21:07:07 step 71240 loss=3.1759 lr=1.12e-04 tok=9.338B dt=18012 wait=153 +21:08:00 pushed ckpt_w23.pt,log_w23.txt +21:11:45 step 71260 loss=3.2349 lr=1.12e-04 tok=9.340B dt=18290 wait=153 +21:13:00 step 71280 loss=3.2029 lr=1.12e-04 tok=9.343B dt=18365 wait=153 +21:14:15 step 71300 loss=3.1908 lr=1.12e-04 tok=9.345B dt=18440 wait=154 +21:15:30 step 71320 loss=3.0802 lr=1.12e-04 tok=9.348B dt=18516 wait=154 +21:16:46 step 71340 loss=3.2319 lr=1.12e-04 tok=9.351B dt=18591 wait=155 +21:18:01 step 71360 loss=3.1794 lr=1.12e-04 tok=9.353B dt=18666 wait=155 +21:19:16 step 71380 loss=3.1703 lr=1.12e-04 tok=9.356B dt=18741 wait=155 +21:20:31 step 71400 loss=3.1972 lr=1.12e-04 tok=9.359B dt=18816 wait=156 +21:20:46 pushed ckpt_w23.pt,log_w23.txt +21:25:08 step 71420 loss=3.1966 lr=1.12e-04 tok=9.361B dt=19093 wait=156 +21:26:23 step 71440 loss=3.2324 lr=1.12e-04 tok=9.364B dt=19169 wait=157 +21:27:39 step 71460 loss=3.2432 lr=1.12e-04 tok=9.366B dt=19244 wait=157 +21:28:54 step 71480 loss=3.1278 lr=1.12e-04 tok=9.369B dt=19319 wait=157 +21:30:09 step 71500 loss=3.1568 lr=1.12e-04 tok=9.372B dt=19394 wait=158 +21:31:25 step 71520 loss=3.1912 lr=1.12e-04 tok=9.374B dt=19470 wait=158 +21:32:40 step 71540 loss=3.3237 lr=1.12e-04 tok=9.377B dt=19545 wait=159 +21:33:29 pushed ckpt_w23.pt,log_w23.txt +21:37:18 step 71560 loss=3.1429 lr=1.12e-04 tok=9.380B dt=19823 wait=159 +21:38:33 step 71580 loss=3.2314 lr=1.12e-04 tok=9.382B dt=19898 wait=159 +21:39:48 step 71600 loss=3.1793 lr=1.12e-04 tok=9.385B dt=19973 wait=160 +21:41:04 step 71620 loss=3.2043 lr=1.12e-04 tok=9.387B dt=20049 wait=160 +21:42:19 step 71640 loss=3.1690 lr=1.12e-04 tok=9.390B dt=20124 wait=161 +21:43:34 step 71660 loss=3.2629 lr=1.12e-04 tok=9.393B dt=20200 wait=161 +21:44:50 step 71680 loss=3.2028 lr=1.12e-04 tok=9.395B dt=20275 wait=162 +21:46:05 step 71700 loss=3.2409 lr=1.12e-04 tok=9.398B dt=20350 wait=162 +21:46:19 pushed ckpt_w23.pt,log_w23.txt +21:50:41 step 71720 loss=3.1604 lr=1.12e-04 tok=9.400B dt=20626 wait=162 +21:51:56 step 71740 loss=3.1750 lr=1.12e-04 tok=9.403B dt=20701 wait=163 +21:53:11 step 71760 loss=3.1639 lr=1.12e-04 tok=9.406B dt=20776 wait=163 +21:54:27 step 71780 loss=3.2162 lr=1.12e-04 tok=9.408B dt=20852 wait=164 +21:55:42 step 71800 loss=3.2578 lr=1.12e-04 tok=9.411B dt=20927 wait=164 +21:56:57 step 71820 loss=3.1698 lr=1.12e-04 tok=9.414B dt=21002 wait=164 +21:58:13 step 71840 loss=3.3288 lr=1.12e-04 tok=9.416B dt=21078 wait=165 +21:59:07 pushed ckpt_w23.pt,log_w23.txt +21:59:07 waited 0s for global r=134 +21:59:19 global r=134 load 4.296 -> 4.266 +22:00:04 step 71860 loss=3.1130 lr=1.12e-04 tok=9.419B dt=21190 wait=165 +22:01:20 step 71880 loss=3.1719 lr=1.12e-04 tok=9.421B dt=21265 wait=166 +22:02:35 step 71900 loss=3.2058 lr=1.12e-04 tok=9.424B dt=21340 wait=166 +22:03:50 step 71920 loss=3.1969 lr=1.12e-04 tok=9.427B dt=21415 wait=166 +22:05:06 step 71940 loss=3.1633 lr=1.12e-04 tok=9.429B dt=21491 wait=167 +22:06:21 step 71960 loss=3.2595 lr=1.12e-04 tok=9.432B dt=21566 wait=167 +22:07:36 step 71980 loss=3.1644 lr=1.12e-04 tok=9.435B dt=21642 wait=168 +22:08:52 step 72000 loss=3.1719 lr=1.12e-04 tok=9.437B dt=21717 wait=168 +22:09:04 pushed ckpt_w23.pt,log_w23.txt +22:13:28 step 72020 loss=3.2068 lr=1.12e-04 tok=9.440B dt=21993 wait=169 +22:14:43 step 72040 loss=3.1247 lr=1.12e-04 tok=9.442B dt=22068 wait=169 +22:15:58 step 72060 loss=3.1210 lr=1.12e-04 tok=9.445B dt=22143 wait=169 +22:17:13 step 72080 loss=3.1635 lr=1.12e-04 tok=9.448B dt=22218 wait=170 +22:18:28 step 72100 loss=3.1277 lr=1.12e-04 tok=9.450B dt=22293 wait=170 +22:19:43 step 72120 loss=3.2289 lr=1.12e-04 tok=9.453B dt=22368 wait=170 +22:20:58 step 72140 loss=3.1874 lr=1.12e-04 tok=9.456B dt=22443 wait=171 +22:21:49 pushed ckpt_w23.pt,log_w23.txt +22:21:49 waited 0s for global r=135 +22:22:00 global r=135 load 4.293 -> 4.267 +22:22:43 step 72160 loss=3.1936 lr=1.12e-04 tok=9.458B dt=22548 wait=171 +22:23:58 step 72180 loss=3.1760 lr=1.12e-04 tok=9.461B dt=22623 wait=172 +22:25:13 step 72200 loss=3.1814 lr=1.12e-04 tok=9.463B dt=22698 wait=172 +22:26:28 step 72220 loss=3.1935 lr=1.12e-04 tok=9.466B dt=22773 wait=172 +22:27:43 step 72240 loss=3.1247 lr=1.11e-04 tok=9.469B dt=22848 wait=173 +22:28:58 step 72260 loss=3.2521 lr=1.11e-04 tok=9.471B dt=22923 wait=173 +22:30:13 step 72280 loss=3.1674 lr=1.11e-04 tok=9.474B dt=22999 wait=174 +22:31:29 step 72300 loss=3.1924 lr=1.11e-04 tok=9.477B dt=23074 wait=174 +22:31:43 pushed ckpt_w23.pt,log_w23.txt +22:31:59 waited 16s for global r=136 +22:32:12 global r=136 load 4.275 -> 4.264 +22:33:32 step 72320 loss=3.1536 lr=1.11e-04 tok=9.479B dt=23197 wait=174 +22:34:47 step 72340 loss=3.1568 lr=1.11e-04 tok=9.482B dt=23272 wait=175 +22:36:02 step 72360 loss=3.1599 lr=1.11e-04 tok=9.484B dt=23347 wait=175 +22:37:17 step 72380 loss=3.2043 lr=1.11e-04 tok=9.487B dt=23422 wait=176 +22:38:32 step 72400 loss=3.1989 lr=1.11e-04 tok=9.490B dt=23497 wait=176 +22:39:47 step 72420 loss=3.1481 lr=1.11e-04 tok=9.492B dt=23572 wait=176 +22:41:02 step 72440 loss=3.1869 lr=1.11e-04 tok=9.495B dt=23648 wait=177 +22:41:53 pushed ckpt_w23.pt,log_w23.txt +22:45:39 step 72460 loss=3.1595 lr=1.11e-04 tok=9.497B dt=23924 wait=177 +22:46:54 step 72480 loss=3.2263 lr=1.11e-04 tok=9.500B dt=23999 wait=178 +22:48:09 step 72500 loss=3.1145 lr=1.11e-04 tok=9.503B dt=24074 wait=178 +22:49:24 step 72520 loss=3.0984 lr=1.11e-04 tok=9.505B dt=24149 wait=178 +22:50:40 step 72540 loss=3.1406 lr=1.11e-04 tok=9.508B dt=24225 wait=179 +22:51:55 step 72560 loss=3.1760 lr=1.11e-04 tok=9.511B dt=24300 wait=179 +22:53:11 step 72580 loss=3.2063 lr=1.11e-04 tok=9.513B dt=24376 wait=180 +22:54:26 step 72600 loss=3.2116 lr=1.11e-04 tok=9.516B dt=24451 wait=180 +22:54:39 pushed ckpt_w23.pt,log_w23.txt +22:54:39 waited 0s for global r=137 +22:54:47 global r=137 load 4.276 -> 4.258 +22:56:09 step 72620 loss=3.1585 lr=1.11e-04 tok=9.518B dt=24554 wait=180 +22:57:24 step 72640 loss=3.1639 lr=1.11e-04 tok=9.521B dt=24629 wait=181 +22:58:39 step 72660 loss=3.1682 lr=1.11e-04 tok=9.524B dt=24705 wait=181 +22:59:55 step 72680 loss=3.2040 lr=1.11e-04 tok=9.526B dt=24780 wait=182 +23:01:10 step 72700 loss=3.1790 lr=1.11e-04 tok=9.529B dt=24855 wait=182 +23:02:25 step 72720 loss=3.1500 lr=1.11e-04 tok=9.532B dt=24930 wait=183 +23:03:40 step 72740 loss=3.2203 lr=1.11e-04 tok=9.534B dt=25005 wait=183 diff --git a/log_w24.txt b/log_w24.txt new file mode 100644 index 0000000000000000000000000000000000000000..3edea792ea4e7405bad7df96b09d79a33134fdbc --- /dev/null +++ b/log_w24.txt @@ -0,0 +1,155 @@ +launch +22:05:11 name=w24 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 3/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:05:17 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:05:27 params 1.08B +22:05:28 resumed step=71700 seen=9.40B round=134 +22:05:28 stream salt=71700 +22:08:08 start: accum=4, 131072 tokens per optimiser step +22:10:25 step 71701 loss=3.3428 lr=1.12e-04 tok=9.398B dt=137 wait=107 +22:10:30 step 71702 loss=3.1767 lr=1.12e-04 tok=9.398B dt=141 wait=107 +22:11:56 step 71720 loss=3.3235 lr=1.12e-04 tok=9.400B dt=227 wait=107 +22:13:29 step 71740 loss=3.2929 lr=1.12e-04 tok=9.403B dt=321 wait=108 +22:14:45 step 71760 loss=3.3257 lr=1.12e-04 tok=9.406B dt=396 wait=108 +22:16:00 step 71780 loss=3.1340 lr=1.12e-04 tok=9.408B dt=471 wait=109 +22:17:14 step 71800 loss=3.1880 lr=1.12e-04 tok=9.411B dt=546 wait=109 +22:18:29 step 71820 loss=3.1240 lr=1.12e-04 tok=9.414B dt=621 wait=109 +22:19:44 step 71840 loss=3.1515 lr=1.12e-04 tok=9.416B dt=696 wait=110 +22:20:36 pushed ckpt_w24.pt,log_w24.txt +22:20:36 waited 0s for global r=135 +22:20:50 global r=135 load 4.297 -> 4.267 +22:21:36 step 71860 loss=3.2357 lr=1.12e-04 tok=9.419B dt=808 wait=110 +22:22:51 step 71880 loss=3.1556 lr=1.12e-04 tok=9.421B dt=883 wait=110 +22:24:06 step 71900 loss=3.1978 lr=1.12e-04 tok=9.424B dt=957 wait=111 +22:25:21 step 71920 loss=3.2195 lr=1.12e-04 tok=9.427B dt=1033 wait=111 +22:26:36 step 71940 loss=3.1892 lr=1.12e-04 tok=9.429B dt=1108 wait=112 +22:27:51 step 71960 loss=3.2719 lr=1.12e-04 tok=9.432B dt=1183 wait=112 +22:29:06 step 71980 loss=3.1527 lr=1.12e-04 tok=9.435B dt=1258 wait=112 +22:30:22 step 72000 loss=3.2088 lr=1.12e-04 tok=9.437B dt=1333 wait=113 +22:30:34 pushed ckpt_w24.pt,log_w24.txt +22:31:49 waited 76s for global r=136 +22:32:02 global r=136 load 4.282 -> 4.264 +22:33:29 step 72020 loss=3.1581 lr=1.12e-04 tok=9.440B dt=1520 wait=113 +22:34:44 step 72040 loss=3.1529 lr=1.12e-04 tok=9.442B dt=1595 wait=114 +22:35:59 step 72060 loss=3.1340 lr=1.12e-04 tok=9.445B dt=1670 wait=114 +22:37:14 step 72080 loss=3.2003 lr=1.12e-04 tok=9.448B dt=1745 wait=114 +22:38:29 step 72100 loss=3.2352 lr=1.12e-04 tok=9.450B dt=1821 wait=115 +22:39:44 step 72120 loss=3.1854 lr=1.12e-04 tok=9.453B dt=1896 wait=115 +22:40:59 step 72140 loss=3.2410 lr=1.12e-04 tok=9.456B dt=1971 wait=116 +22:41:51 pushed ckpt_w24.pt,log_w24.txt +22:45:38 step 72160 loss=3.1819 lr=1.12e-04 tok=9.458B dt=2250 wait=116 +22:46:53 step 72180 loss=3.2136 lr=1.12e-04 tok=9.461B dt=2325 wait=116 +22:48:08 step 72200 loss=3.1110 lr=1.12e-04 tok=9.463B dt=2399 wait=117 +22:49:23 step 72220 loss=3.0575 lr=1.12e-04 tok=9.466B dt=2474 wait=117 +22:50:38 step 72240 loss=3.1025 lr=1.11e-04 tok=9.469B dt=2549 wait=118 +22:51:53 step 72260 loss=3.1311 lr=1.11e-04 tok=9.471B dt=2624 wait=118 +22:53:09 step 72280 loss=3.2395 lr=1.11e-04 tok=9.474B dt=2700 wait=119 +22:54:24 step 72300 loss=3.1914 lr=1.11e-04 tok=9.477B dt=2776 wait=119 +22:54:37 pushed ckpt_w24.pt,log_w24.txt +22:54:38 waited 0s for global r=137 +22:54:45 global r=137 load 4.275 -> 4.258 +22:56:08 step 72320 loss=3.1928 lr=1.11e-04 tok=9.479B dt=2879 wait=120 +22:57:23 step 72340 loss=3.1784 lr=1.11e-04 tok=9.482B dt=2954 wait=120 +22:58:38 step 72360 loss=3.1358 lr=1.11e-04 tok=9.484B dt=3030 wait=120 +22:59:53 step 72380 loss=3.2154 lr=1.11e-04 tok=9.487B dt=3105 wait=121 +23:01:08 step 72400 loss=3.1604 lr=1.11e-04 tok=9.490B dt=3180 wait=121 +23:02:23 step 72420 loss=3.1223 lr=1.11e-04 tok=9.492B dt=3254 wait=122 +23:03:38 step 72440 loss=3.1859 lr=1.11e-04 tok=9.495B dt=3330 wait=122 +23:04:28 pushed ckpt_w24.pt,log_w24.txt +23:05:44 waited 76s for global r=138 +23:05:55 global r=138 load 4.281 -> 4.256 +23:06:40 step 72460 loss=3.1486 lr=1.11e-04 tok=9.497B dt=3512 wait=122 +23:07:55 step 72480 loss=3.1344 lr=1.11e-04 tok=9.500B dt=3587 wait=123 +23:09:10 step 72500 loss=3.1973 lr=1.11e-04 tok=9.503B dt=3662 wait=123 +23:10:25 step 72520 loss=3.2157 lr=1.11e-04 tok=9.505B dt=3737 wait=124 +23:11:40 step 72540 loss=3.1246 lr=1.11e-04 tok=9.508B dt=3812 wait=124 +23:12:55 step 72560 loss=3.2187 lr=1.11e-04 tok=9.511B dt=3887 wait=124 +23:14:10 step 72580 loss=3.0852 lr=1.11e-04 tok=9.513B dt=3962 wait=125 +23:15:26 step 72600 loss=3.1611 lr=1.11e-04 tok=9.516B dt=4037 wait=125 +23:15:40 pushed ckpt_w24.pt,log_w24.txt +23:20:01 step 72620 loss=3.2042 lr=1.11e-04 tok=9.518B dt=4312 wait=126 +23:21:15 step 72640 loss=3.1594 lr=1.11e-04 tok=9.521B dt=4387 wait=126 +23:22:31 step 72660 loss=3.1490 lr=1.11e-04 tok=9.524B dt=4462 wait=126 +23:23:46 step 72680 loss=3.1329 lr=1.11e-04 tok=9.526B dt=4538 wait=127 +23:25:01 step 72700 loss=3.0860 lr=1.11e-04 tok=9.529B dt=4612 wait=127 +23:26:16 step 72720 loss=3.2061 lr=1.11e-04 tok=9.532B dt=4687 wait=128 +23:27:31 step 72740 loss=3.1291 lr=1.11e-04 tok=9.534B dt=4762 wait=128 +23:28:20 pushed ckpt_w24.pt,log_w24.txt +23:28:21 waited 0s for global r=139 +23:28:32 global r=139 load 4.280 -> 4.260 +23:29:16 step 72760 loss=3.1210 lr=1.11e-04 tok=9.537B dt=4867 wait=128 +23:30:31 step 72780 loss=3.1397 lr=1.11e-04 tok=9.539B dt=4942 wait=129 +23:31:46 step 72800 loss=3.2365 lr=1.11e-04 tok=9.542B dt=5017 wait=129 +23:33:01 step 72820 loss=3.2567 lr=1.11e-04 tok=9.545B dt=5093 wait=130 +23:34:16 step 72840 loss=3.1526 lr=1.11e-04 tok=9.547B dt=5168 wait=130 +23:35:31 step 72860 loss=3.2668 lr=1.11e-04 tok=9.550B dt=5243 wait=130 +23:36:46 step 72880 loss=3.0999 lr=1.11e-04 tok=9.553B dt=5318 wait=131 +23:38:01 step 72900 loss=3.1593 lr=1.11e-04 tok=9.555B dt=5393 wait=131 +23:38:13 pushed ckpt_w24.pt,log_w24.txt +23:39:29 waited 76s for global r=140 +23:39:35 global r=140 load 4.274 -> 4.257 +23:40:57 step 72920 loss=3.1467 lr=1.11e-04 tok=9.558B dt=5568 wait=132 +23:42:12 step 72940 loss=3.1644 lr=1.11e-04 tok=9.560B dt=5644 wait=132 +23:43:27 step 72960 loss=3.1473 lr=1.11e-04 tok=9.563B dt=5719 wait=132 +23:44:42 step 72980 loss=3.2345 lr=1.11e-04 tok=9.566B dt=5794 wait=133 +23:45:57 step 73000 loss=3.2166 lr=1.11e-04 tok=9.568B dt=5869 wait=133 +23:47:12 step 73020 loss=3.1673 lr=1.11e-04 tok=9.571B dt=5944 wait=134 +23:48:28 step 73040 loss=3.2208 lr=1.11e-04 tok=9.573B dt=6019 wait=134 +23:49:24 pushed ckpt_w24.pt,log_w24.txt +23:53:08 step 73060 loss=3.1609 lr=1.11e-04 tok=9.576B dt=6300 wait=134 +23:54:23 step 73080 loss=3.2059 lr=1.11e-04 tok=9.579B dt=6375 wait=135 +23:55:38 step 73100 loss=3.1144 lr=1.11e-04 tok=9.581B dt=6450 wait=135 +23:56:53 step 73120 loss=3.0892 lr=1.11e-04 tok=9.584B dt=6525 wait=136 +23:58:08 step 73140 loss=3.1353 lr=1.11e-04 tok=9.587B dt=6600 wait=136 +23:59:23 step 73160 loss=3.1466 lr=1.11e-04 tok=9.589B dt=6674 wait=136 +00:00:38 step 73180 loss=3.1839 lr=1.11e-04 tok=9.592B dt=6749 wait=137 +00:01:53 step 73200 loss=3.1493 lr=1.11e-04 tok=9.594B dt=6825 wait=137 +00:02:05 pushed ckpt_w24.pt,log_w24.txt +00:02:05 waited 0s for global r=141 +00:02:13 global r=141 load 4.277 -> 4.253 +00:03:35 step 73220 loss=3.1534 lr=1.11e-04 tok=9.597B dt=6927 wait=138 +00:04:50 step 73240 loss=3.1977 lr=1.11e-04 tok=9.600B dt=7002 wait=138 +00:06:05 step 73260 loss=3.2114 lr=1.11e-04 tok=9.602B dt=7077 wait=138 +00:07:20 step 73280 loss=3.2611 lr=1.10e-04 tok=9.605B dt=7152 wait=139 +00:08:35 step 73300 loss=3.1783 lr=1.10e-04 tok=9.608B dt=7227 wait=139 +00:09:50 step 73320 loss=3.1127 lr=1.10e-04 tok=9.610B dt=7302 wait=139 +00:11:05 step 73340 loss=3.1551 lr=1.10e-04 tok=9.613B dt=7377 wait=140 +00:11:55 pushed ckpt_w24.pt,log_w24.txt +00:13:11 waited 76s for global r=142 +00:13:22 global r=142 load 4.286 -> 4.263 +00:14:09 step 73360 loss=3.0968 lr=1.10e-04 tok=9.615B dt=7560 wait=140 +00:15:24 step 73380 loss=3.1271 lr=1.10e-04 tok=9.618B dt=7635 wait=141 +00:16:39 step 73400 loss=3.2348 lr=1.10e-04 tok=9.621B dt=7710 wait=141 +00:17:54 step 73420 loss=3.2654 lr=1.10e-04 tok=9.623B dt=7786 wait=142 +00:19:09 step 73440 loss=3.1821 lr=1.10e-04 tok=9.626B dt=7861 wait=142 +00:20:24 step 73460 loss=3.2999 lr=1.10e-04 tok=9.629B dt=7936 wait=142 +00:21:39 step 73480 loss=3.1155 lr=1.10e-04 tok=9.631B dt=8011 wait=143 +00:22:54 step 73500 loss=3.1860 lr=1.10e-04 tok=9.634B dt=8086 wait=143 +00:23:06 pushed ckpt_w24.pt,log_w24.txt +00:27:37 step 73520 loss=3.1988 lr=1.10e-04 tok=9.636B dt=8369 wait=143 +00:28:52 step 73540 loss=3.0802 lr=1.10e-04 tok=9.639B dt=8444 wait=144 +00:30:07 step 73560 loss=3.1111 lr=1.10e-04 tok=9.642B dt=8519 wait=144 +00:31:22 step 73580 loss=3.1260 lr=1.10e-04 tok=9.644B dt=8593 wait=145 +00:32:37 step 73600 loss=3.0593 lr=1.10e-04 tok=9.647B dt=8668 wait=145 +00:33:52 step 73620 loss=3.2443 lr=1.10e-04 tok=9.650B dt=8743 wait=145 +00:35:07 step 73640 loss=3.2104 lr=1.10e-04 tok=9.652B dt=8818 wait=146 +00:35:58 pushed ckpt_w24.pt,log_w24.txt +00:35:58 waited 0s for global r=143 +00:36:07 global r=143 load 4.278 -> 4.254 +00:36:52 step 73660 loss=3.1481 lr=1.10e-04 tok=9.655B dt=8923 wait=146 +00:38:07 step 73680 loss=3.1342 lr=1.10e-04 tok=9.657B dt=8998 wait=147 +00:39:22 step 73700 loss=3.1724 lr=1.10e-04 tok=9.660B dt=9073 wait=147 +00:40:37 step 73720 loss=3.1820 lr=1.10e-04 tok=9.663B dt=9148 wait=147 +00:41:52 step 73740 loss=3.1548 lr=1.10e-04 tok=9.665B dt=9223 wait=148 +00:43:07 step 73760 loss=3.2517 lr=1.10e-04 tok=9.668B dt=9298 wait=148 +00:44:22 step 73780 loss=3.1384 lr=1.10e-04 tok=9.670B dt=9373 wait=149 +00:45:37 step 73800 loss=3.2370 lr=1.10e-04 tok=9.673B dt=9448 wait=149 +00:45:51 pushed ckpt_w24.pt,log_w24.txt +00:46:21 waited 30s for global r=144 +00:46:33 global r=144 load 4.274 -> 4.261 +00:47:55 step 73820 loss=3.1496 lr=1.10e-04 tok=9.676B dt=9587 wait=149 +00:49:10 step 73840 loss=3.1767 lr=1.10e-04 tok=9.678B dt=9662 wait=150 +00:50:25 step 73860 loss=3.1213 lr=1.10e-04 tok=9.681B dt=9737 wait=150 +00:51:40 step 73880 loss=3.1795 lr=1.10e-04 tok=9.684B dt=9811 wait=151 +00:52:55 step 73900 loss=3.1645 lr=1.10e-04 tok=9.686B dt=9886 wait=151 +00:54:10 step 73920 loss=3.1464 lr=1.10e-04 tok=9.689B dt=9961 wait=151 +00:55:25 step 73940 loss=3.2030 lr=1.10e-04 tok=9.691B dt=10036 wait=152 diff --git a/log_w25.txt b/log_w25.txt new file mode 100644 index 0000000000000000000000000000000000000000..64ecd20bcb79e1657489cd0c381f768016dee9ec --- /dev/null +++ b/log_w25.txt @@ -0,0 +1,150 @@ +launch +19:54:40 name=w25 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 4/6 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:54:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:54:55 params 1.08B +19:54:57 resumed step=55200 seen=7.24B round=80 +19:54:57 stream salt=55200 +19:57:26 start: accum=4, 131072 tokens per optimiser step +19:59:06 step 55201 loss=3.5197 lr=2.53e-04 tok=7.235B dt=99 wait=92 +19:59:11 step 55202 loss=3.4048 lr=2.53e-04 tok=7.235B dt=105 wait=92 +20:00:46 step 55220 loss=3.4988 lr=2.53e-04 tok=7.238B dt=200 wait=92 +20:02:25 step 55240 loss=3.4919 lr=2.53e-04 tok=7.240B dt=298 wait=92 +20:03:41 step 55260 loss=3.4809 lr=2.53e-04 tok=7.243B dt=375 wait=93 +20:04:57 step 55280 loss=3.3921 lr=2.53e-04 tok=7.246B dt=450 wait=93 +20:06:12 step 55300 loss=3.4788 lr=2.53e-04 tok=7.248B dt=526 wait=94 +20:07:28 step 55320 loss=3.3775 lr=2.53e-04 tok=7.251B dt=602 wait=94 +20:08:44 step 55340 loss=3.3712 lr=2.53e-04 tok=7.254B dt=678 wait=94 +20:09:36 pushed ckpt_w25.pt,log_w25.txt +20:10:18 step 55360 loss=3.4724 lr=2.53e-04 tok=7.256B dt=772 wait=95 +20:11:34 step 55380 loss=3.4812 lr=2.53e-04 tok=7.259B dt=848 wait=95 +20:12:50 step 55400 loss=3.5381 lr=2.53e-04 tok=7.261B dt=924 wait=96 +20:14:06 step 55420 loss=3.4973 lr=2.53e-04 tok=7.264B dt=1000 wait=96 +20:15:22 step 55440 loss=3.4278 lr=2.53e-04 tok=7.267B dt=1076 wait=96 +20:16:38 step 55460 loss=3.3503 lr=2.53e-04 tok=7.269B dt=1152 wait=97 +20:18:01 global r=81 rebase 4.391 -> 4.404 +20:18:21 step 55480 loss=3.4862 lr=2.53e-04 tok=7.272B dt=1254 wait=97 +20:19:37 step 55500 loss=3.3976 lr=2.53e-04 tok=7.274B dt=1330 wait=98 +20:19:53 pushed ckpt_w25.pt,log_w25.txt +20:21:18 step 55520 loss=3.4234 lr=2.53e-04 tok=7.277B dt=1431 wait=98 +20:22:34 step 55540 loss=3.4298 lr=2.53e-04 tok=7.280B dt=1508 wait=98 +20:23:50 step 55560 loss=3.3888 lr=2.53e-04 tok=7.282B dt=1584 wait=99 +20:25:06 step 55580 loss=3.4705 lr=2.53e-04 tok=7.285B dt=1660 wait=99 +20:26:22 step 55600 loss=3.3947 lr=2.53e-04 tok=7.288B dt=1736 wait=100 +20:27:38 step 55620 loss=3.4889 lr=2.52e-04 tok=7.290B dt=1812 wait=100 +20:28:10 global r=82 rebase 4.388 -> 4.430 +20:29:07 step 55640 loss=3.4147 lr=2.52e-04 tok=7.293B dt=1901 wait=100 +20:29:57 pushed ckpt_w25.pt,log_w25.txt +20:30:39 step 55660 loss=3.4294 lr=2.52e-04 tok=7.295B dt=1993 wait=101 +20:31:56 step 55680 loss=3.4289 lr=2.52e-04 tok=7.298B dt=2069 wait=101 +20:33:12 step 55700 loss=3.4540 lr=2.52e-04 tok=7.301B dt=2145 wait=102 +20:34:28 step 55720 loss=3.3693 lr=2.52e-04 tok=7.303B dt=2221 wait=102 +20:35:44 step 55740 loss=3.4729 lr=2.52e-04 tok=7.306B dt=2297 wait=102 +20:36:59 step 55760 loss=3.3686 lr=2.52e-04 tok=7.309B dt=2373 wait=103 +20:38:12 global r=83 rebase 4.404 -> 4.449 +20:38:31 step 55780 loss=3.4691 lr=2.52e-04 tok=7.311B dt=2465 wait=103 +20:39:47 step 55800 loss=3.4190 lr=2.52e-04 tok=7.314B dt=2541 wait=104 +20:40:02 pushed ckpt_w25.pt,log_w25.txt +20:41:25 step 55820 loss=3.4060 lr=2.52e-04 tok=7.316B dt=2638 wait=104 +20:42:41 step 55840 loss=3.4404 lr=2.52e-04 tok=7.319B dt=2714 wait=104 +20:43:57 step 55860 loss=3.4625 lr=2.52e-04 tok=7.322B dt=2790 wait=105 +20:45:13 step 55880 loss=3.4567 lr=2.52e-04 tok=7.324B dt=2867 wait=105 +20:46:29 step 55900 loss=3.3649 lr=2.52e-04 tok=7.327B dt=2942 wait=106 +20:46:46 global r=84 rebase 4.392 -> 4.447 +20:48:02 step 55920 loss=3.4734 lr=2.52e-04 tok=7.330B dt=3036 wait=106 +20:49:18 step 55940 loss=3.4605 lr=2.52e-04 tok=7.332B dt=3112 wait=106 +20:50:08 pushed ckpt_w25.pt,log_w25.txt +20:50:53 step 55960 loss=3.4074 lr=2.52e-04 tok=7.335B dt=3206 wait=107 +20:52:09 step 55980 loss=3.4670 lr=2.52e-04 tok=7.337B dt=3282 wait=107 +20:53:24 step 56000 loss=3.4394 lr=2.52e-04 tok=7.340B dt=3358 wait=108 +20:54:40 step 56020 loss=3.4011 lr=2.52e-04 tok=7.343B dt=3434 wait=108 +20:55:56 step 56040 loss=3.4337 lr=2.52e-04 tok=7.345B dt=3510 wait=108 +20:57:12 step 56060 loss=3.4010 lr=2.52e-04 tok=7.348B dt=3586 wait=109 +20:58:21 global r=85 rebase 4.382 -> 4.463 +20:58:40 step 56080 loss=3.4416 lr=2.52e-04 tok=7.351B dt=3674 wait=109 +20:59:56 step 56100 loss=3.4445 lr=2.52e-04 tok=7.353B dt=3750 wait=110 +21:00:10 pushed ckpt_w25.pt,log_w25.txt +21:01:39 step 56120 loss=3.3900 lr=2.52e-04 tok=7.356B dt=3852 wait=110 +21:02:55 step 56140 loss=3.4465 lr=2.52e-04 tok=7.358B dt=3928 wait=110 +21:04:11 step 56160 loss=3.4481 lr=2.52e-04 tok=7.361B dt=4004 wait=111 +21:05:27 step 56180 loss=3.4886 lr=2.52e-04 tok=7.364B dt=4080 wait=111 +21:06:42 step 56200 loss=3.3818 lr=2.52e-04 tok=7.366B dt=4156 wait=111 +21:07:00 global r=86 rebase 4.392 -> 4.470 +21:08:16 step 56220 loss=3.4894 lr=2.52e-04 tok=7.369B dt=4250 wait=112 +21:09:32 step 56240 loss=3.4746 lr=2.51e-04 tok=7.371B dt=4326 wait=112 +21:10:28 pushed ckpt_w25.pt,log_w25.txt +21:11:12 step 56260 loss=3.4055 lr=2.51e-04 tok=7.374B dt=4426 wait=113 +21:12:28 step 56280 loss=3.4592 lr=2.51e-04 tok=7.377B dt=4502 wait=113 +21:13:44 step 56300 loss=3.4136 lr=2.51e-04 tok=7.379B dt=4578 wait=113 +21:15:00 step 56320 loss=3.3913 lr=2.51e-04 tok=7.382B dt=4654 wait=114 +21:16:16 step 56340 loss=3.4296 lr=2.51e-04 tok=7.385B dt=4730 wait=114 +21:17:10 global r=87 rebase 4.396 -> 4.477 +21:17:48 step 56360 loss=3.4087 lr=2.51e-04 tok=7.387B dt=4822 wait=115 +21:19:04 step 56380 loss=3.4613 lr=2.51e-04 tok=7.390B dt=4898 wait=115 +21:20:20 step 56400 loss=3.5025 lr=2.51e-04 tok=7.392B dt=4973 wait=115 +21:20:31 pushed ckpt_w25.pt,log_w25.txt +21:21:56 step 56420 loss=3.4901 lr=2.51e-04 tok=7.395B dt=5070 wait=116 +21:23:12 step 56440 loss=3.4193 lr=2.51e-04 tok=7.398B dt=5146 wait=116 +21:24:28 step 56460 loss=3.4785 lr=2.51e-04 tok=7.400B dt=5222 wait=117 +21:25:44 step 56480 loss=3.3319 lr=2.51e-04 tok=7.403B dt=5298 wait=117 +21:27:00 step 56500 loss=3.3933 lr=2.51e-04 tok=7.406B dt=5374 wait=117 +21:27:14 global r=88 rebase 4.390 -> 4.484 +21:28:30 step 56520 loss=3.5086 lr=2.51e-04 tok=7.408B dt=5464 wait=118 +21:29:47 step 56540 loss=3.5090 lr=2.51e-04 tok=7.411B dt=5541 wait=118 +21:30:37 pushed ckpt_w25.pt,log_w25.txt +21:31:21 step 56560 loss=3.4566 lr=2.51e-04 tok=7.413B dt=5635 wait=119 +21:32:37 step 56580 loss=3.4827 lr=2.51e-04 tok=7.416B dt=5711 wait=119 +21:33:53 step 56600 loss=3.4505 lr=2.51e-04 tok=7.419B dt=5787 wait=119 +21:35:09 step 56620 loss=3.3909 lr=2.51e-04 tok=7.421B dt=5863 wait=120 +21:36:26 step 56640 loss=3.4123 lr=2.51e-04 tok=7.424B dt=5939 wait=120 +21:37:15 global r=89 rebase 4.412 -> 4.510 +21:37:53 step 56660 loss=3.4488 lr=2.51e-04 tok=7.427B dt=6027 wait=121 +21:39:09 step 56680 loss=3.5214 lr=2.51e-04 tok=7.429B dt=6103 wait=121 +21:40:25 step 56700 loss=3.5608 lr=2.51e-04 tok=7.432B dt=6179 wait=121 +21:40:37 pushed ckpt_w25.pt,log_w25.txt +21:41:58 step 56720 loss=3.4890 lr=2.51e-04 tok=7.434B dt=6272 wait=122 +21:43:14 step 56740 loss=3.3711 lr=2.51e-04 tok=7.437B dt=6348 wait=122 +21:44:30 step 56760 loss=3.4866 lr=2.51e-04 tok=7.440B dt=6424 wait=123 +21:45:46 step 56780 loss=3.3876 lr=2.51e-04 tok=7.442B dt=6500 wait=123 +21:47:02 step 56800 loss=3.4383 lr=2.51e-04 tok=7.445B dt=6576 wait=123 +21:47:15 global r=90 rebase 4.426 -> 4.521 +21:48:31 step 56820 loss=3.5131 lr=2.51e-04 tok=7.448B dt=6665 wait=124 +21:49:47 step 56840 loss=3.5364 lr=2.50e-04 tok=7.450B dt=6741 wait=124 +21:50:39 pushed ckpt_w25.pt,log_w25.txt +21:51:25 step 56860 loss=3.4927 lr=2.50e-04 tok=7.453B dt=6839 wait=125 +21:52:41 step 56880 loss=3.5087 lr=2.50e-04 tok=7.455B dt=6915 wait=125 +21:53:57 step 56900 loss=3.4539 lr=2.50e-04 tok=7.458B dt=6991 wait=125 +21:55:13 step 56920 loss=3.4008 lr=2.50e-04 tok=7.461B dt=7067 wait=126 +21:56:29 step 56940 loss=3.4484 lr=2.50e-04 tok=7.463B dt=7143 wait=126 +21:57:18 global r=91 rebase 4.410 -> 4.522 +21:57:57 step 56960 loss=3.4682 lr=2.50e-04 tok=7.466B dt=7230 wait=126 +21:59:13 step 56980 loss=3.4994 lr=2.50e-04 tok=7.468B dt=7306 wait=127 +22:00:28 step 57000 loss=3.5703 lr=2.50e-04 tok=7.471B dt=7382 wait=127 +22:15:15 push fail [Errno 110] Connection timed out +22:15:30 global r=92 rebase 4.449 -> 4.570 +22:16:51 step 57020 loss=3.5749 lr=2.50e-04 tok=7.474B dt=8364 wait=128 +22:17:27 global r=93 rebase 4.511 -> 4.493 +22:18:24 step 57040 loss=3.4598 lr=2.50e-04 tok=7.476B dt=8458 wait=128 +22:19:40 step 57060 loss=3.4869 lr=2.50e-04 tok=7.479B dt=8534 wait=128 +22:20:56 step 57080 loss=3.5019 lr=2.50e-04 tok=7.482B dt=8610 wait=129 +22:22:12 step 57100 loss=3.5721 lr=2.50e-04 tok=7.484B dt=8686 wait=129 +22:23:28 step 57120 loss=3.4780 lr=2.50e-04 tok=7.487B dt=8762 wait=130 +22:24:44 step 57140 loss=3.5490 lr=2.50e-04 tok=7.489B dt=8838 wait=130 +22:25:36 pushed ckpt_w25.pt,log_w25.txt +22:26:21 step 57160 loss=3.4102 lr=2.50e-04 tok=7.492B dt=8935 wait=130 +22:27:36 global r=94 rebase 4.453 -> 4.354 +22:27:55 step 57180 loss=3.4537 lr=2.50e-04 tok=7.495B dt=9029 wait=131 +22:29:11 step 57200 loss=3.3435 lr=2.50e-04 tok=7.497B dt=9105 wait=131 +22:30:27 step 57220 loss=3.3737 lr=2.50e-04 tok=7.500B dt=9181 wait=132 +22:31:43 step 57240 loss=3.3915 lr=2.50e-04 tok=7.503B dt=9257 wait=132 +22:32:59 step 57260 loss=3.3845 lr=2.50e-04 tok=7.505B dt=9333 wait=133 +22:34:16 step 57280 loss=3.4029 lr=2.50e-04 tok=7.508B dt=9410 wait=133 +22:35:32 step 57300 loss=3.3080 lr=2.50e-04 tok=7.510B dt=9486 wait=133 +22:35:46 pushed ckpt_w25.pt,log_w25.txt +22:37:09 step 57320 loss=3.3935 lr=2.50e-04 tok=7.513B dt=9583 wait=134 +22:37:43 global r=95 rebase 4.366 -> 4.432 +22:38:40 step 57340 loss=3.3885 lr=2.50e-04 tok=7.516B dt=9674 wait=134 +22:39:56 step 57360 loss=3.5092 lr=2.50e-04 tok=7.518B dt=9750 wait=135 +22:41:12 step 57380 loss=3.4759 lr=2.50e-04 tok=7.521B dt=9826 wait=135 +22:42:28 step 57400 loss=3.4581 lr=2.50e-04 tok=7.524B dt=9902 wait=135 +22:43:44 step 57420 loss=3.3672 lr=2.50e-04 tok=7.526B dt=9978 wait=136 +22:45:00 step 57440 loss=3.4789 lr=2.50e-04 tok=7.529B dt=10054 wait=136 diff --git a/log_w26.txt b/log_w26.txt new file mode 100644 index 0000000000000000000000000000000000000000..2c9931b05db4f05cc9cd4fc8d894802a9d4a1cb6 --- /dev/null +++ b/log_w26.txt @@ -0,0 +1,168 @@ +launch +19:54:39 name=w26 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 5/6 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:54:45 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:54:55 params 1.08B +19:54:56 resumed step=55200 seen=7.24B round=80 +19:54:56 stream salt=55200 +19:57:25 start: accum=4, 131072 tokens per optimiser step +19:59:29 step 55201 loss=3.3928 lr=2.53e-04 tok=7.235B dt=124 wait=92 +19:59:34 step 55202 loss=3.2725 lr=2.53e-04 tok=7.235B dt=129 wait=92 +20:01:10 step 55220 loss=3.5205 lr=2.53e-04 tok=7.238B dt=224 wait=93 +20:02:49 step 55240 loss=3.4867 lr=2.53e-04 tok=7.240B dt=323 wait=93 +20:04:04 step 55260 loss=3.4973 lr=2.53e-04 tok=7.243B dt=399 wait=93 +20:05:20 step 55280 loss=3.3534 lr=2.53e-04 tok=7.246B dt=475 wait=94 +20:06:36 step 55300 loss=3.4307 lr=2.53e-04 tok=7.248B dt=550 wait=94 +20:07:51 step 55320 loss=3.3491 lr=2.53e-04 tok=7.251B dt=626 wait=95 +20:09:07 step 55340 loss=3.3937 lr=2.53e-04 tok=7.254B dt=702 wait=95 +20:10:01 pushed ckpt_w26.pt,log_w26.txt +20:10:48 step 55360 loss=3.5009 lr=2.53e-04 tok=7.256B dt=802 wait=96 +20:12:04 step 55380 loss=3.4859 lr=2.53e-04 tok=7.259B dt=878 wait=96 +20:13:19 step 55400 loss=3.5118 lr=2.53e-04 tok=7.261B dt=954 wait=96 +20:14:35 step 55420 loss=3.4187 lr=2.53e-04 tok=7.264B dt=1030 wait=97 +20:15:51 step 55440 loss=3.4051 lr=2.53e-04 tok=7.267B dt=1105 wait=97 +20:17:06 step 55460 loss=3.4343 lr=2.53e-04 tok=7.269B dt=1181 wait=98 +20:18:24 global r=81 rebase 4.431 -> 4.411 +20:18:43 step 55480 loss=3.5054 lr=2.53e-04 tok=7.272B dt=1278 wait=98 +20:19:59 step 55500 loss=3.3762 lr=2.53e-04 tok=7.274B dt=1353 wait=98 +20:20:10 pushed ckpt_w26.pt,log_w26.txt +20:21:34 step 55520 loss=3.4153 lr=2.53e-04 tok=7.277B dt=1449 wait=99 +20:22:50 step 55540 loss=3.3967 lr=2.53e-04 tok=7.280B dt=1524 wait=99 +20:24:05 step 55560 loss=3.3843 lr=2.53e-04 tok=7.282B dt=1600 wait=100 +20:25:21 step 55580 loss=3.4805 lr=2.53e-04 tok=7.285B dt=1676 wait=100 +20:26:37 step 55600 loss=3.4158 lr=2.53e-04 tok=7.288B dt=1751 wait=100 +20:26:50 global r=82 rebase 4.399 -> 4.405 +20:28:06 step 55620 loss=3.4455 lr=2.52e-04 tok=7.290B dt=1840 wait=101 +20:29:21 step 55640 loss=3.4083 lr=2.52e-04 tok=7.293B dt=1916 wait=101 +20:30:13 pushed ckpt_w26.pt,log_w26.txt +20:30:57 step 55660 loss=3.3863 lr=2.52e-04 tok=7.295B dt=2012 wait=102 +20:32:13 step 55680 loss=3.4490 lr=2.52e-04 tok=7.298B dt=2088 wait=102 +20:33:29 step 55700 loss=3.4263 lr=2.52e-04 tok=7.301B dt=2163 wait=102 +20:34:45 step 55720 loss=3.4006 lr=2.52e-04 tok=7.303B dt=2239 wait=103 +20:36:00 step 55740 loss=3.4114 lr=2.52e-04 tok=7.306B dt=2315 wait=103 +20:37:16 step 55760 loss=3.3875 lr=2.52e-04 tok=7.309B dt=2391 wait=104 +20:38:29 global r=83 rebase 4.406 -> 4.432 +20:38:48 step 55780 loss=3.4248 lr=2.52e-04 tok=7.311B dt=2483 wait=104 +20:40:04 step 55800 loss=3.4103 lr=2.52e-04 tok=7.314B dt=2558 wait=104 +20:40:16 pushed ckpt_w26.pt,log_w26.txt +20:41:39 step 55820 loss=3.4385 lr=2.52e-04 tok=7.316B dt=2653 wait=105 +20:42:54 step 55840 loss=3.4094 lr=2.52e-04 tok=7.319B dt=2729 wait=105 +20:44:10 step 55860 loss=3.4312 lr=2.52e-04 tok=7.322B dt=2805 wait=106 +20:45:26 step 55880 loss=3.4381 lr=2.52e-04 tok=7.324B dt=2881 wait=106 +20:46:41 step 55900 loss=3.3529 lr=2.52e-04 tok=7.327B dt=2956 wait=106 +20:46:53 global r=84 rebase 4.392 -> 4.414 +20:48:09 step 55920 loss=3.4139 lr=2.52e-04 tok=7.330B dt=3043 wait=107 +20:49:25 step 55940 loss=3.4510 lr=2.52e-04 tok=7.332B dt=3119 wait=107 +20:50:15 pushed ckpt_w26.pt,log_w26.txt +20:50:58 step 55960 loss=3.4183 lr=2.52e-04 tok=7.335B dt=3212 wait=108 +20:52:13 step 55980 loss=3.4061 lr=2.52e-04 tok=7.337B dt=3288 wait=108 +20:53:29 step 56000 loss=3.4257 lr=2.52e-04 tok=7.340B dt=3364 wait=108 +20:54:45 step 56020 loss=3.3905 lr=2.52e-04 tok=7.343B dt=3439 wait=109 +20:56:01 step 56040 loss=3.4208 lr=2.52e-04 tok=7.345B dt=3515 wait=109 +20:57:17 step 56060 loss=3.3796 lr=2.52e-04 tok=7.348B dt=3591 wait=110 +20:58:24 global r=85 rebase 4.395 -> 4.454 +20:58:43 step 56080 loss=3.4099 lr=2.52e-04 tok=7.351B dt=3678 wait=110 +20:59:59 step 56100 loss=3.3809 lr=2.52e-04 tok=7.353B dt=3753 wait=110 +21:00:14 pushed ckpt_w26.pt,log_w26.txt +21:01:37 step 56120 loss=3.4427 lr=2.52e-04 tok=7.356B dt=3851 wait=111 +21:02:53 step 56140 loss=3.4529 lr=2.52e-04 tok=7.358B dt=3927 wait=111 +21:04:08 step 56160 loss=3.4623 lr=2.52e-04 tok=7.361B dt=4003 wait=112 +21:05:24 step 56180 loss=3.4937 lr=2.52e-04 tok=7.364B dt=4079 wait=112 +21:06:40 step 56200 loss=3.3392 lr=2.52e-04 tok=7.366B dt=4154 wait=112 +21:06:55 global r=86 rebase 4.406 -> 4.453 +21:08:10 step 56220 loss=3.4071 lr=2.52e-04 tok=7.369B dt=4245 wait=113 +21:09:26 step 56240 loss=3.4421 lr=2.51e-04 tok=7.371B dt=4321 wait=113 +21:10:16 pushed ckpt_w26.pt,log_w26.txt +21:10:58 step 56260 loss=3.4190 lr=2.51e-04 tok=7.374B dt=4413 wait=114 +21:12:14 step 56280 loss=3.4726 lr=2.51e-04 tok=7.377B dt=4489 wait=114 +21:13:29 step 56300 loss=3.4237 lr=2.51e-04 tok=7.379B dt=4564 wait=114 +21:14:45 step 56320 loss=3.3664 lr=2.51e-04 tok=7.382B dt=4640 wait=115 +21:16:01 step 56340 loss=3.4396 lr=2.51e-04 tok=7.385B dt=4716 wait=115 +21:16:51 global r=87 rebase 4.412 -> 4.483 +21:17:29 step 56360 loss=3.4749 lr=2.51e-04 tok=7.387B dt=4804 wait=116 +21:18:45 step 56380 loss=3.4451 lr=2.51e-04 tok=7.390B dt=4880 wait=116 +21:20:01 step 56400 loss=3.4626 lr=2.51e-04 tok=7.392B dt=4955 wait=117 +21:20:13 pushed ckpt_w26.pt,log_w26.txt +21:21:40 step 56420 loss=3.4533 lr=2.51e-04 tok=7.395B dt=5055 wait=117 +21:22:56 step 56440 loss=3.4047 lr=2.51e-04 tok=7.398B dt=5131 wait=117 +21:24:12 step 56460 loss=3.4834 lr=2.51e-04 tok=7.400B dt=5206 wait=118 +21:25:27 step 56480 loss=3.3589 lr=2.51e-04 tok=7.403B dt=5282 wait=118 +21:26:43 step 56500 loss=3.4248 lr=2.51e-04 tok=7.406B dt=5358 wait=119 +21:26:57 global r=88 rebase 4.404 -> 4.447 +21:28:13 step 56520 loss=3.4678 lr=2.51e-04 tok=7.408B dt=5448 wait=119 +21:29:29 step 56540 loss=3.4238 lr=2.51e-04 tok=7.411B dt=5523 wait=119 +21:30:21 pushed ckpt_w26.pt,log_w26.txt +21:31:07 step 56560 loss=3.4380 lr=2.51e-04 tok=7.413B dt=5621 wait=120 +21:32:23 step 56580 loss=3.4801 lr=2.51e-04 tok=7.416B dt=5697 wait=120 +21:33:38 step 56600 loss=3.4649 lr=2.51e-04 tok=7.419B dt=5773 wait=121 +21:34:54 step 56620 loss=3.3985 lr=2.51e-04 tok=7.421B dt=5848 wait=121 +21:36:10 step 56640 loss=3.3943 lr=2.51e-04 tok=7.424B dt=5924 wait=121 +21:37:25 step 56660 loss=3.3797 lr=2.51e-04 tok=7.427B dt=6000 wait=122 +21:38:35 global r=89 rebase 4.419 -> 4.523 +21:38:54 step 56680 loss=3.4614 lr=2.51e-04 tok=7.429B dt=6089 wait=122 +21:40:10 step 56700 loss=3.4969 lr=2.51e-04 tok=7.432B dt=6164 wait=123 +21:40:25 pushed ckpt_w26.pt,log_w26.txt +21:41:47 step 56720 loss=3.4970 lr=2.51e-04 tok=7.434B dt=6262 wait=123 +21:43:03 step 56740 loss=3.4388 lr=2.51e-04 tok=7.437B dt=6338 wait=123 +21:44:19 step 56760 loss=3.4641 lr=2.51e-04 tok=7.440B dt=6414 wait=124 +21:45:35 step 56780 loss=3.4640 lr=2.51e-04 tok=7.442B dt=6489 wait=124 +21:46:50 step 56800 loss=3.3666 lr=2.51e-04 tok=7.445B dt=6565 wait=124 +21:47:04 global r=90 rebase 4.422 -> 4.503 +21:48:20 step 56820 loss=3.5354 lr=2.51e-04 tok=7.448B dt=6655 wait=125 +21:49:36 step 56840 loss=3.5507 lr=2.50e-04 tok=7.450B dt=6731 wait=125 +21:50:29 pushed ckpt_w26.pt,log_w26.txt +21:51:13 step 56860 loss=3.4864 lr=2.50e-04 tok=7.453B dt=6828 wait=126 +21:52:29 step 56880 loss=3.4907 lr=2.50e-04 tok=7.455B dt=6904 wait=126 +21:53:44 step 56900 loss=3.4431 lr=2.50e-04 tok=7.458B dt=6979 wait=126 +21:55:00 step 56920 loss=3.4033 lr=2.50e-04 tok=7.461B dt=7055 wait=127 +21:56:16 step 56940 loss=3.4691 lr=2.50e-04 tok=7.463B dt=7131 wait=127 +21:57:05 global r=91 rebase 4.427 -> 4.544 +21:57:43 step 56960 loss=3.4932 lr=2.50e-04 tok=7.466B dt=7217 wait=128 +21:58:59 step 56980 loss=3.4837 lr=2.50e-04 tok=7.468B dt=7294 wait=128 +22:00:15 step 57000 loss=3.5033 lr=2.50e-04 tok=7.471B dt=7369 wait=128 +22:00:28 pushed ckpt_w26.pt,log_w26.txt +22:01:52 step 57020 loss=3.5205 lr=2.50e-04 tok=7.474B dt=7467 wait=129 +22:03:08 step 57040 loss=3.4446 lr=2.50e-04 tok=7.476B dt=7542 wait=129 +22:04:24 step 57060 loss=3.5459 lr=2.50e-04 tok=7.479B dt=7618 wait=130 +22:05:39 step 57080 loss=3.3651 lr=2.50e-04 tok=7.482B dt=7694 wait=130 +22:06:55 step 57100 loss=3.4190 lr=2.50e-04 tok=7.484B dt=7770 wait=130 +22:07:11 global r=92 rebase 4.436 -> 4.512 +22:08:27 step 57120 loss=3.5077 lr=2.50e-04 tok=7.487B dt=7862 wait=131 +22:09:43 step 57140 loss=3.4931 lr=2.50e-04 tok=7.489B dt=7938 wait=131 +22:10:35 pushed ckpt_w26.pt,log_w26.txt +22:11:18 step 57160 loss=3.4458 lr=2.50e-04 tok=7.492B dt=8032 wait=132 +22:12:33 step 57180 loss=3.5300 lr=2.50e-04 tok=7.495B dt=8108 wait=132 +22:13:49 step 57200 loss=3.5182 lr=2.50e-04 tok=7.497B dt=8184 wait=132 +22:15:04 step 57220 loss=3.4128 lr=2.50e-04 tok=7.500B dt=8259 wait=133 +22:16:20 step 57240 loss=3.4401 lr=2.50e-04 tok=7.503B dt=8335 wait=133 +22:17:12 global r=93 rebase 4.429 -> 4.549 +22:17:50 step 57260 loss=3.4714 lr=2.50e-04 tok=7.505B dt=8425 wait=134 +22:19:06 step 57280 loss=3.4641 lr=2.50e-04 tok=7.508B dt=8500 wait=134 +22:20:21 step 57300 loss=3.5641 lr=2.50e-04 tok=7.510B dt=8576 wait=134 +22:20:34 pushed ckpt_w26.pt,log_w26.txt +22:21:58 step 57320 loss=3.5505 lr=2.50e-04 tok=7.513B dt=8673 wait=135 +22:23:14 step 57340 loss=3.4225 lr=2.50e-04 tok=7.516B dt=8748 wait=135 +22:24:30 step 57360 loss=3.5292 lr=2.50e-04 tok=7.518B dt=8824 wait=136 +22:25:45 step 57380 loss=3.4146 lr=2.50e-04 tok=7.521B dt=8900 wait=136 +22:27:01 step 57400 loss=3.4524 lr=2.50e-04 tok=7.524B dt=8976 wait=136 +22:27:23 global r=94 rebase 4.443 -> 4.584 +22:28:38 step 57420 loss=3.5272 lr=2.50e-04 tok=7.526B dt=9073 wait=137 +22:29:54 step 57440 loss=3.5006 lr=2.50e-04 tok=7.529B dt=9149 wait=137 +22:30:44 pushed ckpt_w26.pt,log_w26.txt +22:31:28 step 57460 loss=3.5033 lr=2.49e-04 tok=7.531B dt=9242 wait=138 +22:32:44 step 57480 loss=3.5906 lr=2.49e-04 tok=7.534B dt=9318 wait=138 +22:33:59 step 57500 loss=3.5489 lr=2.49e-04 tok=7.537B dt=9394 wait=138 +22:35:15 step 57520 loss=3.4738 lr=2.49e-04 tok=7.539B dt=9470 wait=139 +22:36:31 step 57540 loss=3.4833 lr=2.49e-04 tok=7.542B dt=9546 wait=139 +22:37:47 step 57560 loss=3.4465 lr=2.49e-04 tok=7.545B dt=9621 wait=140 +22:38:57 global r=95 rebase 4.447 -> 4.628 +22:39:16 step 57580 loss=3.5253 lr=2.49e-04 tok=7.547B dt=9711 wait=140 +22:40:31 step 57600 loss=3.5510 lr=2.49e-04 tok=7.550B dt=9786 wait=140 +22:40:45 pushed ckpt_w26.pt,log_w26.txt +22:42:04 step 57620 loss=3.4968 lr=2.49e-04 tok=7.552B dt=9879 wait=141 +22:43:21 step 57640 loss=3.5321 lr=2.49e-04 tok=7.555B dt=9955 wait=141 +22:44:36 step 57660 loss=3.5611 lr=2.49e-04 tok=7.558B dt=10031 wait=142 +22:45:52 step 57680 loss=3.5003 lr=2.49e-04 tok=7.560B dt=10107 wait=142 +22:47:08 step 57700 loss=3.4349 lr=2.49e-04 tok=7.563B dt=10182 wait=142 +22:47:20 global r=96 rebase 4.485 -> 4.633 +22:48:36 step 57720 loss=3.5635 lr=2.49e-04 tok=7.565B dt=10271 wait=143 +22:49:52 step 57740 loss=3.5603 lr=2.49e-04 tok=7.568B dt=10347 wait=143 diff --git a/log_w3.txt b/log_w3.txt new file mode 100644 index 0000000000000000000000000000000000000000..ca13eeac9c6c0e1a78e1bee1821f19dc9f243f57 --- /dev/null +++ b/log_w3.txt @@ -0,0 +1,360 @@ +launch +17:18:24 name=w3 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 3/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:18:29 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:18:38 params 1.08B +17:18:43 resumed step=27600 seen=3.62B round=14 +17:18:43 stream salt=27600 +17:21:22 start: accum=4, 131072 tokens per optimiser step +17:24:34 step 27620 loss=3.7216 lr=2.88e-04 tok=3.620B dt=192 +17:26:03 step 27640 loss=3.8961 lr=2.88e-04 tok=3.623B dt=281 +17:27:25 step 27660 loss=3.8146 lr=2.88e-04 tok=3.625B dt=363 +17:28:41 step 27680 loss=3.8579 lr=2.88e-04 tok=3.628B dt=439 +17:29:57 step 27700 loss=3.7128 lr=2.88e-04 tok=3.631B dt=515 +17:31:13 step 27720 loss=3.7755 lr=2.88e-04 tok=3.633B dt=591 +17:32:29 step 27740 loss=3.7935 lr=2.88e-04 tok=3.636B dt=667 +17:33:45 step 27760 loss=3.7884 lr=2.88e-04 tok=3.639B dt=743 +17:35:01 step 27780 loss=3.7078 lr=2.88e-04 tok=3.641B dt=819 +17:36:17 step 27800 loss=3.8033 lr=2.88e-04 tok=3.644B dt=895 +17:37:33 step 27820 loss=3.7608 lr=2.88e-04 tok=3.646B dt=971 +17:38:49 step 27840 loss=3.7491 lr=2.88e-04 tok=3.649B dt=1047 +17:40:05 step 27860 loss=3.7847 lr=2.88e-04 tok=3.652B dt=1122 +17:41:21 step 27880 loss=3.7310 lr=2.88e-04 tok=3.654B dt=1198 +17:42:37 step 27900 loss=3.8228 lr=2.88e-04 tok=3.657B dt=1274 +17:43:53 step 27920 loss=3.8696 lr=2.88e-04 tok=3.660B dt=1350 +17:45:09 step 27940 loss=3.8625 lr=2.88e-04 tok=3.662B dt=1426 +17:46:25 step 27960 loss=3.7981 lr=2.88e-04 tok=3.665B dt=1502 +17:47:41 step 27980 loss=3.8854 lr=2.88e-04 tok=3.667B dt=1578 +17:48:57 step 28000 loss=3.8410 lr=2.88e-04 tok=3.670B dt=1654 +17:50:13 step 28020 loss=3.8138 lr=2.88e-04 tok=3.673B dt=1730 +17:51:28 step 28040 loss=3.7671 lr=2.88e-04 tok=3.675B dt=1806 +17:52:45 step 28060 loss=3.8025 lr=2.88e-04 tok=3.678B dt=1882 +17:54:00 step 28080 loss=3.7396 lr=2.88e-04 tok=3.681B dt=1958 +17:55:16 step 28100 loss=3.7848 lr=2.88e-04 tok=3.683B dt=2034 +17:56:32 step 28120 loss=3.7322 lr=2.87e-04 tok=3.686B dt=2110 +17:57:48 step 28140 loss=3.7397 lr=2.87e-04 tok=3.688B dt=2186 +17:59:04 step 28160 loss=3.8163 lr=2.87e-04 tok=3.691B dt=2262 +18:00:20 step 28180 loss=3.6697 lr=2.87e-04 tok=3.694B dt=2337 +18:01:36 step 28200 loss=3.8043 lr=2.87e-04 tok=3.696B dt=2413 +18:01:50 pushed ckpt_w3.pt +18:01:51 pushed log_w3.txt +18:03:16 step 28220 loss=3.8112 lr=2.87e-04 tok=3.699B dt=2514 +18:04:32 step 28240 loss=3.7767 lr=2.87e-04 tok=3.701B dt=2590 +18:05:48 step 28260 loss=3.7090 lr=2.87e-04 tok=3.704B dt=2666 +18:07:04 step 28280 loss=3.7067 lr=2.87e-04 tok=3.707B dt=2742 +18:08:20 step 28300 loss=3.7486 lr=2.87e-04 tok=3.709B dt=2818 +18:09:36 step 28320 loss=3.8160 lr=2.87e-04 tok=3.712B dt=2894 +18:10:52 step 28340 loss=3.9369 lr=2.87e-04 tok=3.715B dt=2970 +18:12:09 step 28360 loss=3.8608 lr=2.87e-04 tok=3.717B dt=3046 +18:13:24 step 28380 loss=3.8066 lr=2.87e-04 tok=3.720B dt=3122 +18:14:41 step 28400 loss=3.7734 lr=2.87e-04 tok=3.722B dt=3198 +18:15:57 step 28420 loss=3.8046 lr=2.87e-04 tok=3.725B dt=3274 +18:17:13 step 28440 loss=3.7787 lr=2.87e-04 tok=3.728B dt=3350 +18:18:29 step 28460 loss=3.7320 lr=2.87e-04 tok=3.730B dt=3427 +18:19:45 step 28480 loss=3.8684 lr=2.87e-04 tok=3.733B dt=3502 +18:21:01 step 28500 loss=3.7246 lr=2.87e-04 tok=3.736B dt=3578 +18:22:17 step 28520 loss=3.8415 lr=2.87e-04 tok=3.738B dt=3655 +18:23:33 step 28540 loss=3.7494 lr=2.87e-04 tok=3.741B dt=3731 +18:24:51 step 28560 loss=3.7884 lr=2.87e-04 tok=3.743B dt=3808 +18:26:07 step 28580 loss=3.8667 lr=2.87e-04 tok=3.746B dt=3885 +18:27:23 step 28600 loss=3.7681 lr=2.87e-04 tok=3.749B dt=3961 +18:28:39 step 28620 loss=3.7675 lr=2.87e-04 tok=3.751B dt=4037 +18:29:55 step 28640 loss=3.9060 lr=2.87e-04 tok=3.754B dt=4113 +18:31:11 step 28660 loss=3.8257 lr=2.87e-04 tok=3.757B dt=4189 +18:32:27 step 28680 loss=3.7748 lr=2.87e-04 tok=3.759B dt=4264 +18:33:43 step 28700 loss=3.8246 lr=2.87e-04 tok=3.762B dt=4340 +18:34:58 step 28720 loss=3.7739 lr=2.87e-04 tok=3.764B dt=4416 +18:36:14 step 28740 loss=3.8199 lr=2.87e-04 tok=3.767B dt=4492 +18:37:30 step 28760 loss=3.7683 lr=2.87e-04 tok=3.770B dt=4568 +18:38:46 step 28780 loss=3.7397 lr=2.87e-04 tok=3.772B dt=4644 +18:40:02 step 28800 loss=3.8675 lr=2.87e-04 tok=3.775B dt=4720 +18:40:14 pushed ckpt_w3.pt +18:40:15 pushed log_w3.txt +18:40:31 pulled global r=15 4.583 -> 4.460 +18:41:53 step 28820 loss=3.6585 lr=2.87e-04 tok=3.777B dt=4831 +18:43:09 step 28840 loss=3.6121 lr=2.87e-04 tok=3.780B dt=4906 +18:44:24 step 28860 loss=3.6670 lr=2.87e-04 tok=3.783B dt=4982 +18:45:40 step 28880 loss=3.7700 lr=2.87e-04 tok=3.785B dt=5058 +18:46:57 step 28900 loss=3.8450 lr=2.87e-04 tok=3.788B dt=5134 +18:48:13 step 28920 loss=3.7629 lr=2.87e-04 tok=3.791B dt=5210 +18:49:29 step 28940 loss=3.6989 lr=2.87e-04 tok=3.793B dt=5286 +18:50:45 step 28960 loss=3.8183 lr=2.87e-04 tok=3.796B dt=5362 +18:52:01 step 28980 loss=3.8912 lr=2.87e-04 tok=3.798B dt=5438 +18:53:17 step 29000 loss=3.8343 lr=2.87e-04 tok=3.801B dt=5515 +18:54:33 step 29020 loss=3.8174 lr=2.87e-04 tok=3.804B dt=5591 +18:55:50 step 29040 loss=3.8476 lr=2.87e-04 tok=3.806B dt=5668 +18:57:06 step 29060 loss=3.7249 lr=2.87e-04 tok=3.809B dt=5744 +18:58:22 step 29080 loss=3.7878 lr=2.87e-04 tok=3.812B dt=5820 +18:59:38 step 29100 loss=3.7673 lr=2.87e-04 tok=3.814B dt=5896 +19:00:54 step 29120 loss=3.6813 lr=2.87e-04 tok=3.817B dt=5972 +19:02:10 step 29140 loss=3.7403 lr=2.87e-04 tok=3.819B dt=6048 +19:03:26 step 29160 loss=3.8254 lr=2.87e-04 tok=3.822B dt=6124 +19:04:42 step 29180 loss=3.6970 lr=2.87e-04 tok=3.825B dt=6200 +19:05:58 step 29200 loss=3.7531 lr=2.87e-04 tok=3.827B dt=6276 +19:07:14 step 29220 loss=3.7063 lr=2.86e-04 tok=3.830B dt=6352 +19:08:30 step 29240 loss=3.8614 lr=2.86e-04 tok=3.833B dt=6428 +19:09:46 step 29260 loss=3.8204 lr=2.86e-04 tok=3.835B dt=6504 +19:11:02 step 29280 loss=3.7746 lr=2.86e-04 tok=3.838B dt=6580 +19:12:18 step 29300 loss=3.6480 lr=2.86e-04 tok=3.840B dt=6656 +19:13:34 step 29320 loss=3.7622 lr=2.86e-04 tok=3.843B dt=6732 +19:14:50 step 29340 loss=3.6893 lr=2.86e-04 tok=3.846B dt=6808 +19:16:06 step 29360 loss=3.7540 lr=2.86e-04 tok=3.848B dt=6884 +19:17:23 step 29380 loss=3.8075 lr=2.86e-04 tok=3.851B dt=6960 +19:18:38 step 29400 loss=3.7379 lr=2.86e-04 tok=3.854B dt=7036 +19:18:57 pushed ckpt_w3.pt +19:18:58 pushed log_w3.txt +19:20:21 step 29420 loss=3.7567 lr=2.86e-04 tok=3.856B dt=7138 +19:21:38 step 29440 loss=3.7202 lr=2.86e-04 tok=3.859B dt=7216 +19:22:55 step 29460 loss=3.8632 lr=2.86e-04 tok=3.861B dt=7292 +19:24:10 step 29480 loss=3.7861 lr=2.86e-04 tok=3.864B dt=7368 +19:25:26 step 29500 loss=3.7876 lr=2.86e-04 tok=3.867B dt=7444 +19:26:42 step 29520 loss=3.8748 lr=2.86e-04 tok=3.869B dt=7520 +19:27:58 step 29540 loss=3.6829 lr=2.86e-04 tok=3.872B dt=7596 +19:29:14 step 29560 loss=3.7512 lr=2.86e-04 tok=3.874B dt=7672 +19:30:30 step 29580 loss=3.7017 lr=2.86e-04 tok=3.877B dt=7747 +19:31:45 step 29600 loss=3.6975 lr=2.86e-04 tok=3.880B dt=7823 +19:33:01 step 29620 loss=3.7074 lr=2.86e-04 tok=3.882B dt=7899 +19:34:17 step 29640 loss=3.6287 lr=2.86e-04 tok=3.885B dt=7975 +19:35:33 step 29660 loss=3.7775 lr=2.86e-04 tok=3.888B dt=8051 +19:36:49 step 29680 loss=3.7777 lr=2.86e-04 tok=3.890B dt=8127 +19:38:05 step 29700 loss=3.7443 lr=2.86e-04 tok=3.893B dt=8202 +19:39:21 step 29720 loss=3.8119 lr=2.86e-04 tok=3.895B dt=8278 +19:40:37 step 29740 loss=3.7181 lr=2.86e-04 tok=3.898B dt=8354 +19:41:53 step 29760 loss=3.7526 lr=2.86e-04 tok=3.901B dt=8430 +19:43:08 step 29780 loss=3.9423 lr=2.86e-04 tok=3.903B dt=8506 +19:44:24 step 29800 loss=3.8073 lr=2.86e-04 tok=3.906B dt=8582 +19:45:40 step 29820 loss=3.7854 lr=2.86e-04 tok=3.909B dt=8658 +19:46:56 step 29840 loss=3.7711 lr=2.86e-04 tok=3.911B dt=8734 +19:48:12 step 29860 loss=3.7436 lr=2.86e-04 tok=3.914B dt=8810 +19:49:28 step 29880 loss=3.7461 lr=2.86e-04 tok=3.916B dt=8886 +19:50:44 step 29900 loss=3.7652 lr=2.86e-04 tok=3.919B dt=8962 +19:52:00 step 29920 loss=3.8174 lr=2.86e-04 tok=3.922B dt=9037 +19:53:16 step 29940 loss=3.7870 lr=2.86e-04 tok=3.924B dt=9113 +19:54:31 step 29960 loss=3.7520 lr=2.86e-04 tok=3.927B dt=9189 +19:55:47 step 29980 loss=3.7826 lr=2.86e-04 tok=3.930B dt=9265 +19:57:03 step 30000 loss=3.7759 lr=2.86e-04 tok=3.932B dt=9341 +19:57:15 pushed ckpt_w3.pt +19:57:16 pushed log_w3.txt +19:58:37 step 30020 loss=3.8046 lr=2.86e-04 tok=3.935B dt=9435 +19:59:53 step 30040 loss=3.7193 lr=2.86e-04 tok=3.937B dt=9511 +20:01:09 step 30060 loss=3.7329 lr=2.86e-04 tok=3.940B dt=9586 +20:02:25 step 30080 loss=3.8602 lr=2.86e-04 tok=3.943B dt=9662 +20:03:41 step 30100 loss=3.7473 lr=2.86e-04 tok=3.945B dt=9738 +20:04:57 step 30120 loss=3.7576 lr=2.86e-04 tok=3.948B dt=9814 +20:06:13 step 30140 loss=3.6629 lr=2.86e-04 tok=3.951B dt=9890 +20:07:29 step 30160 loss=3.7908 lr=2.86e-04 tok=3.953B dt=9966 +20:08:45 step 30180 loss=3.8496 lr=2.86e-04 tok=3.956B dt=10042 +20:10:01 step 30200 loss=3.7121 lr=2.86e-04 tok=3.958B dt=10118 +20:11:17 step 30220 loss=3.8098 lr=2.86e-04 tok=3.961B dt=10194 +20:12:33 step 30240 loss=3.6384 lr=2.86e-04 tok=3.964B dt=10270 +20:13:49 step 30260 loss=3.6752 lr=2.86e-04 tok=3.966B dt=10346 +20:15:05 step 30280 loss=3.6801 lr=2.85e-04 tok=3.969B dt=10422 +20:16:21 step 30300 loss=3.7513 lr=2.85e-04 tok=3.971B dt=10498 +20:17:36 step 30320 loss=3.7168 lr=2.85e-04 tok=3.974B dt=10574 +20:18:52 step 30340 loss=3.6780 lr=2.85e-04 tok=3.977B dt=10650 +20:20:08 step 30360 loss=3.7890 lr=2.85e-04 tok=3.979B dt=10726 +20:21:24 step 30380 loss=3.8143 lr=2.85e-04 tok=3.982B dt=10802 +20:22:40 step 30400 loss=3.8423 lr=2.85e-04 tok=3.985B dt=10878 +20:23:56 step 30420 loss=3.6728 lr=2.85e-04 tok=3.987B dt=10953 +20:25:12 step 30440 loss=3.7742 lr=2.85e-04 tok=3.990B dt=11029 +20:26:27 step 30460 loss=3.7395 lr=2.85e-04 tok=3.992B dt=11105 +20:27:43 step 30480 loss=3.8152 lr=2.85e-04 tok=3.995B dt=11181 +20:28:59 step 30500 loss=3.7888 lr=2.85e-04 tok=3.998B dt=11257 +20:30:15 step 30520 loss=3.8337 lr=2.85e-04 tok=4.000B dt=11333 +20:31:31 step 30540 loss=3.9415 lr=2.85e-04 tok=4.003B dt=11409 +20:32:47 step 30560 loss=3.7507 lr=2.85e-04 tok=4.006B dt=11484 +20:34:03 step 30580 loss=3.7290 lr=2.85e-04 tok=4.008B dt=11560 +20:35:19 step 30600 loss=3.7205 lr=2.85e-04 tok=4.011B dt=11636 +20:35:31 pushed ckpt_w3.pt +20:35:31 pushed log_w3.txt +20:36:53 step 30620 loss=3.8815 lr=2.85e-04 tok=4.013B dt=11730 +20:38:09 step 30640 loss=3.8332 lr=2.85e-04 tok=4.016B dt=11806 +20:39:24 step 30660 loss=3.8021 lr=2.85e-04 tok=4.019B dt=11882 +20:40:40 step 30680 loss=3.8157 lr=2.85e-04 tok=4.021B dt=11958 +20:41:56 step 30700 loss=3.8244 lr=2.85e-04 tok=4.024B dt=12034 +20:43:12 step 30720 loss=3.7607 lr=2.85e-04 tok=4.027B dt=12110 +20:44:28 step 30740 loss=3.7787 lr=2.85e-04 tok=4.029B dt=12185 +20:45:43 step 30760 loss=3.7278 lr=2.85e-04 tok=4.032B dt=12261 +20:46:59 step 30780 loss=3.7233 lr=2.85e-04 tok=4.034B dt=12337 +20:48:15 step 30800 loss=3.6817 lr=2.85e-04 tok=4.037B dt=12413 +20:49:31 step 30820 loss=3.7899 lr=2.85e-04 tok=4.040B dt=12489 +20:50:47 step 30840 loss=3.7029 lr=2.85e-04 tok=4.042B dt=12564 +20:52:03 step 30860 loss=3.6666 lr=2.85e-04 tok=4.045B dt=12640 +20:53:19 step 30880 loss=3.8372 lr=2.85e-04 tok=4.048B dt=12716 +20:54:34 step 30900 loss=3.7112 lr=2.85e-04 tok=4.050B dt=12792 +20:55:50 step 30920 loss=3.7736 lr=2.85e-04 tok=4.053B dt=12868 +20:57:06 step 30940 loss=3.6843 lr=2.85e-04 tok=4.055B dt=12944 +20:58:22 step 30960 loss=3.7855 lr=2.85e-04 tok=4.058B dt=13020 +20:59:38 step 30980 loss=3.7575 lr=2.85e-04 tok=4.061B dt=13096 +21:00:54 step 31000 loss=3.7610 lr=2.85e-04 tok=4.063B dt=13172 +21:02:10 step 31020 loss=3.7894 lr=2.85e-04 tok=4.066B dt=13248 +21:03:26 step 31040 loss=3.8262 lr=2.85e-04 tok=4.068B dt=13324 +21:04:42 step 31060 loss=3.7331 lr=2.85e-04 tok=4.071B dt=13400 +21:05:58 step 31080 loss=3.8451 lr=2.85e-04 tok=4.074B dt=13476 +21:07:14 step 31100 loss=3.7857 lr=2.85e-04 tok=4.076B dt=13552 +21:08:30 step 31120 loss=3.8943 lr=2.85e-04 tok=4.079B dt=13628 +21:09:46 step 31140 loss=3.7352 lr=2.85e-04 tok=4.082B dt=13704 +21:11:02 step 31160 loss=3.8608 lr=2.85e-04 tok=4.084B dt=13780 +21:12:18 step 31180 loss=3.8175 lr=2.85e-04 tok=4.087B dt=13856 +21:13:34 step 31200 loss=3.8169 lr=2.85e-04 tok=4.089B dt=13932 +21:13:51 pushed ckpt_w3.pt +21:13:51 pushed log_w3.txt +21:15:14 step 31220 loss=3.8527 lr=2.85e-04 tok=4.092B dt=14032 +21:16:30 step 31240 loss=3.7517 lr=2.85e-04 tok=4.095B dt=14108 +21:17:46 step 31260 loss=3.8202 lr=2.85e-04 tok=4.097B dt=14184 +21:19:02 step 31280 loss=3.6417 lr=2.85e-04 tok=4.100B dt=14260 +21:20:18 step 31300 loss=3.7456 lr=2.85e-04 tok=4.103B dt=14335 +21:21:34 step 31320 loss=3.8214 lr=2.84e-04 tok=4.105B dt=14411 +21:22:50 step 31340 loss=3.7960 lr=2.84e-04 tok=4.108B dt=14487 +21:24:06 step 31360 loss=3.7371 lr=2.84e-04 tok=4.110B dt=14563 +21:25:21 step 31380 loss=3.7521 lr=2.84e-04 tok=4.113B dt=14639 +21:26:37 step 31400 loss=3.7190 lr=2.84e-04 tok=4.116B dt=14715 +21:27:53 step 31420 loss=3.7249 lr=2.84e-04 tok=4.118B dt=14791 +21:29:09 step 31440 loss=3.7513 lr=2.84e-04 tok=4.121B dt=14867 +21:30:25 step 31460 loss=3.8124 lr=2.84e-04 tok=4.124B dt=14943 +21:31:41 step 31480 loss=3.7960 lr=2.84e-04 tok=4.126B dt=15018 +21:32:57 step 31500 loss=3.8149 lr=2.84e-04 tok=4.129B dt=15094 +21:34:12 step 31520 loss=3.8001 lr=2.84e-04 tok=4.131B dt=15170 +21:35:28 step 31540 loss=3.7225 lr=2.84e-04 tok=4.134B dt=15246 +21:36:44 step 31560 loss=3.7793 lr=2.84e-04 tok=4.137B dt=15322 +21:38:00 step 31580 loss=3.7868 lr=2.84e-04 tok=4.139B dt=15398 +21:39:18 step 31600 loss=3.7698 lr=2.84e-04 tok=4.142B dt=15476 +21:40:34 step 31620 loss=3.7763 lr=2.84e-04 tok=4.144B dt=15552 +21:41:50 step 31640 loss=3.8671 lr=2.84e-04 tok=4.147B dt=15628 +21:43:06 step 31660 loss=3.6924 lr=2.84e-04 tok=4.150B dt=15704 +21:44:22 step 31680 loss=3.7680 lr=2.84e-04 tok=4.152B dt=15779 +21:45:37 step 31700 loss=3.6776 lr=2.84e-04 tok=4.155B dt=15855 +21:46:53 step 31720 loss=3.7737 lr=2.84e-04 tok=4.158B dt=15931 +21:48:09 step 31740 loss=3.8659 lr=2.84e-04 tok=4.160B dt=16007 +21:49:25 step 31760 loss=3.8501 lr=2.84e-04 tok=4.163B dt=16083 +21:50:41 step 31780 loss=3.8103 lr=2.84e-04 tok=4.165B dt=16159 +21:51:57 step 31800 loss=3.7304 lr=2.84e-04 tok=4.168B dt=16235 +21:52:12 pushed ckpt_w3.pt +21:52:12 pushed log_w3.txt +21:52:23 pulled global r=16 4.567 -> 4.450 +21:53:44 step 31820 loss=3.6295 lr=2.84e-04 tok=4.171B dt=16342 +21:55:00 step 31840 loss=3.6124 lr=2.84e-04 tok=4.173B dt=16418 +21:56:16 step 31860 loss=3.6600 lr=2.84e-04 tok=4.176B dt=16493 +21:57:32 step 31880 loss=3.7190 lr=2.84e-04 tok=4.179B dt=16570 +21:58:48 step 31900 loss=3.7881 lr=2.84e-04 tok=4.181B dt=16646 +22:00:04 step 31920 loss=3.7555 lr=2.84e-04 tok=4.184B dt=16722 +22:01:20 step 31940 loss=3.6787 lr=2.84e-04 tok=4.186B dt=16797 +22:02:36 step 31960 loss=3.7954 lr=2.84e-04 tok=4.189B dt=16873 +22:03:52 step 31980 loss=3.8521 lr=2.84e-04 tok=4.192B dt=16949 +22:05:08 step 32000 loss=3.8475 lr=2.84e-04 tok=4.194B dt=17025 +22:06:24 step 32020 loss=3.8125 lr=2.84e-04 tok=4.197B dt=17101 +22:07:40 step 32040 loss=3.7971 lr=2.84e-04 tok=4.200B dt=17177 +22:08:56 step 32060 loss=3.6904 lr=2.84e-04 tok=4.202B dt=17253 +22:10:12 step 32080 loss=3.7587 lr=2.84e-04 tok=4.205B dt=17329 +22:11:27 step 32100 loss=3.7671 lr=2.84e-04 tok=4.207B dt=17405 +22:12:43 step 32120 loss=3.6588 lr=2.84e-04 tok=4.210B dt=17481 +22:13:59 step 32140 loss=3.6660 lr=2.84e-04 tok=4.213B dt=17557 +22:15:15 step 32160 loss=3.7687 lr=2.84e-04 tok=4.215B dt=17633 +22:16:31 step 32180 loss=3.6588 lr=2.84e-04 tok=4.218B dt=17709 +22:17:47 step 32200 loss=3.7283 lr=2.84e-04 tok=4.221B dt=17785 +22:19:03 step 32220 loss=3.6764 lr=2.84e-04 tok=4.223B dt=17860 +22:20:19 step 32240 loss=3.8325 lr=2.84e-04 tok=4.226B dt=17936 +22:21:35 step 32260 loss=3.8254 lr=2.84e-04 tok=4.228B dt=18012 +22:22:50 step 32280 loss=3.7826 lr=2.84e-04 tok=4.231B dt=18088 +22:24:06 step 32300 loss=3.7182 lr=2.84e-04 tok=4.234B dt=18164 +22:25:22 step 32320 loss=3.7301 lr=2.83e-04 tok=4.236B dt=18240 +22:26:38 step 32340 loss=3.6489 lr=2.83e-04 tok=4.239B dt=18315 +22:27:54 step 32360 loss=3.7599 lr=2.83e-04 tok=4.241B dt=18391 +22:29:09 step 32380 loss=3.7977 lr=2.83e-04 tok=4.244B dt=18467 +22:30:25 step 32400 loss=3.7482 lr=2.83e-04 tok=4.247B dt=18543 +22:30:37 pushed ckpt_w3.pt +22:30:38 pushed log_w3.txt +22:32:00 step 32420 loss=3.7691 lr=2.83e-04 tok=4.249B dt=18638 +22:33:16 step 32440 loss=3.7467 lr=2.83e-04 tok=4.252B dt=18714 +22:34:32 step 32460 loss=3.8385 lr=2.83e-04 tok=4.255B dt=18790 +22:35:48 step 32480 loss=3.7518 lr=2.83e-04 tok=4.257B dt=18866 +22:37:04 step 32500 loss=3.7140 lr=2.83e-04 tok=4.260B dt=18942 +22:38:20 step 32520 loss=3.8247 lr=2.83e-04 tok=4.262B dt=19018 +22:39:36 step 32540 loss=3.6625 lr=2.83e-04 tok=4.265B dt=19093 +22:40:52 step 32560 loss=3.7510 lr=2.83e-04 tok=4.268B dt=19169 +22:42:08 step 32580 loss=3.7091 lr=2.83e-04 tok=4.270B dt=19245 +22:43:23 step 32600 loss=3.7450 lr=2.83e-04 tok=4.273B dt=19321 +22:44:40 step 32620 loss=3.7004 lr=2.83e-04 tok=4.276B dt=19397 +22:45:55 step 32640 loss=3.6398 lr=2.83e-04 tok=4.278B dt=19473 +22:47:12 step 32660 loss=3.7623 lr=2.83e-04 tok=4.281B dt=19549 +22:48:27 step 32680 loss=3.7524 lr=2.83e-04 tok=4.283B dt=19625 +22:49:43 step 32700 loss=3.6808 lr=2.83e-04 tok=4.286B dt=19701 +22:50:59 step 32720 loss=3.7497 lr=2.83e-04 tok=4.289B dt=19777 +22:52:15 step 32740 loss=3.7208 lr=2.83e-04 tok=4.291B dt=19853 +22:53:31 step 32760 loss=3.7089 lr=2.83e-04 tok=4.294B dt=19929 +22:54:47 step 32780 loss=3.8960 lr=2.83e-04 tok=4.297B dt=20005 +22:56:03 step 32800 loss=3.8171 lr=2.83e-04 tok=4.299B dt=20081 +22:57:19 step 32820 loss=3.7948 lr=2.83e-04 tok=4.302B dt=20157 +22:58:36 step 32840 loss=3.7652 lr=2.83e-04 tok=4.304B dt=20233 +22:59:52 step 32860 loss=3.7494 lr=2.83e-04 tok=4.307B dt=20310 +23:01:08 step 32880 loss=3.7029 lr=2.83e-04 tok=4.310B dt=20386 +23:02:24 step 32900 loss=3.7454 lr=2.83e-04 tok=4.312B dt=20462 +23:03:40 step 32920 loss=3.7882 lr=2.83e-04 tok=4.315B dt=20538 +23:04:56 step 32940 loss=3.7281 lr=2.83e-04 tok=4.318B dt=20614 +23:06:12 step 32960 loss=3.7251 lr=2.83e-04 tok=4.320B dt=20690 +23:07:28 step 32980 loss=3.7847 lr=2.83e-04 tok=4.323B dt=20766 +23:08:44 step 33000 loss=3.7493 lr=2.83e-04 tok=4.325B dt=20841 +23:08:56 pushed ckpt_w3.pt +23:08:56 pushed log_w3.txt +23:10:21 step 33020 loss=3.7825 lr=2.83e-04 tok=4.328B dt=20939 +23:11:37 step 33040 loss=3.7147 lr=2.83e-04 tok=4.331B dt=21015 +23:12:53 step 33060 loss=3.6961 lr=2.83e-04 tok=4.333B dt=21091 +23:14:09 step 33080 loss=3.8165 lr=2.83e-04 tok=4.336B dt=21167 +23:15:25 step 33100 loss=3.7149 lr=2.83e-04 tok=4.338B dt=21243 +23:16:41 step 33120 loss=3.7381 lr=2.83e-04 tok=4.341B dt=21319 +23:17:57 step 33140 loss=3.6300 lr=2.83e-04 tok=4.344B dt=21395 +23:19:13 step 33160 loss=3.7502 lr=2.83e-04 tok=4.346B dt=21471 +23:20:30 step 33180 loss=3.7870 lr=2.83e-04 tok=4.349B dt=21547 +23:21:46 step 33200 loss=3.7153 lr=2.83e-04 tok=4.352B dt=21623 +23:23:01 step 33220 loss=3.7474 lr=2.83e-04 tok=4.354B dt=21699 +23:24:17 step 33240 loss=3.6729 lr=2.83e-04 tok=4.357B dt=21775 +23:25:33 step 33260 loss=3.7015 lr=2.83e-04 tok=4.359B dt=21851 +23:26:49 step 33280 loss=3.6492 lr=2.82e-04 tok=4.362B dt=21927 +23:28:05 step 33300 loss=3.7200 lr=2.82e-04 tok=4.365B dt=22003 +23:29:21 step 33320 loss=3.6515 lr=2.82e-04 tok=4.367B dt=22079 +23:30:37 step 33340 loss=3.6157 lr=2.82e-04 tok=4.370B dt=22154 +23:31:53 step 33360 loss=3.7299 lr=2.82e-04 tok=4.373B dt=22230 +23:33:09 step 33380 loss=3.8255 lr=2.82e-04 tok=4.375B dt=22306 +23:34:24 step 33400 loss=3.8681 lr=2.82e-04 tok=4.378B dt=22382 +23:35:40 step 33420 loss=3.7021 lr=2.82e-04 tok=4.380B dt=22458 +23:36:56 step 33440 loss=3.7621 lr=2.82e-04 tok=4.383B dt=22533 +23:38:12 step 33460 loss=3.7228 lr=2.82e-04 tok=4.386B dt=22609 +23:39:28 step 33480 loss=3.8057 lr=2.82e-04 tok=4.388B dt=22685 +23:40:44 step 33500 loss=3.7604 lr=2.82e-04 tok=4.391B dt=22761 +23:42:00 step 33520 loss=3.7723 lr=2.82e-04 tok=4.394B dt=22837 +23:43:16 step 33540 loss=3.8522 lr=2.82e-04 tok=4.396B dt=22913 +23:44:32 step 33560 loss=3.7049 lr=2.82e-04 tok=4.399B dt=22989 +23:45:48 step 33580 loss=3.7373 lr=2.82e-04 tok=4.401B dt=23065 +23:47:04 step 33600 loss=3.7322 lr=2.82e-04 tok=4.404B dt=23141 +23:47:17 pushed ckpt_w3.pt +23:47:17 pushed log_w3.txt +23:48:45 step 33620 loss=3.8974 lr=2.82e-04 tok=4.407B dt=23243 +23:50:01 step 33640 loss=3.8244 lr=2.82e-04 tok=4.409B dt=23319 +23:51:17 step 33660 loss=3.7574 lr=2.82e-04 tok=4.412B dt=23394 +23:52:33 step 33680 loss=3.7185 lr=2.82e-04 tok=4.415B dt=23470 +23:53:49 step 33700 loss=3.7741 lr=2.82e-04 tok=4.417B dt=23546 +23:55:05 step 33720 loss=3.7049 lr=2.82e-04 tok=4.420B dt=23622 +23:56:20 step 33740 loss=3.7307 lr=2.82e-04 tok=4.422B dt=23698 +23:57:36 step 33760 loss=3.6941 lr=2.82e-04 tok=4.425B dt=23774 +23:58:52 step 33780 loss=3.7455 lr=2.82e-04 tok=4.428B dt=23850 +00:00:08 step 33800 loss=3.6968 lr=2.82e-04 tok=4.430B dt=23926 +00:01:24 step 33820 loss=3.7776 lr=2.82e-04 tok=4.433B dt=24002 +00:02:40 step 33840 loss=3.6676 lr=2.82e-04 tok=4.435B dt=24078 +00:03:56 step 33860 loss=3.6284 lr=2.82e-04 tok=4.438B dt=24153 +00:05:12 step 33880 loss=3.8200 lr=2.82e-04 tok=4.441B dt=24229 +00:06:28 step 33900 loss=3.6999 lr=2.82e-04 tok=4.443B dt=24305 +00:07:44 step 33920 loss=3.7540 lr=2.82e-04 tok=4.446B dt=24381 +00:09:00 step 33940 loss=3.7006 lr=2.82e-04 tok=4.449B dt=24457 +00:10:16 step 33960 loss=3.7749 lr=2.82e-04 tok=4.451B dt=24534 +00:11:32 step 33980 loss=3.7716 lr=2.82e-04 tok=4.454B dt=24610 +00:12:49 step 34000 loss=3.7396 lr=2.82e-04 tok=4.456B dt=24687 +00:14:05 step 34020 loss=3.7293 lr=2.82e-04 tok=4.459B dt=24763 +00:15:22 step 34040 loss=3.8173 lr=2.82e-04 tok=4.462B dt=24840 +00:16:38 step 34060 loss=3.6896 lr=2.82e-04 tok=4.464B dt=24916 +00:17:56 step 34080 loss=3.7993 lr=2.82e-04 tok=4.467B dt=24994 +00:19:12 step 34100 loss=3.7634 lr=2.82e-04 tok=4.470B dt=25070 +00:20:28 step 34120 loss=3.8559 lr=2.82e-04 tok=4.472B dt=25146 +00:21:44 step 34140 loss=3.7250 lr=2.82e-04 tok=4.475B dt=25222 +00:23:00 step 34160 loss=3.8544 lr=2.82e-04 tok=4.477B dt=25298 +00:24:16 step 34180 loss=3.7748 lr=2.82e-04 tok=4.480B dt=25374 +00:25:32 step 34200 loss=3.7404 lr=2.82e-04 tok=4.483B dt=25450 +00:25:49 pushed ckpt_w3.pt diff --git a/log_w31.txt b/log_w31.txt new file mode 100644 index 0000000000000000000000000000000000000000..efcf31134905486a991b1246166e5717bbe680d6 --- /dev/null +++ b/log_w31.txt @@ -0,0 +1,591 @@ +launch +13:50:58 name=w31 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +13:51:04 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +13:51:14 params 1.08B +13:51:15 resumed step=73950 seen=9.69B round=145 +13:51:15 stream salt=73950 +13:53:42 start: accum=4, 131072 tokens per optimiser step +13:55:32 step 73951 loss=3.1263 lr=1.10e-04 tok=9.693B dt=111 wait=82 +13:55:37 step 73952 loss=2.9732 lr=1.10e-04 tok=9.693B dt=116 wait=82 +13:56:12 step 73960 loss=3.2070 lr=1.10e-04 tok=9.694B dt=151 wait=82 +13:57:39 step 73980 loss=3.2932 lr=1.10e-04 tok=9.697B dt=237 wait=83 +13:59:02 step 74000 loss=3.2386 lr=1.10e-04 tok=9.699B dt=320 wait=83 +14:00:17 step 74020 loss=3.2517 lr=1.10e-04 tok=9.702B dt=395 wait=84 +14:01:32 step 74040 loss=3.2237 lr=1.10e-04 tok=9.705B dt=471 wait=84 +14:02:47 step 74060 loss=3.1344 lr=1.10e-04 tok=9.707B dt=545 wait=84 +14:04:02 step 74080 loss=3.1607 lr=1.10e-04 tok=9.710B dt=620 wait=85 +14:05:17 step 74100 loss=3.2230 lr=1.10e-04 tok=9.712B dt=695 wait=85 +14:05:31 pushed ckpt_w31.pt,log_w31.txt +14:07:02 waited 91s for global r=146 +14:07:19 global r=146 load 4.285 -> 4.258 +14:08:38 step 74120 loss=3.0986 lr=1.10e-04 tok=9.715B dt=897 wait=86 +14:09:53 step 74140 loss=3.1275 lr=1.10e-04 tok=9.718B dt=971 wait=86 +14:11:08 step 74160 loss=3.1069 lr=1.10e-04 tok=9.720B dt=1046 wait=86 +14:12:23 step 74180 loss=3.2030 lr=1.10e-04 tok=9.723B dt=1121 wait=87 +14:13:38 step 74200 loss=3.1678 lr=1.10e-04 tok=9.726B dt=1196 wait=87 +14:14:53 step 74220 loss=3.1306 lr=1.10e-04 tok=9.728B dt=1271 wait=87 +14:16:08 step 74240 loss=3.1632 lr=1.10e-04 tok=9.731B dt=1346 wait=88 +14:16:58 pushed ckpt_w31.pt,log_w31.txt +14:18:29 waited 91s for global r=147 +14:18:38 global r=147 load 4.280 -> 4.254 +14:19:21 step 74260 loss=3.1496 lr=1.10e-04 tok=9.733B dt=1539 wait=88 +14:20:36 step 74280 loss=3.1052 lr=1.10e-04 tok=9.736B dt=1614 wait=88 +14:21:51 step 74300 loss=3.2195 lr=1.10e-04 tok=9.739B dt=1689 wait=89 +14:23:06 step 74320 loss=3.2397 lr=1.09e-04 tok=9.741B dt=1765 wait=89 +14:24:21 step 74340 loss=3.1846 lr=1.09e-04 tok=9.744B dt=1840 wait=89 +14:25:36 step 74360 loss=3.2434 lr=1.09e-04 tok=9.747B dt=1915 wait=90 +14:26:51 step 74380 loss=3.1090 lr=1.09e-04 tok=9.749B dt=1990 wait=90 +14:28:06 step 74400 loss=3.1642 lr=1.09e-04 tok=9.752B dt=2065 wait=91 +14:28:25 pushed ckpt_w31.pt,log_w31.txt +14:29:41 waited 76s for global r=148 +14:29:54 global r=148 load 4.278 -> 4.253 +14:31:15 step 74420 loss=3.1563 lr=1.09e-04 tok=9.754B dt=2254 wait=91 +14:32:30 step 74440 loss=3.1510 lr=1.09e-04 tok=9.757B dt=2329 wait=91 +14:33:45 step 74460 loss=3.1644 lr=1.09e-04 tok=9.760B dt=2404 wait=92 +14:35:00 step 74480 loss=3.1789 lr=1.09e-04 tok=9.762B dt=2479 wait=92 +14:36:15 step 74500 loss=3.1980 lr=1.09e-04 tok=9.765B dt=2554 wait=92 +14:37:30 step 74520 loss=3.1513 lr=1.09e-04 tok=9.767B dt=2629 wait=93 +14:38:45 step 74540 loss=3.2256 lr=1.09e-04 tok=9.770B dt=2704 wait=93 +14:39:35 pushed ckpt_w31.pt,log_w31.txt +14:41:05 waited 91s for global r=149 +14:41:14 global r=149 load 4.265 -> 4.248 +14:41:57 step 74560 loss=3.1414 lr=1.09e-04 tok=9.773B dt=2895 wait=93 +14:43:12 step 74580 loss=3.1321 lr=1.09e-04 tok=9.775B dt=2970 wait=94 +14:44:27 step 74600 loss=3.2240 lr=1.09e-04 tok=9.778B dt=3045 wait=94 +14:45:42 step 74620 loss=3.2254 lr=1.09e-04 tok=9.781B dt=3120 wait=95 +14:46:57 step 74640 loss=3.1332 lr=1.09e-04 tok=9.783B dt=3195 wait=95 +14:48:12 step 74660 loss=3.2010 lr=1.09e-04 tok=9.786B dt=3270 wait=95 +14:49:27 step 74680 loss=3.0954 lr=1.09e-04 tok=9.788B dt=3345 wait=96 +14:50:42 step 74700 loss=3.1258 lr=1.09e-04 tok=9.791B dt=3420 wait=96 +14:50:55 pushed ckpt_w31.pt,log_w31.txt +14:51:56 waited 61s for global r=150 +14:52:05 global r=150 load 4.274 -> 4.247 +14:53:29 step 74720 loss=3.1141 lr=1.09e-04 tok=9.794B dt=3587 wait=96 +14:54:44 step 74740 loss=3.1645 lr=1.09e-04 tok=9.796B dt=3662 wait=97 +14:55:59 step 74760 loss=3.1781 lr=1.09e-04 tok=9.799B dt=3737 wait=97 +14:57:14 step 74780 loss=3.2667 lr=1.09e-04 tok=9.802B dt=3812 wait=98 +14:58:29 step 74800 loss=3.2205 lr=1.09e-04 tok=9.804B dt=3887 wait=98 +14:59:44 step 74820 loss=3.1403 lr=1.09e-04 tok=9.807B dt=3962 wait=98 +15:00:59 step 74840 loss=3.1440 lr=1.09e-04 tok=9.809B dt=4037 wait=99 +15:01:50 pushed ckpt_w31.pt,log_w31.txt +15:03:05 waited 76s for global r=151 +15:03:19 global r=151 load 4.267 -> 4.246 +15:04:07 step 74860 loss=3.1386 lr=1.09e-04 tok=9.812B dt=4225 wait=99 +15:05:21 step 74880 loss=3.1373 lr=1.09e-04 tok=9.815B dt=4300 wait=99 +15:06:36 step 74900 loss=3.1791 lr=1.09e-04 tok=9.817B dt=4375 wait=100 +15:07:52 step 74920 loss=3.2014 lr=1.09e-04 tok=9.820B dt=4450 wait=100 +15:09:06 step 74940 loss=3.1276 lr=1.09e-04 tok=9.823B dt=4525 wait=100 +15:10:22 step 74960 loss=3.2505 lr=1.09e-04 tok=9.825B dt=4600 wait=101 +15:11:37 step 74980 loss=3.1233 lr=1.09e-04 tok=9.828B dt=4675 wait=101 +15:12:52 step 75000 loss=3.1374 lr=1.09e-04 tok=9.830B dt=4750 wait=101 +15:13:06 pushed ckpt_w31.pt,log_w31.txt +15:14:37 waited 91s for global r=152 +15:14:47 global r=152 load 4.267 -> 4.246 +15:16:06 step 75020 loss=3.1111 lr=1.09e-04 tok=9.833B dt=4944 wait=102 +15:17:21 step 75040 loss=3.1575 lr=1.09e-04 tok=9.836B dt=5019 wait=102 +15:18:36 step 75060 loss=3.1198 lr=1.09e-04 tok=9.838B dt=5094 wait=103 +15:19:50 step 75080 loss=3.2225 lr=1.09e-04 tok=9.841B dt=5169 wait=103 +15:21:05 step 75100 loss=3.2070 lr=1.09e-04 tok=9.844B dt=5244 wait=103 +15:22:20 step 75120 loss=3.1210 lr=1.09e-04 tok=9.846B dt=5319 wait=104 +15:23:35 step 75140 loss=3.1851 lr=1.09e-04 tok=9.849B dt=5394 wait=104 +15:24:27 pushed ckpt_w31.pt,log_w31.txt +15:25:42 waited 76s for global r=153 +15:25:51 global r=153 load 4.263 -> 4.244 +15:26:37 step 75160 loss=3.1172 lr=1.09e-04 tok=9.851B dt=5575 wait=104 +15:27:52 step 75180 loss=3.1490 lr=1.09e-04 tok=9.854B dt=5650 wait=105 +15:29:07 step 75200 loss=3.2066 lr=1.09e-04 tok=9.857B dt=5725 wait=105 +15:30:22 step 75220 loss=3.1875 lr=1.09e-04 tok=9.859B dt=5800 wait=105 +15:31:36 step 75240 loss=3.1202 lr=1.09e-04 tok=9.862B dt=5875 wait=106 +15:32:51 step 75260 loss=3.2027 lr=1.09e-04 tok=9.864B dt=5950 wait=106 +15:34:06 step 75280 loss=3.1041 lr=1.09e-04 tok=9.867B dt=6025 wait=107 +15:35:21 step 75300 loss=3.1633 lr=1.09e-04 tok=9.870B dt=6100 wait=107 +15:35:34 pushed ckpt_w31.pt,log_w31.txt +15:36:50 waited 76s for global r=154 +15:37:01 global r=154 load 4.262 -> 4.248 +15:38:23 step 75320 loss=3.1449 lr=1.09e-04 tok=9.872B dt=6281 wait=107 +15:39:38 step 75340 loss=3.1287 lr=1.08e-04 tok=9.875B dt=6356 wait=108 +15:40:53 step 75360 loss=3.1498 lr=1.08e-04 tok=9.878B dt=6431 wait=108 +15:42:08 step 75380 loss=3.1929 lr=1.08e-04 tok=9.880B dt=6506 wait=108 +15:43:23 step 75400 loss=3.2101 lr=1.08e-04 tok=9.883B dt=6581 wait=109 +15:44:38 step 75420 loss=3.1339 lr=1.08e-04 tok=9.885B dt=6657 wait=109 +15:45:53 step 75440 loss=3.1994 lr=1.08e-04 tok=9.888B dt=6732 wait=109 +15:46:44 pushed ckpt_w31.pt,log_w31.txt +15:48:00 waited 76s for global r=155 +15:48:10 global r=155 load 4.266 -> 4.244 +15:48:55 step 75460 loss=3.1401 lr=1.08e-04 tok=9.891B dt=6913 wait=110 +15:50:09 step 75480 loss=3.1239 lr=1.08e-04 tok=9.893B dt=6988 wait=110 +15:51:24 step 75500 loss=3.1969 lr=1.08e-04 tok=9.896B dt=7063 wait=111 +15:52:39 step 75520 loss=3.1742 lr=1.08e-04 tok=9.899B dt=7137 wait=111 +15:53:54 step 75540 loss=3.1009 lr=1.08e-04 tok=9.901B dt=7212 wait=111 +15:55:09 step 75560 loss=3.1909 lr=1.08e-04 tok=9.904B dt=7287 wait=112 +15:56:24 step 75580 loss=3.1242 lr=1.08e-04 tok=9.906B dt=7362 wait=112 +15:57:39 step 75600 loss=3.1864 lr=1.08e-04 tok=9.909B dt=7437 wait=112 +15:57:51 pushed ckpt_w31.pt,log_w31.txt +15:59:22 waited 91s for global r=156 +15:59:35 global r=156 load 4.267 -> 4.246 +16:00:53 step 75620 loss=3.1639 lr=1.08e-04 tok=9.912B dt=7632 wait=113 +16:02:09 step 75640 loss=3.1622 lr=1.08e-04 tok=9.914B dt=7707 wait=113 +16:03:24 step 75660 loss=3.1121 lr=1.08e-04 tok=9.917B dt=7782 wait=114 +16:04:38 step 75680 loss=3.1669 lr=1.08e-04 tok=9.920B dt=7857 wait=114 +16:05:53 step 75700 loss=3.1718 lr=1.08e-04 tok=9.922B dt=7932 wait=114 +16:07:08 step 75720 loss=3.1378 lr=1.08e-04 tok=9.925B dt=8007 wait=115 +16:08:23 step 75740 loss=3.1709 lr=1.08e-04 tok=9.927B dt=8082 wait=115 +16:09:14 pushed ckpt_w31.pt,log_w31.txt +16:10:29 waited 76s for global r=157 +16:10:42 global r=157 load 4.267 -> 4.245 +16:11:26 step 75760 loss=3.1405 lr=1.08e-04 tok=9.930B dt=8264 wait=115 +16:12:41 step 75780 loss=3.1278 lr=1.08e-04 tok=9.933B dt=8339 wait=116 +16:13:56 step 75800 loss=3.2193 lr=1.08e-04 tok=9.935B dt=8415 wait=116 +16:15:12 step 75820 loss=3.2557 lr=1.08e-04 tok=9.938B dt=8490 wait=117 +16:16:27 step 75840 loss=3.1783 lr=1.08e-04 tok=9.941B dt=8565 wait=117 +16:17:42 step 75860 loss=3.2620 lr=1.08e-04 tok=9.943B dt=8640 wait=117 +16:18:57 step 75880 loss=3.1070 lr=1.08e-04 tok=9.946B dt=8715 wait=118 +16:20:12 step 75900 loss=3.1533 lr=1.08e-04 tok=9.948B dt=8790 wait=118 +16:20:25 pushed ckpt_w31.pt,log_w31.txt +16:21:41 waited 76s for global r=158 +16:21:51 global r=158 load 4.265 -> 4.244 +16:23:11 step 75920 loss=3.0957 lr=1.08e-04 tok=9.951B dt=8970 wait=118 +16:24:26 step 75940 loss=3.0963 lr=1.08e-04 tok=9.954B dt=9045 wait=119 +16:25:41 step 75960 loss=3.1011 lr=1.08e-04 tok=9.956B dt=9120 wait=119 +16:26:56 step 75980 loss=3.1629 lr=1.08e-04 tok=9.959B dt=9194 wait=119 +16:28:11 step 76000 loss=3.1651 lr=1.08e-04 tok=9.961B dt=9269 wait=120 +16:29:26 step 76020 loss=3.1721 lr=1.08e-04 tok=9.964B dt=9344 wait=120 +16:30:41 step 76040 loss=3.1746 lr=1.08e-04 tok=9.967B dt=9419 wait=121 +16:31:36 pushed ckpt_w31.pt,log_w31.txt +16:32:52 waited 76s for global r=159 +16:33:01 global r=159 load 4.257 -> 4.240 +16:33:46 step 76060 loss=3.1264 lr=1.08e-04 tok=9.969B dt=9604 wait=121 +16:35:01 step 76080 loss=3.1257 lr=1.08e-04 tok=9.972B dt=9679 wait=121 +16:36:16 step 76100 loss=3.1729 lr=1.08e-04 tok=9.975B dt=9754 wait=122 +16:37:31 step 76120 loss=3.1800 lr=1.08e-04 tok=9.977B dt=9829 wait=122 +16:38:46 step 76140 loss=3.1483 lr=1.08e-04 tok=9.980B dt=9904 wait=122 +16:40:01 step 76160 loss=3.2029 lr=1.08e-04 tok=9.982B dt=9979 wait=123 +16:41:16 step 76180 loss=3.1233 lr=1.08e-04 tok=9.985B dt=10054 wait=123 +16:42:31 step 76200 loss=3.1677 lr=1.08e-04 tok=9.988B dt=10129 wait=123 +16:42:43 pushed ckpt_w31.pt,log_w31.txt +16:43:58 waited 76s for global r=160 +16:44:08 global r=160 load 4.262 -> 4.244 +16:45:29 step 76220 loss=3.1200 lr=1.08e-04 tok=9.990B dt=10307 wait=124 +16:46:44 step 76240 loss=3.1511 lr=1.08e-04 tok=9.993B dt=10382 wait=124 +16:47:59 step 76260 loss=3.1742 lr=1.08e-04 tok=9.996B dt=10457 wait=125 +16:49:14 step 76280 loss=3.1897 lr=1.08e-04 tok=9.998B dt=10532 wait=125 +16:50:29 step 76300 loss=3.1731 lr=1.08e-04 tok=10.001B dt=10607 wait=125 +16:51:44 step 76320 loss=3.1267 lr=1.08e-04 tok=10.003B dt=10682 wait=126 +16:52:59 step 76340 loss=3.1414 lr=1.08e-04 tok=10.006B dt=10757 wait=126 +16:53:51 pushed ckpt_w31.pt,log_w31.txt +16:55:07 waited 76s for global r=161 +16:55:16 global r=161 load 4.259 -> 4.239 +16:55:59 step 76360 loss=3.1199 lr=1.07e-04 tok=10.009B dt=10937 wait=126 +16:57:14 step 76380 loss=3.1052 lr=1.07e-04 tok=10.011B dt=11012 wait=127 +16:58:29 step 76400 loss=3.1507 lr=1.07e-04 tok=10.014B dt=11087 wait=127 +16:59:43 step 76420 loss=3.1867 lr=1.07e-04 tok=10.017B dt=11162 wait=127 +17:00:58 step 76440 loss=3.1461 lr=1.07e-04 tok=10.019B dt=11237 wait=128 +17:02:13 step 76460 loss=3.1942 lr=1.07e-04 tok=10.022B dt=11312 wait=128 +17:03:28 step 76480 loss=3.1381 lr=1.07e-04 tok=10.024B dt=11387 wait=129 +17:04:43 step 76500 loss=3.1906 lr=1.07e-04 tok=10.027B dt=11462 wait=129 +17:05:00 pushed ckpt_w31.pt,log_w31.txt +17:06:30 waited 91s for global r=162 +17:06:40 global r=162 load 4.264 -> 4.244 +17:08:01 step 76520 loss=3.1846 lr=1.07e-04 tok=10.030B dt=11659 wait=129 +17:09:16 step 76540 loss=3.1857 lr=1.07e-04 tok=10.032B dt=11734 wait=130 +17:10:31 step 76560 loss=3.1416 lr=1.07e-04 tok=10.035B dt=11809 wait=130 +17:11:46 step 76580 loss=3.2042 lr=1.07e-04 tok=10.037B dt=11884 wait=130 +17:13:01 step 76600 loss=3.1773 lr=1.07e-04 tok=10.040B dt=11959 wait=131 +17:14:16 step 76620 loss=3.1112 lr=1.07e-04 tok=10.043B dt=12034 wait=131 +17:15:31 step 76640 loss=3.1291 lr=1.07e-04 tok=10.045B dt=12109 wait=131 +17:16:21 pushed ckpt_w31.pt,log_w31.txt +17:17:37 waited 76s for global r=163 +17:17:46 global r=163 load 4.259 -> 4.241 +17:18:28 step 76660 loss=3.1074 lr=1.07e-04 tok=10.048B dt=12287 wait=132 +17:19:43 step 76680 loss=3.1039 lr=1.07e-04 tok=10.051B dt=12362 wait=132 +17:20:58 step 76700 loss=3.1673 lr=1.07e-04 tok=10.053B dt=12437 wait=133 +17:22:13 step 76720 loss=3.2316 lr=1.07e-04 tok=10.056B dt=12512 wait=133 +17:23:28 step 76740 loss=3.1420 lr=1.07e-04 tok=10.058B dt=12587 wait=133 +17:24:43 step 76760 loss=3.2207 lr=1.07e-04 tok=10.061B dt=12662 wait=134 +17:25:58 step 76780 loss=3.0863 lr=1.07e-04 tok=10.064B dt=12737 wait=134 +17:27:13 step 76800 loss=3.1388 lr=1.07e-04 tok=10.066B dt=12812 wait=134 +17:27:28 pushed ckpt_w31.pt,log_w31.txt +17:28:59 waited 91s for global r=164 +17:29:08 global r=164 load 4.264 -> 4.241 +17:30:27 step 76820 loss=3.1085 lr=1.07e-04 tok=10.069B dt=13005 wait=135 +17:31:41 step 76840 loss=3.0945 lr=1.07e-04 tok=10.072B dt=13080 wait=135 +17:32:56 step 76860 loss=3.1093 lr=1.07e-04 tok=10.074B dt=13154 wait=135 +17:34:11 step 76880 loss=3.1518 lr=1.07e-04 tok=10.077B dt=13229 wait=136 +17:35:26 step 76900 loss=3.1702 lr=1.07e-04 tok=10.079B dt=13304 wait=136 +17:36:41 step 76920 loss=3.1328 lr=1.07e-04 tok=10.082B dt=13379 wait=137 +17:37:56 step 76940 loss=3.1979 lr=1.07e-04 tok=10.085B dt=13454 wait=137 +17:38:49 pushed ckpt_w31.pt,log_w31.txt +17:40:50 waited 121s for global r=165 +17:41:00 global r=165 load 4.265 -> 4.236 +17:41:49 step 76960 loss=3.1401 lr=1.07e-04 tok=10.087B dt=13687 wait=137 +17:43:04 step 76980 loss=3.1432 lr=1.07e-04 tok=10.090B dt=13762 wait=138 +17:44:19 step 77000 loss=3.2222 lr=1.07e-04 tok=10.093B dt=13837 wait=138 +17:45:34 step 77020 loss=3.2060 lr=1.07e-04 tok=10.095B dt=13912 wait=138 +17:46:49 step 77040 loss=3.1097 lr=1.07e-04 tok=10.098B dt=13987 wait=139 +17:48:03 step 77060 loss=3.1828 lr=1.07e-04 tok=10.100B dt=14062 wait=139 +17:49:18 step 77080 loss=3.0888 lr=1.07e-04 tok=10.103B dt=14137 wait=139 +17:50:33 step 77100 loss=3.1615 lr=1.07e-04 tok=10.106B dt=14212 wait=140 +17:50:49 pushed ckpt_w31.pt,log_w31.txt +17:52:04 waited 76s for global r=166 +17:52:18 global r=166 load 4.256 -> 4.238 +17:53:36 step 77120 loss=3.1060 lr=1.07e-04 tok=10.108B dt=14395 wait=140 +17:54:51 step 77140 loss=3.1638 lr=1.07e-04 tok=10.111B dt=14470 wait=141 +17:56:06 step 77160 loss=3.1172 lr=1.07e-04 tok=10.114B dt=14544 wait=141 +17:57:21 step 77180 loss=3.1777 lr=1.07e-04 tok=10.116B dt=14619 wait=141 +17:58:36 step 77200 loss=3.1622 lr=1.07e-04 tok=10.119B dt=14694 wait=142 +17:59:51 step 77220 loss=3.0773 lr=1.07e-04 tok=10.121B dt=14769 wait=142 +18:01:06 step 77240 loss=3.2001 lr=1.07e-04 tok=10.124B dt=14844 wait=142 +18:01:56 pushed ckpt_w31.pt,log_w31.txt +18:03:27 waited 91s for global r=167 +18:03:36 global r=167 load 4.260 -> 4.238 +18:04:21 step 77260 loss=3.1105 lr=1.07e-04 tok=10.127B dt=15039 wait=143 +18:05:36 step 77280 loss=3.1626 lr=1.07e-04 tok=10.129B dt=15114 wait=143 +18:06:51 step 77300 loss=3.2103 lr=1.07e-04 tok=10.132B dt=15189 wait=143 +18:08:06 step 77320 loss=3.2021 lr=1.07e-04 tok=10.134B dt=15264 wait=144 +18:09:21 step 77340 loss=3.0989 lr=1.07e-04 tok=10.137B dt=15339 wait=144 +18:10:36 step 77360 loss=3.1997 lr=1.06e-04 tok=10.140B dt=15414 wait=145 +18:11:51 step 77380 loss=3.0970 lr=1.06e-04 tok=10.142B dt=15489 wait=145 +18:13:06 step 77400 loss=3.1320 lr=1.06e-04 tok=10.145B dt=15564 wait=145 +18:13:22 pushed ckpt_w31.pt,log_w31.txt +18:14:38 waited 76s for global r=168 +18:14:50 global r=168 load 4.264 -> 4.239 +18:16:10 step 77420 loss=3.1312 lr=1.06e-04 tok=10.148B dt=15748 wait=146 +18:17:25 step 77440 loss=3.1880 lr=1.06e-04 tok=10.150B dt=15823 wait=146 +18:18:40 step 77460 loss=3.1727 lr=1.06e-04 tok=10.153B dt=15898 wait=146 +18:19:55 step 77480 loss=3.1725 lr=1.06e-04 tok=10.155B dt=15973 wait=147 +18:21:10 step 77500 loss=3.1852 lr=1.06e-04 tok=10.158B dt=16048 wait=147 +18:22:24 step 77520 loss=3.1396 lr=1.06e-04 tok=10.161B dt=16123 wait=147 +18:23:39 step 77540 loss=3.1816 lr=1.06e-04 tok=10.163B dt=16198 wait=148 +18:24:28 pushed ckpt_w31.pt,log_w31.txt +18:25:44 waited 76s for global r=169 +18:25:53 global r=169 load 4.252 -> 4.235 +18:26:36 step 77560 loss=3.1244 lr=1.06e-04 tok=10.166B dt=16374 wait=148 +18:27:50 step 77580 loss=3.1155 lr=1.06e-04 tok=10.169B dt=16449 wait=148 +18:29:05 step 77600 loss=3.1685 lr=1.06e-04 tok=10.171B dt=16524 wait=149 +18:30:20 step 77620 loss=3.2002 lr=1.06e-04 tok=10.174B dt=16599 wait=149 +18:31:35 step 77640 loss=3.1348 lr=1.06e-04 tok=10.176B dt=16673 wait=149 +18:32:50 step 77660 loss=3.2235 lr=1.06e-04 tok=10.179B dt=16749 wait=150 +18:34:05 step 77680 loss=3.1166 lr=1.06e-04 tok=10.182B dt=16823 wait=150 +18:35:20 step 77700 loss=3.1347 lr=1.06e-04 tok=10.184B dt=16898 wait=151 +18:35:35 pushed ckpt_w31.pt,log_w31.txt +18:37:06 waited 91s for global r=170 +18:37:20 global r=170 load 4.255 -> 4.235 +18:38:41 step 77720 loss=3.1063 lr=1.06e-04 tok=10.187B dt=17099 wait=151 +18:39:56 step 77740 loss=3.1293 lr=1.06e-04 tok=10.190B dt=17174 wait=151 +18:41:11 step 77760 loss=3.1716 lr=1.06e-04 tok=10.192B dt=17249 wait=152 +18:42:26 step 77780 loss=3.1813 lr=1.06e-04 tok=10.195B dt=17324 wait=152 +18:43:41 step 77800 loss=3.1368 lr=1.06e-04 tok=10.197B dt=17399 wait=152 +18:44:56 step 77820 loss=3.0847 lr=1.06e-04 tok=10.200B dt=17474 wait=153 +18:46:11 step 77840 loss=3.1284 lr=1.06e-04 tok=10.203B dt=17549 wait=153 +18:47:01 pushed ckpt_w31.pt,log_w31.txt +18:48:32 waited 91s for global r=171 +18:48:47 global r=171 load 4.253 -> 4.234 +18:49:28 step 77860 loss=3.0874 lr=1.06e-04 tok=10.205B dt=17746 wait=153 +18:50:43 step 77880 loss=3.0961 lr=1.06e-04 tok=10.208B dt=17821 wait=154 +18:51:58 step 77900 loss=3.1714 lr=1.06e-04 tok=10.211B dt=17896 wait=154 +18:53:13 step 77920 loss=3.2138 lr=1.06e-04 tok=10.213B dt=17971 wait=154 +18:54:28 step 77940 loss=3.1829 lr=1.06e-04 tok=10.216B dt=18046 wait=155 +18:55:43 step 77960 loss=3.2612 lr=1.06e-04 tok=10.218B dt=18121 wait=155 +18:56:58 step 77980 loss=3.1244 lr=1.06e-04 tok=10.221B dt=18196 wait=155 +18:58:13 step 78000 loss=3.1463 lr=1.06e-04 tok=10.224B dt=18271 wait=156 +18:58:26 pushed ckpt_w31.pt,log_w31.txt +19:00:27 waited 121s for global r=172 +19:00:44 global r=172 load 4.261 -> 4.235 +19:02:04 step 78020 loss=3.1251 lr=1.06e-04 tok=10.226B dt=18503 wait=156 +19:03:19 step 78040 loss=3.1096 lr=1.06e-04 tok=10.229B dt=18578 wait=156 +19:04:34 step 78060 loss=3.1315 lr=1.06e-04 tok=10.231B dt=18652 wait=157 +19:05:49 step 78080 loss=3.1715 lr=1.06e-04 tok=10.234B dt=18727 wait=157 +19:07:04 step 78100 loss=3.1433 lr=1.06e-04 tok=10.237B dt=18802 wait=158 +19:08:19 step 78120 loss=3.1015 lr=1.06e-04 tok=10.239B dt=18877 wait=158 +19:09:34 step 78140 loss=3.1741 lr=1.06e-04 tok=10.242B dt=18952 wait=158 +19:10:24 pushed ckpt_w31.pt,log_w31.txt +19:11:54 waited 91s for global r=173 +19:12:11 global r=173 load 4.259 -> 4.236 +19:12:56 step 78160 loss=3.1036 lr=1.06e-04 tok=10.245B dt=19154 wait=159 +19:14:11 step 78180 loss=3.1374 lr=1.06e-04 tok=10.247B dt=19229 wait=159 +19:15:26 step 78200 loss=3.2094 lr=1.06e-04 tok=10.250B dt=19304 wait=159 +19:16:41 step 78220 loss=3.2168 lr=1.06e-04 tok=10.252B dt=19379 wait=160 +19:17:56 step 78240 loss=3.1307 lr=1.06e-04 tok=10.255B dt=19454 wait=160 +19:19:11 step 78260 loss=3.1821 lr=1.06e-04 tok=10.258B dt=19529 wait=160 +19:20:26 step 78280 loss=3.0551 lr=1.06e-04 tok=10.260B dt=19604 wait=161 +19:21:41 step 78300 loss=3.1105 lr=1.06e-04 tok=10.263B dt=19679 wait=161 +19:21:53 pushed ckpt_w31.pt,log_w31.txt +19:24:09 waited 136s for global r=174 +19:24:23 global r=174 load 4.251 -> 4.241 +19:25:43 step 78320 loss=3.1395 lr=1.06e-04 tok=10.266B dt=19922 wait=161 +19:26:58 step 78340 loss=3.1796 lr=1.06e-04 tok=10.268B dt=19996 wait=162 +19:28:13 step 78360 loss=3.1523 lr=1.06e-04 tok=10.271B dt=20072 wait=162 +19:29:28 step 78380 loss=3.2046 lr=1.05e-04 tok=10.273B dt=20147 wait=163 +19:30:43 step 78400 loss=3.1733 lr=1.05e-04 tok=10.276B dt=20221 wait=163 +19:31:58 step 78420 loss=3.1396 lr=1.05e-04 tok=10.279B dt=20296 wait=163 +19:33:13 step 78440 loss=3.1730 lr=1.05e-04 tok=10.281B dt=20371 wait=164 +19:34:03 pushed ckpt_w31.pt,log_w31.txt +19:37:48 step 78460 loss=3.1135 lr=1.05e-04 tok=10.284B dt=20646 wait=164 +19:39:03 step 78480 loss=3.1604 lr=1.05e-04 tok=10.287B dt=20721 wait=164 +19:40:18 step 78500 loss=3.0976 lr=1.05e-04 tok=10.289B dt=20796 wait=165 +19:41:33 step 78520 loss=3.0624 lr=1.05e-04 tok=10.292B dt=20871 wait=165 +19:42:48 step 78540 loss=3.1255 lr=1.05e-04 tok=10.294B dt=20946 wait=165 +19:44:02 step 78560 loss=3.1524 lr=1.05e-04 tok=10.297B dt=21021 wait=166 +19:45:17 step 78580 loss=3.1627 lr=1.05e-04 tok=10.300B dt=21096 wait=166 +19:46:33 step 78600 loss=3.1325 lr=1.05e-04 tok=10.302B dt=21171 wait=166 +19:46:48 pushed ckpt_w31.pt,log_w31.txt +19:51:07 step 78620 loss=3.1481 lr=1.05e-04 tok=10.305B dt=21445 wait=167 +19:52:22 step 78640 loss=3.1351 lr=1.05e-04 tok=10.308B dt=21520 wait=167 +19:53:37 step 78660 loss=3.1702 lr=1.05e-04 tok=10.310B dt=21595 wait=167 +19:54:52 step 78680 loss=3.1791 lr=1.05e-04 tok=10.313B dt=21670 wait=168 +19:56:07 step 78700 loss=3.1686 lr=1.05e-04 tok=10.315B dt=21745 wait=168 +19:57:22 step 78720 loss=3.1583 lr=1.05e-04 tok=10.318B dt=21820 wait=169 +19:58:37 step 78740 loss=3.1740 lr=1.05e-04 tok=10.321B dt=21895 wait=169 +19:59:27 pushed ckpt_w31.pt,log_w31.txt +20:03:12 step 78760 loss=3.1966 lr=1.05e-04 tok=10.323B dt=22171 wait=169 +20:04:27 step 78780 loss=3.1593 lr=1.05e-04 tok=10.326B dt=22245 wait=170 +20:05:42 step 78800 loss=3.1483 lr=1.05e-04 tok=10.328B dt=22320 wait=170 +20:06:57 step 78820 loss=3.1048 lr=1.05e-04 tok=10.331B dt=22395 wait=170 +20:08:11 step 78840 loss=3.1600 lr=1.05e-04 tok=10.334B dt=22470 wait=171 +20:09:26 step 78860 loss=3.2470 lr=1.05e-04 tok=10.336B dt=22544 wait=171 +20:10:41 step 78880 loss=3.1864 lr=1.05e-04 tok=10.339B dt=22620 wait=171 +20:11:56 step 78900 loss=3.1607 lr=1.05e-04 tok=10.342B dt=22695 wait=172 +20:12:11 pushed ckpt_w31.pt,log_w31.txt +20:16:31 step 78920 loss=3.2217 lr=1.05e-04 tok=10.344B dt=22969 wait=172 +20:17:46 step 78940 loss=3.1774 lr=1.05e-04 tok=10.347B dt=23044 wait=173 +20:19:01 step 78960 loss=3.1306 lr=1.05e-04 tok=10.349B dt=23120 wait=173 +20:20:16 step 78980 loss=3.1172 lr=1.05e-04 tok=10.352B dt=23195 wait=173 +20:21:31 step 79000 loss=3.1149 lr=1.05e-04 tok=10.355B dt=23270 wait=174 +20:22:46 step 79020 loss=3.1051 lr=1.05e-04 tok=10.357B dt=23344 wait=174 +20:24:01 step 79040 loss=3.1378 lr=1.05e-04 tok=10.360B dt=23419 wait=174 +20:24:53 pushed ckpt_w31.pt,log_w31.txt +20:28:36 step 79060 loss=3.1512 lr=1.05e-04 tok=10.363B dt=23694 wait=175 +20:29:51 step 79080 loss=3.1008 lr=1.05e-04 tok=10.365B dt=23769 wait=175 +20:31:06 step 79100 loss=3.1343 lr=1.05e-04 tok=10.368B dt=23844 wait=176 +20:32:21 step 79120 loss=3.2059 lr=1.05e-04 tok=10.370B dt=23919 wait=176 +20:33:36 step 79140 loss=3.2412 lr=1.05e-04 tok=10.373B dt=23994 wait=176 +20:34:51 step 79160 loss=3.1906 lr=1.05e-04 tok=10.376B dt=24069 wait=177 +20:36:06 step 79180 loss=3.1846 lr=1.05e-04 tok=10.378B dt=24145 wait=177 +20:37:21 step 79200 loss=3.2688 lr=1.05e-04 tok=10.381B dt=24220 wait=178 +20:37:34 pushed ckpt_w31.pt,log_w31.txt +20:41:58 step 79220 loss=3.1857 lr=1.05e-04 tok=10.384B dt=24496 wait=178 +20:43:13 step 79240 loss=3.1688 lr=1.05e-04 tok=10.386B dt=24571 wait=178 +20:44:28 step 79260 loss=3.1921 lr=1.05e-04 tok=10.389B dt=24646 wait=179 +20:45:42 step 79280 loss=3.0485 lr=1.05e-04 tok=10.391B dt=24721 wait=179 +20:46:57 step 79300 loss=3.1035 lr=1.05e-04 tok=10.394B dt=24796 wait=179 +20:48:12 step 79320 loss=3.1494 lr=1.05e-04 tok=10.397B dt=24871 wait=180 +20:49:27 step 79340 loss=3.1272 lr=1.05e-04 tok=10.399B dt=24946 wait=180 +20:50:17 pushed ckpt_w31.pt,log_w31.txt +20:50:17 waited 0s for global r=175 +20:50:25 global r=175 load 4.267 -> 4.265 +20:51:10 step 79360 loss=3.1270 lr=1.05e-04 tok=10.402B dt=25049 wait=181 +20:52:26 step 79380 loss=3.1745 lr=1.04e-04 tok=10.404B dt=25124 wait=181 +20:53:41 step 79400 loss=3.2212 lr=1.04e-04 tok=10.407B dt=25199 wait=181 +20:54:56 step 79420 loss=3.2404 lr=1.04e-04 tok=10.410B dt=25274 wait=182 +20:56:11 step 79440 loss=3.1679 lr=1.04e-04 tok=10.412B dt=25349 wait=182 +20:57:26 step 79460 loss=3.2167 lr=1.04e-04 tok=10.415B dt=25425 wait=183 +20:58:41 step 79480 loss=3.0925 lr=1.04e-04 tok=10.418B dt=25500 wait=183 +20:59:56 step 79500 loss=3.1339 lr=1.04e-04 tok=10.420B dt=25575 wait=183 +21:00:09 pushed ckpt_w31.pt,log_w31.txt +21:01:24 waited 76s for global r=176 +21:01:36 global r=176 load 4.279 -> 4.279 +21:02:57 step 79520 loss=3.1094 lr=1.04e-04 tok=10.423B dt=25756 wait=184 +21:04:12 step 79540 loss=3.1683 lr=1.04e-04 tok=10.425B dt=25831 wait=184 +21:05:28 step 79560 loss=3.1412 lr=1.04e-04 tok=10.428B dt=25906 wait=185 +21:06:43 step 79580 loss=3.2089 lr=1.04e-04 tok=10.431B dt=25981 wait=185 +21:07:58 step 79600 loss=3.2034 lr=1.04e-04 tok=10.433B dt=26056 wait=185 +21:09:13 step 79620 loss=3.1276 lr=1.04e-04 tok=10.436B dt=26131 wait=186 +21:10:28 step 79640 loss=3.1772 lr=1.04e-04 tok=10.439B dt=26206 wait=186 +21:11:18 pushed ckpt_w31.pt,log_w31.txt +21:12:19 waited 61s for global r=177 +21:12:29 global r=177 load 4.265 -> 4.265 +21:13:12 step 79660 loss=3.1484 lr=1.04e-04 tok=10.441B dt=26371 wait=187 +21:14:27 step 79680 loss=3.2101 lr=1.04e-04 tok=10.444B dt=26446 wait=187 +21:15:42 step 79700 loss=3.2238 lr=1.04e-04 tok=10.446B dt=26521 wait=187 +21:16:58 step 79720 loss=3.2605 lr=1.04e-04 tok=10.449B dt=26596 wait=188 +21:18:13 step 79740 loss=3.1964 lr=1.04e-04 tok=10.452B dt=26671 wait=188 +21:19:28 step 79760 loss=3.2756 lr=1.04e-04 tok=10.454B dt=26746 wait=189 +21:20:43 step 79780 loss=3.1109 lr=1.04e-04 tok=10.457B dt=26821 wait=189 +21:21:58 step 79800 loss=3.1713 lr=1.04e-04 tok=10.460B dt=26896 wait=189 +21:22:10 pushed ckpt_w31.pt,log_w31.txt +21:23:26 waited 76s for global r=178 +21:23:37 global r=178 load 4.262 -> 4.262 +21:24:55 step 79820 loss=3.1187 lr=1.04e-04 tok=10.462B dt=27074 wait=190 +21:26:10 step 79840 loss=3.1435 lr=1.04e-04 tok=10.465B dt=27149 wait=190 +21:27:25 step 79860 loss=3.1438 lr=1.04e-04 tok=10.467B dt=27224 wait=191 +21:28:40 step 79880 loss=3.1888 lr=1.04e-04 tok=10.470B dt=27299 wait=191 +21:29:55 step 79900 loss=3.1737 lr=1.04e-04 tok=10.473B dt=27374 wait=191 +21:31:11 step 79920 loss=3.1532 lr=1.04e-04 tok=10.475B dt=27449 wait=192 +21:32:26 step 79940 loss=3.1808 lr=1.04e-04 tok=10.478B dt=27524 wait=192 +21:33:16 pushed ckpt_w31.pt,log_w31.txt +21:37:01 step 79960 loss=3.1623 lr=1.04e-04 tok=10.481B dt=27800 wait=193 +21:38:16 step 79980 loss=3.2305 lr=1.04e-04 tok=10.483B dt=27875 wait=193 +21:39:32 step 80000 loss=3.1295 lr=1.04e-04 tok=10.486B dt=27950 wait=193 +21:40:47 step 80020 loss=3.0917 lr=1.04e-04 tok=10.488B dt=28025 wait=194 +21:42:02 step 80040 loss=3.1284 lr=1.04e-04 tok=10.491B dt=28100 wait=194 +21:43:16 step 80060 loss=3.1283 lr=1.04e-04 tok=10.494B dt=28175 wait=194 +21:44:31 step 80080 loss=3.1767 lr=1.04e-04 tok=10.496B dt=28250 wait=195 +21:45:47 step 80100 loss=3.1300 lr=1.04e-04 tok=10.499B dt=28325 wait=195 +21:46:00 pushed ckpt_w31.pt,log_w31.txt +21:46:00 waited 0s for global r=179 +21:46:07 global r=179 load 4.270 -> 4.255 +21:47:26 step 80120 loss=3.1080 lr=1.04e-04 tok=10.501B dt=28425 wait=196 +21:48:41 step 80140 loss=3.1401 lr=1.04e-04 tok=10.504B dt=28500 wait=196 +21:49:57 step 80160 loss=3.1594 lr=1.04e-04 tok=10.507B dt=28575 wait=196 +21:51:12 step 80180 loss=3.2244 lr=1.04e-04 tok=10.509B dt=28650 wait=197 +21:52:27 step 80200 loss=3.2039 lr=1.04e-04 tok=10.512B dt=28725 wait=197 +21:53:42 step 80220 loss=3.1949 lr=1.04e-04 tok=10.515B dt=28800 wait=198 +21:54:57 step 80240 loss=3.1880 lr=1.04e-04 tok=10.517B dt=28876 wait=198 +21:55:47 pushed ckpt_w31.pt,log_w31.txt +21:56:03 waited 15s for global r=180 +21:56:09 global r=180 load 4.262 -> 4.253 +21:56:54 step 80260 loss=3.1418 lr=1.04e-04 tok=10.520B dt=28992 wait=198 +21:58:09 step 80280 loss=3.1089 lr=1.04e-04 tok=10.522B dt=29067 wait=199 +21:59:24 step 80300 loss=3.1636 lr=1.04e-04 tok=10.525B dt=29142 wait=199 +22:00:39 step 80320 loss=3.2084 lr=1.04e-04 tok=10.528B dt=29217 wait=200 +22:01:54 step 80340 loss=3.0824 lr=1.04e-04 tok=10.530B dt=29292 wait=200 +22:03:09 step 80360 loss=3.1949 lr=1.04e-04 tok=10.533B dt=29367 wait=200 +22:04:24 step 80380 loss=3.0777 lr=1.03e-04 tok=10.536B dt=29442 wait=201 +22:05:39 step 80400 loss=3.1324 lr=1.03e-04 tok=10.538B dt=29517 wait=201 +22:05:56 pushed ckpt_w31.pt,log_w31.txt +22:10:18 step 80420 loss=3.1374 lr=1.03e-04 tok=10.541B dt=29796 wait=202 +22:11:33 step 80440 loss=3.0954 lr=1.03e-04 tok=10.543B dt=29871 wait=202 +22:12:48 step 80460 loss=3.1261 lr=1.03e-04 tok=10.546B dt=29946 wait=202 +22:14:03 step 80480 loss=3.1291 lr=1.03e-04 tok=10.549B dt=30021 wait=203 +22:15:18 step 80500 loss=3.1060 lr=1.03e-04 tok=10.551B dt=30096 wait=203 +22:16:33 step 80520 loss=3.1772 lr=1.03e-04 tok=10.554B dt=30171 wait=203 +22:17:48 step 80540 loss=3.1761 lr=1.03e-04 tok=10.557B dt=30246 wait=204 +22:18:37 pushed ckpt_w31.pt,log_w31.txt +22:18:38 waited 0s for global r=181 +22:18:45 global r=181 load 4.262 -> 4.244 +22:19:31 step 80560 loss=3.1589 lr=1.03e-04 tok=10.559B dt=30349 wait=204 +22:20:46 step 80580 loss=3.1280 lr=1.03e-04 tok=10.562B dt=30424 wait=205 +22:22:01 step 80600 loss=3.1732 lr=1.03e-04 tok=10.564B dt=30499 wait=205 +22:23:16 step 80620 loss=3.1906 lr=1.03e-04 tok=10.567B dt=30574 wait=205 +22:24:31 step 80640 loss=3.0973 lr=1.03e-04 tok=10.570B dt=30649 wait=206 +22:25:46 step 80660 loss=3.2023 lr=1.03e-04 tok=10.572B dt=30724 wait=206 +22:27:01 step 80680 loss=3.0984 lr=1.03e-04 tok=10.575B dt=30799 wait=206 +22:28:16 step 80700 loss=3.1357 lr=1.03e-04 tok=10.578B dt=30874 wait=207 +22:28:29 pushed ckpt_w31.pt,log_w31.txt +22:29:14 waited 45s for global r=182 +22:29:21 global r=182 load 4.260 -> 4.239 +22:30:47 step 80720 loss=3.1074 lr=1.03e-04 tok=10.580B dt=31026 wait=207 +22:32:03 step 80740 loss=3.1542 lr=1.03e-04 tok=10.583B dt=31101 wait=208 +22:33:18 step 80760 loss=3.1741 lr=1.03e-04 tok=10.585B dt=31176 wait=208 +22:34:33 step 80780 loss=3.2235 lr=1.03e-04 tok=10.588B dt=31251 wait=208 +22:35:48 step 80800 loss=3.1948 lr=1.03e-04 tok=10.591B dt=31326 wait=209 +22:37:03 step 80820 loss=3.1204 lr=1.03e-04 tok=10.593B dt=31401 wait=209 +22:38:18 step 80840 loss=3.1815 lr=1.03e-04 tok=10.596B dt=31476 wait=210 +22:39:08 pushed ckpt_w31.pt,log_w31.txt +22:42:51 step 80860 loss=3.0919 lr=1.03e-04 tok=10.598B dt=31749 wait=210 +22:44:06 step 80880 loss=3.1358 lr=1.03e-04 tok=10.601B dt=31824 wait=210 +22:45:21 step 80900 loss=3.0777 lr=1.03e-04 tok=10.604B dt=31899 wait=211 +22:46:36 step 80920 loss=3.0743 lr=1.03e-04 tok=10.606B dt=31974 wait=211 +22:47:52 step 80940 loss=3.1204 lr=1.03e-04 tok=10.609B dt=32050 wait=212 +22:49:07 step 80960 loss=3.1294 lr=1.03e-04 tok=10.612B dt=32125 wait=212 +22:50:21 step 80980 loss=3.1763 lr=1.03e-04 tok=10.614B dt=32200 wait=212 +22:51:37 step 81000 loss=3.1312 lr=1.03e-04 tok=10.617B dt=32275 wait=213 +22:51:52 pushed ckpt_w31.pt,log_w31.txt +22:51:52 waited 0s for global r=183 +22:51:58 global r=183 load 4.260 -> 4.247 +22:53:19 step 81020 loss=3.1437 lr=1.03e-04 tok=10.619B dt=32378 wait=213 +22:54:35 step 81040 loss=3.1503 lr=1.03e-04 tok=10.622B dt=32453 wait=214 +22:55:50 step 81060 loss=3.1707 lr=1.03e-04 tok=10.625B dt=32528 wait=214 +22:57:05 step 81080 loss=3.1968 lr=1.03e-04 tok=10.627B dt=32603 wait=214 +22:58:20 step 81100 loss=3.2013 lr=1.03e-04 tok=10.630B dt=32678 wait=215 +22:59:35 step 81120 loss=3.1011 lr=1.03e-04 tok=10.633B dt=32753 wait=215 +23:00:50 step 81140 loss=3.1635 lr=1.03e-04 tok=10.635B dt=32828 wait=215 +23:01:42 pushed ckpt_w31.pt,log_w31.txt +23:02:43 waited 61s for global r=184 +23:02:52 global r=184 load 4.257 -> 4.242 +23:03:36 step 81160 loss=3.0824 lr=1.03e-04 tok=10.638B dt=32994 wait=216 +23:04:51 step 81180 loss=3.0777 lr=1.03e-04 tok=10.640B dt=33069 wait=216 +23:06:06 step 81200 loss=3.1505 lr=1.03e-04 tok=10.643B dt=33144 wait=217 +23:07:21 step 81220 loss=3.1958 lr=1.03e-04 tok=10.646B dt=33219 wait=217 +23:08:36 step 81240 loss=3.1535 lr=1.03e-04 tok=10.648B dt=33294 wait=217 +23:09:51 step 81260 loss=3.2443 lr=1.03e-04 tok=10.651B dt=33369 wait=218 +23:11:06 step 81280 loss=3.1237 lr=1.03e-04 tok=10.654B dt=33445 wait=218 +23:12:21 step 81300 loss=3.1641 lr=1.03e-04 tok=10.656B dt=33520 wait=219 +23:12:34 pushed ckpt_w31.pt,log_w31.txt +23:16:54 step 81320 loss=3.1645 lr=1.03e-04 tok=10.659B dt=33792 wait=219 +23:18:09 step 81340 loss=3.0987 lr=1.03e-04 tok=10.661B dt=33867 wait=219 +23:19:24 step 81360 loss=3.1184 lr=1.02e-04 tok=10.664B dt=33942 wait=220 +23:20:39 step 81380 loss=3.1298 lr=1.02e-04 tok=10.667B dt=34017 wait=220 +23:21:54 step 81400 loss=3.0761 lr=1.02e-04 tok=10.669B dt=34093 wait=221 +23:23:09 step 81420 loss=3.1851 lr=1.02e-04 tok=10.672B dt=34168 wait=221 +23:24:24 step 81440 loss=3.1144 lr=1.02e-04 tok=10.675B dt=34243 wait=221 +23:25:14 pushed ckpt_w31.pt,log_w31.txt +23:25:14 waited 0s for global r=185 +23:25:23 global r=185 load 4.268 -> 4.242 +23:26:06 step 81460 loss=3.1157 lr=1.02e-04 tok=10.677B dt=34344 wait=222 +23:27:21 step 81480 loss=3.1235 lr=1.02e-04 tok=10.680B dt=34419 wait=222 +23:28:36 step 81500 loss=3.1399 lr=1.02e-04 tok=10.682B dt=34495 wait=223 +23:29:51 step 81520 loss=3.2064 lr=1.02e-04 tok=10.685B dt=34570 wait=223 +23:31:07 step 81540 loss=3.1367 lr=1.02e-04 tok=10.688B dt=34645 wait=223 +23:32:22 step 81560 loss=3.2566 lr=1.02e-04 tok=10.690B dt=34720 wait=224 +23:33:37 step 81580 loss=3.1360 lr=1.02e-04 tok=10.693B dt=34795 wait=224 +23:34:52 step 81600 loss=3.1920 lr=1.02e-04 tok=10.695B dt=34871 wait=225 +23:35:06 pushed ckpt_w31.pt,log_w31.txt +23:36:22 waited 76s for global r=186 +23:36:28 global r=186 load 4.249 -> 4.236 +23:37:50 step 81620 loss=3.0871 lr=1.02e-04 tok=10.698B dt=35048 wait=225 +23:39:05 step 81640 loss=3.1140 lr=1.02e-04 tok=10.701B dt=35123 wait=225 +23:40:20 step 81660 loss=3.1269 lr=1.02e-04 tok=10.703B dt=35198 wait=226 +23:41:34 step 81680 loss=3.1571 lr=1.02e-04 tok=10.706B dt=35273 wait=226 +23:42:49 step 81700 loss=3.1566 lr=1.02e-04 tok=10.709B dt=35348 wait=226 +23:44:04 step 81720 loss=3.0755 lr=1.02e-04 tok=10.711B dt=35423 wait=227 +23:45:19 step 81740 loss=3.1388 lr=1.02e-04 tok=10.714B dt=35498 wait=227 +23:46:13 pushed ckpt_w31.pt,log_w31.txt +23:49:58 step 81760 loss=3.1323 lr=1.02e-04 tok=10.716B dt=35776 wait=228 +23:51:13 step 81780 loss=3.1986 lr=1.02e-04 tok=10.719B dt=35851 wait=228 +23:52:28 step 81800 loss=3.1565 lr=1.02e-04 tok=10.722B dt=35927 wait=228 +23:53:43 step 81820 loss=3.1123 lr=1.02e-04 tok=10.724B dt=36002 wait=229 +23:54:59 step 81840 loss=3.1422 lr=1.02e-04 tok=10.727B dt=36077 wait=229 +23:56:14 step 81860 loss=3.1068 lr=1.02e-04 tok=10.730B dt=36152 wait=230 +23:57:28 step 81880 loss=3.1183 lr=1.02e-04 tok=10.732B dt=36227 wait=230 +23:58:43 step 81900 loss=3.0862 lr=1.02e-04 tok=10.735B dt=36302 wait=230 +23:59:00 pushed ckpt_w31.pt,log_w31.txt +23:59:00 waited 0s for global r=187 +23:59:07 global r=187 load 4.261 -> 4.231 +00:00:29 step 81920 loss=3.0806 lr=1.02e-04 tok=10.737B dt=36407 wait=231 +00:01:43 step 81940 loss=3.1340 lr=1.02e-04 tok=10.740B dt=36482 wait=231 +00:02:59 step 81960 loss=3.1454 lr=1.02e-04 tok=10.743B dt=36557 wait=232 +00:04:14 step 81980 loss=3.2302 lr=1.02e-04 tok=10.745B dt=36632 wait=232 +00:05:29 step 82000 loss=3.1878 lr=1.02e-04 tok=10.748B dt=36708 wait=232 +00:06:44 step 82020 loss=3.1406 lr=1.02e-04 tok=10.751B dt=36783 wait=233 +00:07:59 step 82040 loss=3.1694 lr=1.02e-04 tok=10.753B dt=36858 wait=233 +00:08:51 pushed ckpt_w31.pt,log_w31.txt +00:09:37 waited 46s for global r=188 +00:09:46 global r=188 load 4.256 -> 4.232 +00:10:28 step 82060 loss=3.1244 lr=1.02e-04 tok=10.756B dt=37007 wait=234 +00:11:43 step 82080 loss=3.1161 lr=1.02e-04 tok=10.758B dt=37082 wait=234 +00:12:58 step 82100 loss=3.1419 lr=1.02e-04 tok=10.761B dt=37157 wait=234 +00:14:14 step 82120 loss=3.1848 lr=1.02e-04 tok=10.764B dt=37232 wait=235 +00:15:29 step 82140 loss=3.1351 lr=1.02e-04 tok=10.766B dt=37307 wait=235 +00:16:44 step 82160 loss=3.1953 lr=1.02e-04 tok=10.769B dt=37382 wait=235 +00:17:59 step 82180 loss=3.0724 lr=1.02e-04 tok=10.771B dt=37457 wait=236 +00:19:14 step 82200 loss=3.1539 lr=1.02e-04 tok=10.774B dt=37533 wait=236 +00:19:28 pushed ckpt_w31.pt,log_w31.txt +00:23:49 step 82220 loss=3.2419 lr=1.02e-04 tok=10.777B dt=37807 wait=237 +00:25:04 step 82240 loss=3.1712 lr=1.02e-04 tok=10.779B dt=37882 wait=237 +00:26:19 step 82260 loss=3.1322 lr=1.02e-04 tok=10.782B dt=37957 wait=237 +00:27:34 step 82280 loss=3.1166 lr=1.02e-04 tok=10.785B dt=38032 wait=238 +00:28:49 step 82300 loss=3.0829 lr=1.02e-04 tok=10.787B dt=38107 wait=238 +00:30:04 step 82320 loss=3.1323 lr=1.02e-04 tok=10.790B dt=38182 wait=239 +00:31:19 step 82340 loss=3.1103 lr=1.01e-04 tok=10.792B dt=38257 wait=239 +00:32:09 pushed ckpt_w31.pt,log_w31.txt +00:32:10 waited 0s for global r=189 +00:32:20 global r=189 load 4.256 -> 4.233 +00:33:03 step 82360 loss=3.0800 lr=1.01e-04 tok=10.795B dt=38361 wait=239 +00:34:18 step 82380 loss=3.0870 lr=1.01e-04 tok=10.798B dt=38436 wait=240 +00:35:33 step 82400 loss=3.1762 lr=1.01e-04 tok=10.800B dt=38511 wait=240 +00:36:48 step 82420 loss=3.2316 lr=1.01e-04 tok=10.803B dt=38587 wait=240 +00:38:03 step 82440 loss=3.1403 lr=1.01e-04 tok=10.806B dt=38662 wait=241 +00:39:19 step 82460 loss=3.2272 lr=1.01e-04 tok=10.808B dt=38737 wait=241 +00:40:34 step 82480 loss=3.0904 lr=1.01e-04 tok=10.811B dt=38812 wait=242 +00:41:49 step 82500 loss=3.1251 lr=1.01e-04 tok=10.813B dt=38887 wait=242 +00:42:01 pushed ckpt_w31.pt,log_w31.txt +00:42:47 waited 45s for global r=190 +00:42:54 global r=190 load 4.256 -> 4.238 +00:44:15 step 82520 loss=3.0884 lr=1.01e-04 tok=10.816B dt=39033 wait=242 +00:45:30 step 82540 loss=3.1410 lr=1.01e-04 tok=10.819B dt=39108 wait=243 +00:46:45 step 82560 loss=3.1242 lr=1.01e-04 tok=10.821B dt=39183 wait=243 +00:48:00 step 82580 loss=3.2061 lr=1.01e-04 tok=10.824B dt=39258 wait=243 +00:49:15 step 82600 loss=3.1212 lr=1.01e-04 tok=10.827B dt=39333 wait=244 +00:50:30 step 82620 loss=3.1228 lr=1.01e-04 tok=10.829B dt=39408 wait=244 +00:51:45 step 82640 loss=3.1747 lr=1.01e-04 tok=10.832B dt=39484 wait=245 diff --git a/log_w32.txt b/log_w32.txt new file mode 100644 index 0000000000000000000000000000000000000000..3655a50935411cb60c8f16d1dcc60786030130fb --- /dev/null +++ b/log_w32.txt @@ -0,0 +1,221 @@ +launch +21:25:22 name=w32 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:25:28 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:25:38 params 1.08B +21:25:40 resumed step=79800 seen=10.46B round=178 +21:25:40 stream salt=79800 +21:27:45 start: accum=4, 131072 tokens per optimiser step +21:29:18 step 79801 loss=3.2210 lr=1.04e-04 tok=10.460B dt=93 wait=62 +21:29:23 step 79802 loss=3.1065 lr=1.04e-04 tok=10.460B dt=98 wait=62 +21:30:44 step 79820 loss=3.2572 lr=1.04e-04 tok=10.462B dt=179 wait=62 +21:32:04 step 79840 loss=3.2302 lr=1.04e-04 tok=10.465B dt=259 wait=63 +21:33:19 step 79860 loss=3.2755 lr=1.04e-04 tok=10.467B dt=334 wait=63 +21:34:34 step 79880 loss=3.1904 lr=1.04e-04 tok=10.470B dt=409 wait=63 +21:35:49 step 79900 loss=3.2116 lr=1.04e-04 tok=10.473B dt=484 wait=64 +21:37:03 step 79920 loss=3.1241 lr=1.04e-04 tok=10.475B dt=558 wait=64 +21:38:18 step 79940 loss=3.1228 lr=1.04e-04 tok=10.478B dt=633 wait=64 +21:39:10 pushed ckpt_w32.pt,log_w32.txt +21:40:11 waited 61s for global r=179 +21:40:24 global r=179 load 4.286 -> 4.255 +21:41:06 step 79960 loss=3.1971 lr=1.04e-04 tok=10.481B dt=801 wait=65 +21:42:21 step 79980 loss=3.0861 lr=1.04e-04 tok=10.483B dt=876 wait=65 +21:43:36 step 80000 loss=3.1545 lr=1.04e-04 tok=10.486B dt=951 wait=66 +21:44:51 step 80020 loss=3.1750 lr=1.04e-04 tok=10.488B dt=1025 wait=66 +21:46:05 step 80040 loss=3.0650 lr=1.04e-04 tok=10.491B dt=1100 wait=66 +21:47:20 step 80060 loss=3.1864 lr=1.04e-04 tok=10.494B dt=1175 wait=67 +21:48:35 step 80080 loss=3.0961 lr=1.04e-04 tok=10.496B dt=1250 wait=67 +21:49:50 step 80100 loss=3.1656 lr=1.04e-04 tok=10.499B dt=1325 wait=67 +21:50:03 pushed ckpt_w32.pt,log_w32.txt +21:54:24 step 80120 loss=3.2026 lr=1.04e-04 tok=10.501B dt=1599 wait=68 +21:55:39 step 80140 loss=3.1625 lr=1.04e-04 tok=10.504B dt=1674 wait=68 +21:56:54 step 80160 loss=3.1410 lr=1.04e-04 tok=10.507B dt=1749 wait=69 +21:58:08 step 80180 loss=3.0930 lr=1.04e-04 tok=10.509B dt=1823 wait=69 +21:59:23 step 80200 loss=3.0661 lr=1.04e-04 tok=10.512B dt=1898 wait=69 +22:00:38 step 80220 loss=3.1674 lr=1.04e-04 tok=10.515B dt=1973 wait=70 +22:01:53 step 80240 loss=3.1551 lr=1.04e-04 tok=10.517B dt=2048 wait=70 +22:02:45 pushed ckpt_w32.pt,log_w32.txt +22:02:45 waited 0s for global r=180 +22:02:52 global r=180 load 4.280 -> 4.253 +22:03:34 step 80260 loss=3.1052 lr=1.04e-04 tok=10.520B dt=2149 wait=70 +22:04:49 step 80280 loss=3.1313 lr=1.04e-04 tok=10.522B dt=2224 wait=71 +22:06:04 step 80300 loss=3.2418 lr=1.04e-04 tok=10.525B dt=2299 wait=71 +22:07:19 step 80320 loss=3.2306 lr=1.04e-04 tok=10.528B dt=2374 wait=72 +22:08:34 step 80340 loss=3.1443 lr=1.04e-04 tok=10.530B dt=2449 wait=72 +22:09:49 step 80360 loss=3.2203 lr=1.04e-04 tok=10.533B dt=2524 wait=72 +22:11:04 step 80380 loss=3.0985 lr=1.03e-04 tok=10.536B dt=2598 wait=73 +22:12:18 step 80400 loss=3.1411 lr=1.03e-04 tok=10.538B dt=2673 wait=73 +22:12:33 pushed ckpt_w32.pt,log_w32.txt +22:13:49 waited 76s for global r=181 +22:13:57 global r=181 load 4.267 -> 4.244 +22:15:19 step 80420 loss=3.1215 lr=1.03e-04 tok=10.541B dt=2854 wait=73 +22:16:34 step 80440 loss=3.1409 lr=1.03e-04 tok=10.543B dt=2929 wait=74 +22:17:49 step 80460 loss=3.1552 lr=1.03e-04 tok=10.546B dt=3004 wait=74 +22:19:04 step 80480 loss=3.1881 lr=1.03e-04 tok=10.549B dt=3078 wait=74 +22:20:18 step 80500 loss=3.1784 lr=1.03e-04 tok=10.551B dt=3153 wait=75 +22:21:33 step 80520 loss=3.1314 lr=1.03e-04 tok=10.554B dt=3228 wait=75 +22:22:48 step 80540 loss=3.1583 lr=1.03e-04 tok=10.557B dt=3303 wait=76 +22:23:38 pushed ckpt_w32.pt,log_w32.txt +22:27:21 step 80560 loss=3.1503 lr=1.03e-04 tok=10.559B dt=3576 wait=76 +22:28:36 step 80580 loss=3.1766 lr=1.03e-04 tok=10.562B dt=3651 wait=76 +22:29:51 step 80600 loss=3.1145 lr=1.03e-04 tok=10.564B dt=3726 wait=77 +22:31:06 step 80620 loss=3.0944 lr=1.03e-04 tok=10.567B dt=3801 wait=77 +22:32:21 step 80640 loss=3.1251 lr=1.03e-04 tok=10.570B dt=3876 wait=77 +22:33:36 step 80660 loss=3.1154 lr=1.03e-04 tok=10.572B dt=3951 wait=78 +22:34:50 step 80680 loss=3.1184 lr=1.03e-04 tok=10.575B dt=4025 wait=78 +22:36:05 step 80700 loss=3.1383 lr=1.03e-04 tok=10.578B dt=4100 wait=78 +22:36:18 pushed ckpt_w32.pt,log_w32.txt +22:36:18 waited 0s for global r=182 +22:36:24 global r=182 load 4.262 -> 4.239 +22:37:48 step 80720 loss=3.1269 lr=1.03e-04 tok=10.580B dt=4203 wait=79 +22:39:03 step 80740 loss=3.1803 lr=1.03e-04 tok=10.583B dt=4278 wait=79 +22:40:18 step 80760 loss=3.1731 lr=1.03e-04 tok=10.585B dt=4353 wait=80 +22:41:33 step 80780 loss=3.1814 lr=1.03e-04 tok=10.588B dt=4428 wait=80 +22:42:48 step 80800 loss=3.1851 lr=1.03e-04 tok=10.591B dt=4503 wait=80 +22:44:02 step 80820 loss=3.1093 lr=1.03e-04 tok=10.593B dt=4577 wait=81 +22:45:17 step 80840 loss=3.1644 lr=1.03e-04 tok=10.596B dt=4652 wait=81 +22:46:10 pushed ckpt_w32.pt,log_w32.txt +22:47:26 waited 76s for global r=183 +22:47:32 global r=183 load 4.272 -> 4.247 +22:48:15 step 80860 loss=3.0837 lr=1.03e-04 tok=10.598B dt=4830 wait=81 +22:49:30 step 80880 loss=3.1044 lr=1.03e-04 tok=10.601B dt=4905 wait=82 +22:50:45 step 80900 loss=3.1769 lr=1.03e-04 tok=10.604B dt=4980 wait=82 +22:52:00 step 80920 loss=3.1788 lr=1.03e-04 tok=10.606B dt=5054 wait=82 +22:53:14 step 80940 loss=3.1116 lr=1.03e-04 tok=10.609B dt=5129 wait=83 +22:54:29 step 80960 loss=3.2382 lr=1.03e-04 tok=10.612B dt=5204 wait=83 +22:55:44 step 80980 loss=3.0845 lr=1.03e-04 tok=10.614B dt=5279 wait=83 +22:56:59 step 81000 loss=3.1474 lr=1.03e-04 tok=10.617B dt=5354 wait=84 +22:57:12 pushed ckpt_w32.pt,log_w32.txt +23:01:36 step 81020 loss=3.1713 lr=1.03e-04 tok=10.619B dt=5631 wait=84 +23:02:51 step 81040 loss=3.1124 lr=1.03e-04 tok=10.622B dt=5706 wait=84 +23:04:06 step 81060 loss=3.1113 lr=1.03e-04 tok=10.625B dt=5781 wait=85 +23:05:21 step 81080 loss=3.0985 lr=1.03e-04 tok=10.627B dt=5856 wait=85 +23:06:35 step 81100 loss=3.0927 lr=1.03e-04 tok=10.630B dt=5930 wait=86 +23:07:50 step 81120 loss=3.2063 lr=1.03e-04 tok=10.633B dt=6005 wait=86 +23:09:05 step 81140 loss=3.1711 lr=1.03e-04 tok=10.635B dt=6080 wait=86 +23:09:58 pushed ckpt_w32.pt,log_w32.txt +23:09:58 waited 0s for global r=184 +23:10:04 global r=184 load 4.259 -> 4.242 +23:10:50 step 81160 loss=3.1640 lr=1.03e-04 tok=10.638B dt=6185 wait=87 +23:12:05 step 81180 loss=3.1504 lr=1.03e-04 tok=10.640B dt=6260 wait=87 +23:13:20 step 81200 loss=3.1938 lr=1.03e-04 tok=10.643B dt=6335 wait=87 +23:14:34 step 81220 loss=3.1837 lr=1.03e-04 tok=10.646B dt=6409 wait=88 +23:15:49 step 81240 loss=3.0849 lr=1.03e-04 tok=10.648B dt=6484 wait=88 +23:17:04 step 81260 loss=3.2027 lr=1.03e-04 tok=10.651B dt=6559 wait=88 +23:18:18 step 81280 loss=3.0608 lr=1.03e-04 tok=10.654B dt=6633 wait=89 +23:19:33 step 81300 loss=3.0846 lr=1.03e-04 tok=10.656B dt=6708 wait=89 +23:19:47 pushed ckpt_w32.pt,log_w32.txt +23:20:48 waited 61s for global r=185 +23:20:58 global r=185 load 4.259 -> 4.242 +23:22:22 step 81320 loss=3.0955 lr=1.03e-04 tok=10.659B dt=6877 wait=89 +23:23:36 step 81340 loss=3.1484 lr=1.03e-04 tok=10.661B dt=6951 wait=90 +23:24:52 step 81360 loss=3.1572 lr=1.02e-04 tok=10.664B dt=7026 wait=90 +23:26:06 step 81380 loss=3.2378 lr=1.02e-04 tok=10.667B dt=7101 wait=90 +23:27:22 step 81400 loss=3.2165 lr=1.02e-04 tok=10.669B dt=7177 wait=91 +23:28:36 step 81420 loss=3.1627 lr=1.02e-04 tok=10.672B dt=7251 wait=91 +23:29:51 step 81440 loss=3.1790 lr=1.02e-04 tok=10.675B dt=7326 wait=91 +23:30:45 pushed ckpt_w32.pt,log_w32.txt +23:34:31 step 81460 loss=3.1279 lr=1.02e-04 tok=10.677B dt=7606 wait=92 +23:35:46 step 81480 loss=3.1713 lr=1.02e-04 tok=10.680B dt=7681 wait=92 +23:37:01 step 81500 loss=3.0536 lr=1.02e-04 tok=10.682B dt=7756 wait=93 +23:38:16 step 81520 loss=3.0366 lr=1.02e-04 tok=10.685B dt=7831 wait=93 +23:39:31 step 81540 loss=3.0925 lr=1.02e-04 tok=10.688B dt=7906 wait=93 +23:40:45 step 81560 loss=3.1381 lr=1.02e-04 tok=10.690B dt=7980 wait=94 +23:42:00 step 81580 loss=3.2027 lr=1.02e-04 tok=10.693B dt=8055 wait=94 +23:43:15 step 81600 loss=3.1721 lr=1.02e-04 tok=10.695B dt=8130 wait=94 +23:43:28 pushed ckpt_w32.pt,log_w32.txt +23:43:28 waited 0s for global r=186 +23:43:34 global r=186 load 4.258 -> 4.236 +23:44:58 step 81620 loss=3.1381 lr=1.02e-04 tok=10.698B dt=8233 wait=95 +23:46:13 step 81640 loss=3.1775 lr=1.02e-04 tok=10.701B dt=8308 wait=95 +23:47:28 step 81660 loss=3.1369 lr=1.02e-04 tok=10.703B dt=8383 wait=95 +23:48:43 step 81680 loss=3.1774 lr=1.02e-04 tok=10.706B dt=8458 wait=96 +23:49:57 step 81700 loss=3.1649 lr=1.02e-04 tok=10.709B dt=8532 wait=96 +23:51:12 step 81720 loss=3.1103 lr=1.02e-04 tok=10.711B dt=8607 wait=96 +23:52:27 step 81740 loss=3.1587 lr=1.02e-04 tok=10.714B dt=8682 wait=97 +23:53:17 pushed ckpt_w32.pt,log_w32.txt +23:54:17 waited 61s for global r=187 +23:54:24 global r=187 load 4.254 -> 4.231 +23:55:05 step 81760 loss=3.1268 lr=1.02e-04 tok=10.716B dt=8840 wait=97 +23:56:20 step 81780 loss=3.1150 lr=1.02e-04 tok=10.719B dt=8915 wait=98 +23:57:35 step 81800 loss=3.2089 lr=1.02e-04 tok=10.722B dt=8990 wait=98 +23:58:50 step 81820 loss=3.2244 lr=1.02e-04 tok=10.724B dt=9065 wait=98 +00:00:05 step 81840 loss=3.1480 lr=1.02e-04 tok=10.727B dt=9140 wait=99 +00:01:20 step 81860 loss=3.2096 lr=1.02e-04 tok=10.730B dt=9215 wait=99 +00:02:36 step 81880 loss=3.1040 lr=1.02e-04 tok=10.732B dt=9290 wait=99 +00:03:50 step 81900 loss=3.1159 lr=1.02e-04 tok=10.735B dt=9365 wait=100 +00:04:05 pushed ckpt_w32.pt,log_w32.txt +00:08:27 step 81920 loss=3.1507 lr=1.02e-04 tok=10.737B dt=9641 wait=100 +00:09:41 step 81940 loss=3.0762 lr=1.02e-04 tok=10.740B dt=9716 wait=100 +00:10:56 step 81960 loss=3.0720 lr=1.02e-04 tok=10.743B dt=9791 wait=101 +00:12:11 step 81980 loss=3.1147 lr=1.02e-04 tok=10.745B dt=9866 wait=101 +00:13:26 step 82000 loss=3.0849 lr=1.02e-04 tok=10.748B dt=9941 wait=102 +00:14:41 step 82020 loss=3.1540 lr=1.02e-04 tok=10.751B dt=10016 wait=102 +00:15:56 step 82040 loss=3.1252 lr=1.02e-04 tok=10.753B dt=10091 wait=102 +00:16:52 pushed ckpt_w32.pt,log_w32.txt +00:16:52 waited 0s for global r=188 +00:17:01 global r=188 load 4.264 -> 4.232 +00:17:44 step 82060 loss=3.1310 lr=1.02e-04 tok=10.756B dt=10199 wait=103 +00:18:59 step 82080 loss=3.1136 lr=1.02e-04 tok=10.758B dt=10274 wait=103 +00:20:14 step 82100 loss=3.1832 lr=1.02e-04 tok=10.761B dt=10349 wait=103 +00:21:29 step 82120 loss=3.2116 lr=1.02e-04 tok=10.764B dt=10424 wait=104 +00:22:44 step 82140 loss=3.1595 lr=1.02e-04 tok=10.766B dt=10499 wait=104 +00:23:59 step 82160 loss=3.2335 lr=1.02e-04 tok=10.769B dt=10574 wait=104 +00:25:14 step 82180 loss=3.0995 lr=1.02e-04 tok=10.771B dt=10649 wait=105 +00:26:29 step 82200 loss=3.1607 lr=1.02e-04 tok=10.774B dt=10723 wait=105 +00:26:41 pushed ckpt_w32.pt,log_w32.txt +00:28:12 waited 91s for global r=189 +00:28:21 global r=189 load 4.249 -> 4.233 +00:29:45 step 82220 loss=3.1121 lr=1.02e-04 tok=10.777B dt=10920 wait=105 +00:31:00 step 82240 loss=3.1171 lr=1.02e-04 tok=10.779B dt=10995 wait=106 +00:32:15 step 82260 loss=3.1248 lr=1.02e-04 tok=10.782B dt=11070 wait=106 +00:33:29 step 82280 loss=3.1653 lr=1.02e-04 tok=10.785B dt=11144 wait=107 +00:34:44 step 82300 loss=3.1453 lr=1.02e-04 tok=10.787B dt=11219 wait=107 +00:35:59 step 82320 loss=3.0845 lr=1.02e-04 tok=10.790B dt=11294 wait=107 +00:37:14 step 82340 loss=3.1495 lr=1.01e-04 tok=10.792B dt=11369 wait=108 +00:38:04 pushed ckpt_w32.pt,log_w32.txt +00:41:49 step 82360 loss=3.1178 lr=1.01e-04 tok=10.795B dt=11644 wait=108 +00:43:04 step 82380 loss=3.1533 lr=1.01e-04 tok=10.798B dt=11719 wait=108 +00:44:19 step 82400 loss=3.0828 lr=1.01e-04 tok=10.800B dt=11794 wait=109 +00:45:34 step 82420 loss=3.0540 lr=1.01e-04 tok=10.803B dt=11869 wait=109 +00:46:49 step 82440 loss=3.1297 lr=1.01e-04 tok=10.806B dt=11944 wait=109 +00:48:04 step 82460 loss=3.1516 lr=1.01e-04 tok=10.808B dt=12019 wait=110 +00:49:19 step 82480 loss=3.1967 lr=1.01e-04 tok=10.811B dt=12094 wait=110 +00:50:34 step 82500 loss=3.1148 lr=1.01e-04 tok=10.813B dt=12169 wait=110 +00:50:47 pushed ckpt_w32.pt,log_w32.txt +00:50:47 waited 0s for global r=190 +00:50:56 global r=190 load 4.263 -> 4.238 +00:52:20 step 82520 loss=3.0831 lr=1.01e-04 tok=10.816B dt=12275 wait=111 +00:53:35 step 82540 loss=3.1311 lr=1.01e-04 tok=10.819B dt=12350 wait=111 +00:54:50 step 82560 loss=3.1522 lr=1.01e-04 tok=10.821B dt=12425 wait=111 +00:56:05 step 82580 loss=3.1686 lr=1.01e-04 tok=10.824B dt=12500 wait=112 +00:57:20 step 82600 loss=3.1456 lr=1.01e-04 tok=10.827B dt=12575 wait=112 +00:58:35 step 82620 loss=3.1370 lr=1.01e-04 tok=10.829B dt=12650 wait=112 +00:59:50 step 82640 loss=3.1861 lr=1.01e-04 tok=10.832B dt=12724 wait=113 +01:00:40 pushed ckpt_w32.pt,log_w32.txt +01:01:56 waited 76s for global r=191 +01:02:03 global r=191 load 4.254 -> 4.234 +01:02:47 step 82660 loss=3.1266 lr=1.01e-04 tok=10.834B dt=12902 wait=113 +01:04:02 step 82680 loss=3.1480 lr=1.01e-04 tok=10.837B dt=12977 wait=114 +01:05:17 step 82700 loss=3.2016 lr=1.01e-04 tok=10.840B dt=13052 wait=114 +01:06:32 step 82720 loss=3.2375 lr=1.01e-04 tok=10.842B dt=13127 wait=114 +01:07:47 step 82740 loss=3.1334 lr=1.01e-04 tok=10.845B dt=13202 wait=115 +01:09:02 step 82760 loss=3.1807 lr=1.01e-04 tok=10.848B dt=13277 wait=115 +01:10:16 step 82780 loss=3.0596 lr=1.01e-04 tok=10.850B dt=13351 wait=115 +01:11:31 step 82800 loss=3.1484 lr=1.01e-04 tok=10.853B dt=13426 wait=116 +01:11:45 pushed ckpt_w32.pt,log_w32.txt +01:16:08 step 82820 loss=3.1263 lr=1.01e-04 tok=10.855B dt=13703 wait=116 +01:17:23 step 82840 loss=3.0581 lr=1.01e-04 tok=10.858B dt=13778 wait=116 +01:18:37 step 82860 loss=3.1106 lr=1.01e-04 tok=10.861B dt=13852 wait=117 +01:19:52 step 82880 loss=3.1346 lr=1.01e-04 tok=10.863B dt=13927 wait=117 +01:21:07 step 82900 loss=3.0594 lr=1.01e-04 tok=10.866B dt=14002 wait=117 +01:22:22 step 82920 loss=3.1794 lr=1.01e-04 tok=10.868B dt=14077 wait=118 +01:23:36 step 82940 loss=3.0895 lr=1.01e-04 tok=10.871B dt=14151 wait=118 +01:24:26 pushed ckpt_w32.pt,log_w32.txt +01:28:10 step 82960 loss=3.0885 lr=1.01e-04 tok=10.874B dt=14424 wait=118 +01:29:24 step 82980 loss=3.1136 lr=1.01e-04 tok=10.876B dt=14499 wait=119 +01:30:39 step 83000 loss=3.1534 lr=1.01e-04 tok=10.879B dt=14574 wait=119 +01:31:54 step 83020 loss=3.1512 lr=1.01e-04 tok=10.882B dt=14649 wait=119 +01:33:09 step 83040 loss=3.1574 lr=1.01e-04 tok=10.884B dt=14724 wait=120 +01:34:24 step 83060 loss=3.1370 lr=1.01e-04 tok=10.887B dt=14799 wait=120 +01:35:39 step 83080 loss=3.2188 lr=1.01e-04 tok=10.889B dt=14874 wait=120 +01:36:54 step 83100 loss=3.1012 lr=1.01e-04 tok=10.892B dt=14949 wait=121 diff --git a/log_w33.txt b/log_w33.txt new file mode 100644 index 0000000000000000000000000000000000000000..f2952a6b25a2791510e286bccc4313db5fc4a14a --- /dev/null +++ b/log_w33.txt @@ -0,0 +1,221 @@ +launch +21:25:30 name=w33 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:25:36 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:25:45 params 1.08B +21:25:46 resumed step=79800 seen=10.46B round=178 +21:25:46 stream salt=79800 +21:27:51 start: accum=4, 131072 tokens per optimiser step +21:29:27 step 79801 loss=3.1954 lr=1.04e-04 tok=10.460B dt=96 wait=66 +21:29:31 step 79802 loss=3.0814 lr=1.04e-04 tok=10.460B dt=101 wait=66 +21:30:56 step 79820 loss=3.2653 lr=1.04e-04 tok=10.462B dt=186 wait=66 +21:32:16 step 79840 loss=3.2130 lr=1.04e-04 tok=10.465B dt=265 wait=66 +21:33:31 step 79860 loss=3.2578 lr=1.04e-04 tok=10.467B dt=340 wait=67 +21:34:46 step 79880 loss=3.1572 lr=1.04e-04 tok=10.470B dt=415 wait=67 +21:36:01 step 79900 loss=3.2340 lr=1.04e-04 tok=10.473B dt=490 wait=67 +21:37:16 step 79920 loss=3.1703 lr=1.04e-04 tok=10.475B dt=565 wait=68 +21:38:30 step 79940 loss=3.1965 lr=1.04e-04 tok=10.478B dt=640 wait=68 +21:39:22 pushed ckpt_w33.pt,log_w33.txt +21:40:07 waited 45s for global r=179 +21:40:21 global r=179 load 4.280 -> 4.255 +21:41:04 step 79960 loss=3.2214 lr=1.04e-04 tok=10.481B dt=793 wait=68 +21:42:18 step 79980 loss=3.1146 lr=1.04e-04 tok=10.483B dt=867 wait=69 +21:43:33 step 80000 loss=3.1592 lr=1.04e-04 tok=10.486B dt=942 wait=69 +21:44:48 step 80020 loss=3.1923 lr=1.04e-04 tok=10.488B dt=1017 wait=69 +21:46:02 step 80040 loss=3.1173 lr=1.04e-04 tok=10.491B dt=1092 wait=70 +21:47:17 step 80060 loss=3.2009 lr=1.04e-04 tok=10.494B dt=1167 wait=70 +21:48:32 step 80080 loss=3.0877 lr=1.04e-04 tok=10.496B dt=1241 wait=71 +21:49:47 step 80100 loss=3.1409 lr=1.04e-04 tok=10.499B dt=1317 wait=71 +21:50:00 pushed ckpt_w33.pt,log_w33.txt +21:54:22 step 80120 loss=3.2226 lr=1.04e-04 tok=10.501B dt=1591 wait=71 +21:55:37 step 80140 loss=3.1723 lr=1.04e-04 tok=10.504B dt=1666 wait=72 +21:56:52 step 80160 loss=3.1772 lr=1.04e-04 tok=10.507B dt=1741 wait=72 +21:58:07 step 80180 loss=3.1592 lr=1.04e-04 tok=10.509B dt=1816 wait=72 +21:59:21 step 80200 loss=3.0749 lr=1.04e-04 tok=10.512B dt=1891 wait=73 +22:00:36 step 80220 loss=3.1682 lr=1.04e-04 tok=10.515B dt=1965 wait=73 +22:01:51 step 80240 loss=3.1113 lr=1.04e-04 tok=10.517B dt=2040 wait=73 +22:02:41 pushed ckpt_w33.pt,log_w33.txt +22:02:41 waited 1s for global r=180 +22:02:51 global r=180 load 4.267 -> 4.253 +22:03:35 step 80260 loss=3.1249 lr=1.04e-04 tok=10.520B dt=2145 wait=74 +22:04:50 step 80280 loss=3.1287 lr=1.04e-04 tok=10.522B dt=2219 wait=74 +22:06:05 step 80300 loss=3.1710 lr=1.04e-04 tok=10.525B dt=2294 wait=74 +22:07:20 step 80320 loss=3.2152 lr=1.04e-04 tok=10.528B dt=2369 wait=75 +22:08:35 step 80340 loss=3.1558 lr=1.04e-04 tok=10.530B dt=2444 wait=75 +22:09:50 step 80360 loss=3.2292 lr=1.04e-04 tok=10.533B dt=2519 wait=76 +22:11:05 step 80380 loss=3.0958 lr=1.03e-04 tok=10.536B dt=2594 wait=76 +22:12:20 step 80400 loss=3.1815 lr=1.03e-04 tok=10.538B dt=2669 wait=76 +22:12:36 pushed ckpt_w33.pt,log_w33.txt +22:13:36 waited 61s for global r=181 +22:13:45 global r=181 load 4.262 -> 4.244 +22:15:07 step 80420 loss=3.1068 lr=1.03e-04 tok=10.541B dt=2836 wait=77 +22:16:21 step 80440 loss=3.1222 lr=1.03e-04 tok=10.543B dt=2910 wait=77 +22:17:36 step 80460 loss=3.1145 lr=1.03e-04 tok=10.546B dt=2985 wait=77 +22:18:51 step 80480 loss=3.1497 lr=1.03e-04 tok=10.549B dt=3060 wait=78 +22:20:05 step 80500 loss=3.1882 lr=1.03e-04 tok=10.551B dt=3135 wait=78 +22:21:20 step 80520 loss=3.1486 lr=1.03e-04 tok=10.554B dt=3210 wait=78 +22:22:35 step 80540 loss=3.2074 lr=1.03e-04 tok=10.557B dt=3284 wait=79 +22:23:24 pushed ckpt_w33.pt,log_w33.txt +22:27:08 step 80560 loss=3.1585 lr=1.03e-04 tok=10.559B dt=3557 wait=79 +22:28:23 step 80580 loss=3.1919 lr=1.03e-04 tok=10.562B dt=3632 wait=79 +22:29:38 step 80600 loss=3.0666 lr=1.03e-04 tok=10.564B dt=3707 wait=80 +22:30:52 step 80620 loss=3.0320 lr=1.03e-04 tok=10.567B dt=3782 wait=80 +22:32:07 step 80640 loss=3.0597 lr=1.03e-04 tok=10.570B dt=3856 wait=80 +22:33:22 step 80660 loss=3.0994 lr=1.03e-04 tok=10.572B dt=3931 wait=81 +22:34:36 step 80680 loss=3.1615 lr=1.03e-04 tok=10.575B dt=4006 wait=81 +22:35:51 step 80700 loss=3.1470 lr=1.03e-04 tok=10.578B dt=4080 wait=81 +22:36:05 pushed ckpt_w33.pt,log_w33.txt +22:36:05 waited 0s for global r=182 +22:36:13 global r=182 load 4.270 -> 4.239 +22:37:34 step 80720 loss=3.1325 lr=1.03e-04 tok=10.580B dt=4183 wait=82 +22:38:48 step 80740 loss=3.1826 lr=1.03e-04 tok=10.583B dt=4258 wait=82 +22:40:03 step 80760 loss=3.1800 lr=1.03e-04 tok=10.585B dt=4333 wait=82 +22:41:18 step 80780 loss=3.2271 lr=1.03e-04 tok=10.588B dt=4408 wait=83 +22:42:33 step 80800 loss=3.2165 lr=1.03e-04 tok=10.591B dt=4483 wait=83 +22:43:48 step 80820 loss=3.1178 lr=1.03e-04 tok=10.593B dt=4557 wait=83 +22:45:03 step 80840 loss=3.1309 lr=1.03e-04 tok=10.596B dt=4632 wait=84 +22:45:52 pushed ckpt_w33.pt,log_w33.txt +22:47:38 waited 106s for global r=183 +22:47:45 global r=183 load 4.266 -> 4.247 +22:48:30 step 80860 loss=3.1052 lr=1.03e-04 tok=10.598B dt=4840 wait=84 +22:49:45 step 80880 loss=3.1061 lr=1.03e-04 tok=10.601B dt=4915 wait=84 +22:51:00 step 80900 loss=3.1804 lr=1.03e-04 tok=10.604B dt=4989 wait=85 +22:52:15 step 80920 loss=3.2004 lr=1.03e-04 tok=10.606B dt=5064 wait=85 +22:53:30 step 80940 loss=3.1070 lr=1.03e-04 tok=10.609B dt=5139 wait=85 +22:54:44 step 80960 loss=3.2015 lr=1.03e-04 tok=10.612B dt=5214 wait=86 +22:55:59 step 80980 loss=3.1022 lr=1.03e-04 tok=10.614B dt=5289 wait=86 +22:57:14 step 81000 loss=3.1828 lr=1.03e-04 tok=10.617B dt=5364 wait=87 +22:57:30 pushed ckpt_w33.pt,log_w33.txt +23:01:53 step 81020 loss=3.1727 lr=1.03e-04 tok=10.619B dt=5642 wait=87 +23:03:08 step 81040 loss=3.1357 lr=1.03e-04 tok=10.622B dt=5717 wait=87 +23:04:22 step 81060 loss=3.0900 lr=1.03e-04 tok=10.625B dt=5792 wait=88 +23:05:37 step 81080 loss=3.1247 lr=1.03e-04 tok=10.627B dt=5866 wait=88 +23:06:52 step 81100 loss=3.0928 lr=1.03e-04 tok=10.630B dt=5941 wait=88 +23:08:07 step 81120 loss=3.1682 lr=1.03e-04 tok=10.633B dt=6016 wait=89 +23:09:21 step 81140 loss=3.1199 lr=1.03e-04 tok=10.635B dt=6090 wait=89 +23:10:11 pushed ckpt_w33.pt,log_w33.txt +23:10:11 waited 0s for global r=184 +23:10:19 global r=184 load 4.263 -> 4.242 +23:11:04 step 81160 loss=3.1301 lr=1.03e-04 tok=10.638B dt=6193 wait=89 +23:12:19 step 81180 loss=3.1654 lr=1.03e-04 tok=10.640B dt=6268 wait=90 +23:13:34 step 81200 loss=3.2130 lr=1.03e-04 tok=10.643B dt=6343 wait=90 +23:14:49 step 81220 loss=3.2430 lr=1.03e-04 tok=10.646B dt=6418 wait=90 +23:16:04 step 81240 loss=3.1608 lr=1.03e-04 tok=10.648B dt=6493 wait=91 +23:17:19 step 81260 loss=3.2179 lr=1.03e-04 tok=10.651B dt=6568 wait=91 +23:18:33 step 81280 loss=3.0453 lr=1.03e-04 tok=10.654B dt=6643 wait=91 +23:19:48 step 81300 loss=3.0888 lr=1.03e-04 tok=10.656B dt=6717 wait=92 +23:20:00 pushed ckpt_w33.pt,log_w33.txt +23:21:00 waited 61s for global r=185 +23:21:08 global r=185 load 4.261 -> 4.242 +23:22:32 step 81320 loss=3.0817 lr=1.03e-04 tok=10.659B dt=6881 wait=92 +23:23:47 step 81340 loss=3.1125 lr=1.03e-04 tok=10.661B dt=6956 wait=92 +23:25:02 step 81360 loss=3.1161 lr=1.02e-04 tok=10.664B dt=7031 wait=93 +23:26:16 step 81380 loss=3.2277 lr=1.02e-04 tok=10.667B dt=7106 wait=93 +23:27:31 step 81400 loss=3.1672 lr=1.02e-04 tok=10.669B dt=7181 wait=93 +23:28:46 step 81420 loss=3.1463 lr=1.02e-04 tok=10.672B dt=7256 wait=94 +23:30:01 step 81440 loss=3.2102 lr=1.02e-04 tok=10.675B dt=7331 wait=94 +23:30:53 pushed ckpt_w33.pt,log_w33.txt +23:34:36 step 81460 loss=3.1606 lr=1.02e-04 tok=10.677B dt=7605 wait=94 +23:35:51 step 81480 loss=3.2113 lr=1.02e-04 tok=10.680B dt=7680 wait=95 +23:37:05 step 81500 loss=3.1123 lr=1.02e-04 tok=10.682B dt=7755 wait=95 +23:38:20 step 81520 loss=3.0590 lr=1.02e-04 tok=10.685B dt=7829 wait=95 +23:39:35 step 81540 loss=3.1129 lr=1.02e-04 tok=10.688B dt=7904 wait=96 +23:40:50 step 81560 loss=3.1142 lr=1.02e-04 tok=10.690B dt=7979 wait=96 +23:42:04 step 81580 loss=3.1247 lr=1.02e-04 tok=10.693B dt=8054 wait=96 +23:43:19 step 81600 loss=3.1084 lr=1.02e-04 tok=10.695B dt=8128 wait=97 +23:43:34 pushed ckpt_w33.pt,log_w33.txt +23:43:34 waited 0s for global r=186 +23:43:42 global r=186 load 4.255 -> 4.236 +23:45:03 step 81620 loss=3.0866 lr=1.02e-04 tok=10.698B dt=8232 wait=97 +23:46:18 step 81640 loss=3.1591 lr=1.02e-04 tok=10.701B dt=8307 wait=97 +23:47:32 step 81660 loss=3.1294 lr=1.02e-04 tok=10.703B dt=8382 wait=98 +23:48:47 step 81680 loss=3.1640 lr=1.02e-04 tok=10.706B dt=8456 wait=98 +23:50:02 step 81700 loss=3.1492 lr=1.02e-04 tok=10.709B dt=8531 wait=98 +23:51:16 step 81720 loss=3.1116 lr=1.02e-04 tok=10.711B dt=8606 wait=99 +23:52:31 step 81740 loss=3.1338 lr=1.02e-04 tok=10.714B dt=8681 wait=99 +23:53:23 pushed ckpt_w33.pt,log_w33.txt +23:54:24 waited 61s for global r=187 +23:54:32 global r=187 load 4.246 -> 4.231 +23:55:13 step 81760 loss=3.1234 lr=1.02e-04 tok=10.716B dt=8842 wait=99 +23:56:28 step 81780 loss=3.1403 lr=1.02e-04 tok=10.719B dt=8917 wait=100 +23:57:42 step 81800 loss=3.2172 lr=1.02e-04 tok=10.722B dt=8992 wait=100 +23:58:58 step 81820 loss=3.2408 lr=1.02e-04 tok=10.724B dt=9067 wait=100 +00:00:12 step 81840 loss=3.1944 lr=1.02e-04 tok=10.727B dt=9142 wait=101 +00:01:27 step 81860 loss=3.2590 lr=1.02e-04 tok=10.730B dt=9217 wait=101 +00:02:42 step 81880 loss=3.0785 lr=1.02e-04 tok=10.732B dt=9292 wait=101 +00:03:57 step 81900 loss=3.1367 lr=1.02e-04 tok=10.735B dt=9366 wait=102 +00:04:11 pushed ckpt_w33.pt,log_w33.txt +00:08:39 step 81920 loss=3.1439 lr=1.02e-04 tok=10.737B dt=9648 wait=102 +00:09:54 step 81940 loss=3.0315 lr=1.02e-04 tok=10.740B dt=9723 wait=102 +00:11:08 step 81960 loss=3.0776 lr=1.02e-04 tok=10.743B dt=9797 wait=103 +00:12:23 step 81980 loss=3.1127 lr=1.02e-04 tok=10.745B dt=9872 wait=103 +00:13:38 step 82000 loss=3.1075 lr=1.02e-04 tok=10.748B dt=9947 wait=103 +00:14:52 step 82020 loss=3.1787 lr=1.02e-04 tok=10.751B dt=10022 wait=104 +00:16:07 step 82040 loss=3.1303 lr=1.02e-04 tok=10.753B dt=10097 wait=104 +00:16:57 pushed ckpt_w33.pt,log_w33.txt +00:16:57 waited 0s for global r=188 +00:17:06 global r=188 load 4.263 -> 4.232 +00:17:46 step 82060 loss=3.0824 lr=1.02e-04 tok=10.756B dt=10196 wait=104 +00:19:01 step 82080 loss=3.1363 lr=1.02e-04 tok=10.758B dt=10270 wait=105 +00:20:16 step 82100 loss=3.1958 lr=1.02e-04 tok=10.761B dt=10345 wait=105 +00:21:31 step 82120 loss=3.2043 lr=1.02e-04 tok=10.764B dt=10420 wait=106 +00:22:46 step 82140 loss=3.1018 lr=1.02e-04 tok=10.766B dt=10495 wait=106 +00:24:00 step 82160 loss=3.2074 lr=1.02e-04 tok=10.769B dt=10570 wait=106 +00:25:15 step 82180 loss=3.0785 lr=1.02e-04 tok=10.771B dt=10644 wait=107 +00:26:30 step 82200 loss=3.1390 lr=1.02e-04 tok=10.774B dt=10719 wait=107 +00:26:42 pushed ckpt_w33.pt,log_w33.txt +00:28:13 waited 91s for global r=189 +00:28:19 global r=189 load 4.258 -> 4.233 +00:29:39 step 82220 loss=3.1386 lr=1.02e-04 tok=10.777B dt=10908 wait=107 +00:30:54 step 82240 loss=3.1895 lr=1.02e-04 tok=10.779B dt=10983 wait=108 +00:32:09 step 82260 loss=3.1609 lr=1.02e-04 tok=10.782B dt=11058 wait=108 +00:33:24 step 82280 loss=3.1618 lr=1.02e-04 tok=10.785B dt=11133 wait=108 +00:34:39 step 82300 loss=3.1765 lr=1.02e-04 tok=10.787B dt=11208 wait=109 +00:35:53 step 82320 loss=3.1282 lr=1.02e-04 tok=10.790B dt=11283 wait=109 +00:37:08 step 82340 loss=3.1673 lr=1.01e-04 tok=10.792B dt=11358 wait=109 +00:37:59 pushed ckpt_w33.pt,log_w33.txt +00:41:45 step 82360 loss=3.1241 lr=1.01e-04 tok=10.795B dt=11634 wait=110 +00:43:00 step 82380 loss=3.1431 lr=1.01e-04 tok=10.798B dt=11709 wait=110 +00:44:14 step 82400 loss=3.0809 lr=1.01e-04 tok=10.800B dt=11784 wait=110 +00:45:29 step 82420 loss=3.0830 lr=1.01e-04 tok=10.803B dt=11858 wait=111 +00:46:44 step 82440 loss=3.1084 lr=1.01e-04 tok=10.806B dt=11933 wait=111 +00:47:58 step 82460 loss=3.1438 lr=1.01e-04 tok=10.808B dt=12008 wait=111 +00:49:13 step 82480 loss=3.1722 lr=1.01e-04 tok=10.811B dt=12082 wait=112 +00:50:28 step 82500 loss=3.1160 lr=1.01e-04 tok=10.813B dt=12157 wait=112 +00:50:42 pushed ckpt_w33.pt,log_w33.txt +00:50:42 waited 0s for global r=190 +00:50:49 global r=190 load 4.258 -> 4.238 +00:52:09 step 82520 loss=3.1371 lr=1.01e-04 tok=10.816B dt=12258 wait=112 +00:53:24 step 82540 loss=3.1694 lr=1.01e-04 tok=10.819B dt=12333 wait=113 +00:54:39 step 82560 loss=3.1482 lr=1.01e-04 tok=10.821B dt=12408 wait=113 +00:55:53 step 82580 loss=3.1845 lr=1.01e-04 tok=10.824B dt=12483 wait=113 +00:57:08 step 82600 loss=3.1459 lr=1.01e-04 tok=10.827B dt=12557 wait=114 +00:58:23 step 82620 loss=3.0733 lr=1.01e-04 tok=10.829B dt=12632 wait=114 +00:59:38 step 82640 loss=3.1493 lr=1.01e-04 tok=10.832B dt=12707 wait=114 +01:00:28 pushed ckpt_w33.pt,log_w33.txt +01:01:59 waited 91s for global r=191 +01:02:06 global r=191 load 4.249 -> 4.234 +01:02:49 step 82660 loss=3.0706 lr=1.01e-04 tok=10.834B dt=12899 wait=115 +01:04:04 step 82680 loss=3.1385 lr=1.01e-04 tok=10.837B dt=12973 wait=115 +01:05:19 step 82700 loss=3.1845 lr=1.01e-04 tok=10.840B dt=13048 wait=115 +01:06:34 step 82720 loss=3.1867 lr=1.01e-04 tok=10.842B dt=13123 wait=116 +01:07:49 step 82740 loss=3.1336 lr=1.01e-04 tok=10.845B dt=13198 wait=116 +01:09:04 step 82760 loss=3.2279 lr=1.01e-04 tok=10.848B dt=13273 wait=117 +01:10:18 step 82780 loss=3.1127 lr=1.01e-04 tok=10.850B dt=13348 wait=117 +01:11:33 step 82800 loss=3.1638 lr=1.01e-04 tok=10.853B dt=13423 wait=117 +01:11:45 pushed ckpt_w33.pt,log_w33.txt +01:16:08 step 82820 loss=3.1700 lr=1.01e-04 tok=10.855B dt=13697 wait=118 +01:17:23 step 82840 loss=3.0872 lr=1.01e-04 tok=10.858B dt=13772 wait=118 +01:18:37 step 82860 loss=3.0687 lr=1.01e-04 tok=10.861B dt=13846 wait=118 +01:19:52 step 82880 loss=3.1155 lr=1.01e-04 tok=10.863B dt=13921 wait=119 +01:21:07 step 82900 loss=3.0862 lr=1.01e-04 tok=10.866B dt=13996 wait=119 +01:22:21 step 82920 loss=3.1826 lr=1.01e-04 tok=10.868B dt=14071 wait=119 +01:23:36 step 82940 loss=3.1491 lr=1.01e-04 tok=10.871B dt=14146 wait=120 +01:24:26 pushed ckpt_w33.pt,log_w33.txt +01:28:08 step 82960 loss=3.1278 lr=1.01e-04 tok=10.874B dt=14418 wait=120 +01:29:23 step 82980 loss=3.1040 lr=1.01e-04 tok=10.876B dt=14492 wait=120 +01:30:38 step 83000 loss=3.1172 lr=1.01e-04 tok=10.879B dt=14567 wait=121 +01:31:52 step 83020 loss=3.1347 lr=1.01e-04 tok=10.882B dt=14642 wait=121 +01:33:07 step 83040 loss=3.1468 lr=1.01e-04 tok=10.884B dt=14716 wait=121 +01:34:22 step 83060 loss=3.1073 lr=1.01e-04 tok=10.887B dt=14791 wait=122 +01:35:36 step 83080 loss=3.1262 lr=1.01e-04 tok=10.889B dt=14866 wait=122 +01:36:51 step 83100 loss=3.0912 lr=1.01e-04 tok=10.892B dt=14941 wait=122 diff --git a/log_w4.txt b/log_w4.txt new file mode 100644 index 0000000000000000000000000000000000000000..7dc2fa2f74832150b987877806f946bc5a6f1518 --- /dev/null +++ b/log_w4.txt @@ -0,0 +1,231 @@ +launch +17:18:25 name=w4 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:18:29 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:18:39 params 1.08B +17:18:40 resumed step=27000 seen=3.54B round=14 +17:18:40 stream salt=27000 +17:21:15 start: accum=4, 131072 tokens per optimiser step +17:24:25 step 27020 loss=3.7918 lr=2.88e-04 tok=3.542B dt=190 +17:25:53 step 27040 loss=3.9118 lr=2.88e-04 tok=3.544B dt=278 +17:27:15 step 27060 loss=3.8577 lr=2.88e-04 tok=3.547B dt=360 +17:28:31 step 27080 loss=3.8829 lr=2.88e-04 tok=3.549B dt=436 +17:29:47 step 27100 loss=3.7343 lr=2.88e-04 tok=3.552B dt=512 +17:31:03 step 27120 loss=3.7971 lr=2.88e-04 tok=3.555B dt=587 +17:32:18 step 27140 loss=3.8013 lr=2.88e-04 tok=3.557B dt=663 +17:33:34 step 27160 loss=3.8160 lr=2.88e-04 tok=3.560B dt=739 +17:34:50 step 27180 loss=3.7391 lr=2.88e-04 tok=3.563B dt=815 +17:36:06 step 27200 loss=3.8222 lr=2.88e-04 tok=3.565B dt=890 +17:37:22 step 27220 loss=3.7734 lr=2.88e-04 tok=3.568B dt=966 +17:38:37 step 27240 loss=3.7554 lr=2.88e-04 tok=3.570B dt=1042 +17:39:53 step 27260 loss=3.8099 lr=2.88e-04 tok=3.573B dt=1118 +17:41:09 step 27280 loss=3.7447 lr=2.88e-04 tok=3.576B dt=1194 +17:42:25 step 27300 loss=3.7840 lr=2.88e-04 tok=3.578B dt=1269 +17:43:40 step 27320 loss=3.9015 lr=2.88e-04 tok=3.581B dt=1345 +17:44:56 step 27340 loss=3.8597 lr=2.88e-04 tok=3.584B dt=1421 +17:46:12 step 27360 loss=3.7957 lr=2.88e-04 tok=3.586B dt=1497 +17:47:28 step 27380 loss=3.9026 lr=2.88e-04 tok=3.589B dt=1573 +17:48:44 step 27400 loss=3.8632 lr=2.88e-04 tok=3.591B dt=1649 +17:50:00 step 27420 loss=3.8047 lr=2.88e-04 tok=3.594B dt=1724 +17:51:15 step 27440 loss=3.7005 lr=2.88e-04 tok=3.597B dt=1800 +17:52:31 step 27460 loss=3.7637 lr=2.88e-04 tok=3.599B dt=1876 +17:53:47 step 27480 loss=3.7008 lr=2.88e-04 tok=3.602B dt=1951 +17:55:02 step 27500 loss=3.8045 lr=2.88e-04 tok=3.604B dt=2027 +17:56:18 step 27520 loss=3.7835 lr=2.88e-04 tok=3.607B dt=2103 +17:57:34 step 27540 loss=3.7598 lr=2.88e-04 tok=3.610B dt=2178 +17:58:49 step 27560 loss=3.8527 lr=2.88e-04 tok=3.612B dt=2254 +18:00:05 step 27580 loss=3.7463 lr=2.88e-04 tok=3.615B dt=2330 +18:01:21 step 27600 loss=3.8412 lr=2.88e-04 tok=3.618B dt=2406 +18:01:33 pushed ckpt_w4.pt +18:01:34 pushed log_w4.txt +18:02:58 step 27620 loss=3.8153 lr=2.88e-04 tok=3.620B dt=2503 +18:04:14 step 27640 loss=3.7678 lr=2.88e-04 tok=3.623B dt=2579 +18:05:30 step 27660 loss=3.7025 lr=2.88e-04 tok=3.625B dt=2654 +18:06:45 step 27680 loss=3.7368 lr=2.88e-04 tok=3.628B dt=2730 +18:08:01 step 27700 loss=3.7534 lr=2.88e-04 tok=3.631B dt=2806 +18:09:17 step 27720 loss=3.8554 lr=2.88e-04 tok=3.633B dt=2881 +18:10:33 step 27740 loss=3.9223 lr=2.88e-04 tok=3.636B dt=2957 +18:11:49 step 27760 loss=3.8582 lr=2.88e-04 tok=3.639B dt=3033 +18:13:04 step 27780 loss=3.7687 lr=2.88e-04 tok=3.641B dt=3109 +18:14:20 step 27800 loss=3.7500 lr=2.88e-04 tok=3.644B dt=3185 +18:15:36 step 27820 loss=3.8385 lr=2.88e-04 tok=3.646B dt=3261 +18:16:52 step 27840 loss=3.7748 lr=2.88e-04 tok=3.649B dt=3337 +18:18:08 step 27860 loss=3.7596 lr=2.88e-04 tok=3.652B dt=3412 +18:19:24 step 27880 loss=3.8352 lr=2.88e-04 tok=3.654B dt=3488 +18:20:39 step 27900 loss=3.7066 lr=2.88e-04 tok=3.657B dt=3564 +18:21:55 step 27920 loss=3.8123 lr=2.88e-04 tok=3.660B dt=3640 +18:23:11 step 27940 loss=3.7300 lr=2.88e-04 tok=3.662B dt=3715 +18:24:27 step 27960 loss=3.7779 lr=2.88e-04 tok=3.665B dt=3791 +18:25:42 step 27980 loss=3.9086 lr=2.88e-04 tok=3.667B dt=3867 +18:26:58 step 28000 loss=3.7873 lr=2.88e-04 tok=3.670B dt=3943 +18:28:14 step 28020 loss=3.7961 lr=2.88e-04 tok=3.673B dt=4019 +18:29:30 step 28040 loss=3.8944 lr=2.88e-04 tok=3.675B dt=4095 +18:30:46 step 28060 loss=3.8099 lr=2.88e-04 tok=3.678B dt=4170 +18:32:01 step 28080 loss=3.7538 lr=2.88e-04 tok=3.681B dt=4246 +18:33:17 step 28100 loss=3.7919 lr=2.88e-04 tok=3.683B dt=4322 +18:34:33 step 28120 loss=3.7586 lr=2.87e-04 tok=3.686B dt=4398 +18:35:49 step 28140 loss=3.8613 lr=2.87e-04 tok=3.688B dt=4473 +18:37:04 step 28160 loss=3.7690 lr=2.87e-04 tok=3.691B dt=4549 +18:38:20 step 28180 loss=3.7668 lr=2.87e-04 tok=3.694B dt=4625 +18:39:36 step 28200 loss=3.9081 lr=2.87e-04 tok=3.696B dt=4701 +18:39:48 pushed ckpt_w4.pt +18:39:48 pushed log_w4.txt +18:40:04 pulled global r=15 4.498 -> 4.380 +18:41:25 step 28220 loss=3.6474 lr=2.87e-04 tok=3.699B dt=4810 +18:42:41 step 28240 loss=3.6080 lr=2.87e-04 tok=3.701B dt=4886 +18:43:57 step 28260 loss=3.6675 lr=2.87e-04 tok=3.704B dt=4961 +18:45:12 step 28280 loss=3.7168 lr=2.87e-04 tok=3.707B dt=5037 +18:46:28 step 28300 loss=3.8048 lr=2.87e-04 tok=3.709B dt=5113 +18:47:44 step 28320 loss=3.7575 lr=2.87e-04 tok=3.712B dt=5189 +18:49:00 step 28340 loss=3.7300 lr=2.87e-04 tok=3.715B dt=5265 +18:50:16 step 28360 loss=3.8157 lr=2.87e-04 tok=3.717B dt=5340 +18:51:32 step 28380 loss=3.9190 lr=2.87e-04 tok=3.720B dt=5416 +18:52:47 step 28400 loss=3.8577 lr=2.87e-04 tok=3.722B dt=5492 +18:54:03 step 28420 loss=3.8373 lr=2.87e-04 tok=3.725B dt=5568 +18:55:19 step 28440 loss=3.8229 lr=2.87e-04 tok=3.728B dt=5644 +18:56:35 step 28460 loss=3.7052 lr=2.87e-04 tok=3.730B dt=5720 +18:57:51 step 28480 loss=3.7689 lr=2.87e-04 tok=3.733B dt=5795 +18:59:06 step 28500 loss=3.7359 lr=2.87e-04 tok=3.736B dt=5871 +19:00:22 step 28520 loss=3.6640 lr=2.87e-04 tok=3.738B dt=5947 +19:01:38 step 28540 loss=3.7362 lr=2.87e-04 tok=3.741B dt=6022 +19:02:53 step 28560 loss=3.8437 lr=2.87e-04 tok=3.743B dt=6098 +19:04:09 step 28580 loss=3.7454 lr=2.87e-04 tok=3.746B dt=6174 +19:05:25 step 28600 loss=3.7937 lr=2.87e-04 tok=3.749B dt=6250 +19:06:41 step 28620 loss=3.7303 lr=2.87e-04 tok=3.751B dt=6325 +19:07:57 step 28640 loss=3.8477 lr=2.87e-04 tok=3.754B dt=6401 +19:09:12 step 28660 loss=3.8156 lr=2.87e-04 tok=3.757B dt=6477 +19:10:28 step 28680 loss=3.7397 lr=2.87e-04 tok=3.759B dt=6553 +19:11:44 step 28700 loss=3.6849 lr=2.87e-04 tok=3.762B dt=6628 +19:13:00 step 28720 loss=3.7444 lr=2.87e-04 tok=3.764B dt=6704 +19:14:15 step 28740 loss=3.7024 lr=2.87e-04 tok=3.767B dt=6780 +19:15:31 step 28760 loss=3.7667 lr=2.87e-04 tok=3.770B dt=6856 +19:16:47 step 28780 loss=3.7843 lr=2.87e-04 tok=3.772B dt=6932 +19:18:03 step 28800 loss=3.7791 lr=2.87e-04 tok=3.775B dt=7007 +19:18:14 pushed ckpt_w4.pt +19:18:15 pushed log_w4.txt +19:19:39 step 28820 loss=3.7784 lr=2.87e-04 tok=3.777B dt=7104 +19:20:55 step 28840 loss=3.7361 lr=2.87e-04 tok=3.780B dt=7180 +19:22:11 step 28860 loss=3.8760 lr=2.87e-04 tok=3.783B dt=7255 +19:23:26 step 28880 loss=3.8380 lr=2.87e-04 tok=3.785B dt=7331 +19:24:42 step 28900 loss=3.7768 lr=2.87e-04 tok=3.788B dt=7407 +19:25:58 step 28920 loss=3.8349 lr=2.87e-04 tok=3.791B dt=7483 +19:27:14 step 28940 loss=3.6763 lr=2.87e-04 tok=3.793B dt=7559 +19:28:30 step 28960 loss=3.7423 lr=2.87e-04 tok=3.796B dt=7634 +19:29:45 step 28980 loss=3.7006 lr=2.87e-04 tok=3.798B dt=7710 +19:31:01 step 29000 loss=3.7460 lr=2.87e-04 tok=3.801B dt=7785 +19:32:17 step 29020 loss=3.7240 lr=2.87e-04 tok=3.804B dt=7861 +19:33:32 step 29040 loss=3.6785 lr=2.87e-04 tok=3.806B dt=7937 +19:34:48 step 29060 loss=3.8274 lr=2.87e-04 tok=3.809B dt=8013 +19:36:04 step 29080 loss=3.7402 lr=2.87e-04 tok=3.812B dt=8089 +19:37:20 step 29100 loss=3.7177 lr=2.87e-04 tok=3.814B dt=8164 +19:38:36 step 29120 loss=3.7733 lr=2.87e-04 tok=3.817B dt=8240 +19:39:51 step 29140 loss=3.7010 lr=2.87e-04 tok=3.819B dt=8316 +19:41:07 step 29160 loss=3.7454 lr=2.87e-04 tok=3.822B dt=8392 +19:42:23 step 29180 loss=3.8874 lr=2.87e-04 tok=3.825B dt=8468 +19:43:39 step 29200 loss=3.8090 lr=2.87e-04 tok=3.827B dt=8543 +19:44:54 step 29220 loss=3.7885 lr=2.86e-04 tok=3.830B dt=8619 +19:46:10 step 29240 loss=3.7626 lr=2.86e-04 tok=3.833B dt=8695 +19:47:26 step 29260 loss=3.7293 lr=2.86e-04 tok=3.835B dt=8771 +19:48:42 step 29280 loss=3.7128 lr=2.86e-04 tok=3.838B dt=8847 +19:49:58 step 29300 loss=3.7568 lr=2.86e-04 tok=3.840B dt=8922 +19:51:13 step 29320 loss=3.7923 lr=2.86e-04 tok=3.843B dt=8998 +19:52:29 step 29340 loss=3.7242 lr=2.86e-04 tok=3.846B dt=9074 +19:53:45 step 29360 loss=3.7554 lr=2.86e-04 tok=3.848B dt=9149 +19:55:00 step 29380 loss=3.8169 lr=2.86e-04 tok=3.851B dt=9225 +19:56:16 step 29400 loss=3.8096 lr=2.86e-04 tok=3.854B dt=9301 +19:56:30 pushed ckpt_w4.pt +19:56:31 pushed log_w4.txt +19:57:55 step 29420 loss=3.8306 lr=2.86e-04 tok=3.856B dt=9399 +19:59:10 step 29440 loss=3.7073 lr=2.86e-04 tok=3.859B dt=9475 +20:00:26 step 29460 loss=3.7405 lr=2.86e-04 tok=3.861B dt=9551 +20:01:42 step 29480 loss=3.8151 lr=2.86e-04 tok=3.864B dt=9627 +20:02:58 step 29500 loss=3.7810 lr=2.86e-04 tok=3.867B dt=9702 +20:04:14 step 29520 loss=3.7839 lr=2.86e-04 tok=3.869B dt=9778 +20:05:29 step 29540 loss=3.6492 lr=2.86e-04 tok=3.872B dt=9854 +20:06:45 step 29560 loss=3.7815 lr=2.86e-04 tok=3.874B dt=9930 +20:08:01 step 29580 loss=3.8263 lr=2.86e-04 tok=3.877B dt=10006 +20:09:17 step 29600 loss=3.7196 lr=2.86e-04 tok=3.880B dt=10082 +20:10:33 step 29620 loss=3.7751 lr=2.86e-04 tok=3.882B dt=10157 +20:11:48 step 29640 loss=3.6813 lr=2.86e-04 tok=3.885B dt=10233 +20:13:04 step 29660 loss=3.6835 lr=2.86e-04 tok=3.888B dt=10309 +20:14:20 step 29680 loss=3.7391 lr=2.86e-04 tok=3.890B dt=10384 +20:15:36 step 29700 loss=3.7801 lr=2.86e-04 tok=3.893B dt=10460 +20:16:51 step 29720 loss=3.6963 lr=2.86e-04 tok=3.895B dt=10536 +20:18:07 step 29740 loss=3.6930 lr=2.86e-04 tok=3.898B dt=10612 +20:19:23 step 29760 loss=3.7540 lr=2.86e-04 tok=3.901B dt=10687 +20:20:39 step 29780 loss=3.8098 lr=2.86e-04 tok=3.903B dt=10763 +20:21:54 step 29800 loss=3.8501 lr=2.86e-04 tok=3.906B dt=10839 +20:23:10 step 29820 loss=3.7056 lr=2.86e-04 tok=3.909B dt=10914 +20:24:25 step 29840 loss=3.8260 lr=2.86e-04 tok=3.911B dt=10990 +20:25:41 step 29860 loss=3.7968 lr=2.86e-04 tok=3.914B dt=11066 +20:26:57 step 29880 loss=3.8141 lr=2.86e-04 tok=3.916B dt=11142 +20:28:13 step 29900 loss=3.7675 lr=2.86e-04 tok=3.919B dt=11218 +20:29:29 step 29920 loss=3.8053 lr=2.86e-04 tok=3.922B dt=11293 +20:30:45 step 29940 loss=3.8790 lr=2.86e-04 tok=3.924B dt=11369 +20:32:01 step 29960 loss=3.7345 lr=2.86e-04 tok=3.927B dt=11445 +20:33:16 step 29980 loss=3.7434 lr=2.86e-04 tok=3.930B dt=11521 +20:34:32 step 30000 loss=3.7307 lr=2.86e-04 tok=3.932B dt=11597 +20:34:46 pushed ckpt_w4.pt +20:34:46 pushed log_w4.txt +20:36:08 step 30020 loss=3.8979 lr=2.86e-04 tok=3.935B dt=11692 +20:37:23 step 30040 loss=3.8280 lr=2.86e-04 tok=3.937B dt=11768 +20:38:39 step 30060 loss=3.7638 lr=2.86e-04 tok=3.940B dt=11844 +20:39:55 step 30080 loss=3.7496 lr=2.86e-04 tok=3.943B dt=11920 +20:41:11 step 30100 loss=3.8327 lr=2.86e-04 tok=3.945B dt=11995 +20:42:26 step 30120 loss=3.7455 lr=2.86e-04 tok=3.948B dt=12071 +20:43:42 step 30140 loss=3.7550 lr=2.86e-04 tok=3.951B dt=12147 +20:44:58 step 30160 loss=3.7733 lr=2.86e-04 tok=3.953B dt=12222 +20:46:13 step 30180 loss=3.7334 lr=2.86e-04 tok=3.956B dt=12298 +20:47:29 step 30200 loss=3.7216 lr=2.86e-04 tok=3.958B dt=12374 +20:48:45 step 30220 loss=3.8166 lr=2.86e-04 tok=3.961B dt=12450 +20:50:01 step 30240 loss=3.7483 lr=2.86e-04 tok=3.964B dt=12525 +20:51:16 step 30260 loss=3.6590 lr=2.86e-04 tok=3.966B dt=12601 +20:52:32 step 30280 loss=3.8445 lr=2.85e-04 tok=3.969B dt=12677 +20:53:48 step 30300 loss=3.6787 lr=2.85e-04 tok=3.971B dt=12752 +20:55:04 step 30320 loss=3.7741 lr=2.85e-04 tok=3.974B dt=12828 +20:56:19 step 30340 loss=3.7076 lr=2.85e-04 tok=3.977B dt=12904 +20:57:35 step 30360 loss=3.7870 lr=2.85e-04 tok=3.979B dt=12980 +20:58:51 step 30380 loss=3.8117 lr=2.85e-04 tok=3.982B dt=13056 +21:00:07 step 30400 loss=3.7737 lr=2.85e-04 tok=3.985B dt=13132 +21:01:23 step 30420 loss=3.8111 lr=2.85e-04 tok=3.987B dt=13207 +21:02:39 step 30440 loss=3.8413 lr=2.85e-04 tok=3.990B dt=13283 +21:03:54 step 30460 loss=3.7469 lr=2.85e-04 tok=3.992B dt=13359 +21:05:10 step 30480 loss=3.8286 lr=2.85e-04 tok=3.995B dt=13435 +21:06:26 step 30500 loss=3.7952 lr=2.85e-04 tok=3.998B dt=13511 +21:07:42 step 30520 loss=3.8798 lr=2.85e-04 tok=4.000B dt=13587 +21:08:58 step 30540 loss=3.7418 lr=2.85e-04 tok=4.003B dt=13662 +21:10:14 step 30560 loss=3.8524 lr=2.85e-04 tok=4.006B dt=13738 +21:11:29 step 30580 loss=3.7912 lr=2.85e-04 tok=4.008B dt=13814 +21:12:45 step 30600 loss=3.7947 lr=2.85e-04 tok=4.011B dt=13890 +21:12:58 pushed ckpt_w4.pt +21:12:58 pushed log_w4.txt +21:14:21 step 30620 loss=3.8046 lr=2.85e-04 tok=4.013B dt=13986 +21:15:37 step 30640 loss=3.7351 lr=2.85e-04 tok=4.016B dt=14061 +21:16:52 step 30660 loss=3.8049 lr=2.85e-04 tok=4.019B dt=14137 +21:18:08 step 30680 loss=3.6651 lr=2.85e-04 tok=4.021B dt=14213 +21:19:24 step 30700 loss=3.7831 lr=2.85e-04 tok=4.024B dt=14289 +21:20:40 step 30720 loss=3.8319 lr=2.85e-04 tok=4.027B dt=14364 +21:21:56 step 30740 loss=3.8041 lr=2.85e-04 tok=4.029B dt=14440 +21:23:11 step 30760 loss=3.7261 lr=2.85e-04 tok=4.032B dt=14516 +21:24:27 step 30780 loss=3.7504 lr=2.85e-04 tok=4.034B dt=14592 +21:25:43 step 30800 loss=3.7490 lr=2.85e-04 tok=4.037B dt=14667 +21:26:59 step 30820 loss=3.7238 lr=2.85e-04 tok=4.040B dt=14743 +21:28:15 step 30840 loss=3.7913 lr=2.85e-04 tok=4.042B dt=14819 +21:29:30 step 30860 loss=3.8402 lr=2.85e-04 tok=4.045B dt=14895 +21:30:46 step 30880 loss=3.8130 lr=2.85e-04 tok=4.048B dt=14971 +21:32:02 step 30900 loss=3.7728 lr=2.85e-04 tok=4.050B dt=15047 +21:33:18 step 30920 loss=3.7555 lr=2.85e-04 tok=4.053B dt=15122 +21:34:33 step 30940 loss=3.7195 lr=2.85e-04 tok=4.055B dt=15198 +21:35:49 step 30960 loss=3.7624 lr=2.85e-04 tok=4.058B dt=15274 +21:37:05 step 30980 loss=3.7577 lr=2.85e-04 tok=4.061B dt=15350 +21:38:21 step 31000 loss=3.7302 lr=2.85e-04 tok=4.063B dt=15425 +21:39:37 step 31020 loss=3.7853 lr=2.85e-04 tok=4.066B dt=15501 +21:40:52 step 31040 loss=3.8567 lr=2.85e-04 tok=4.068B dt=15577 +21:42:08 step 31060 loss=3.7223 lr=2.85e-04 tok=4.071B dt=15653 +21:43:24 step 31080 loss=3.7459 lr=2.85e-04 tok=4.074B dt=15728 +21:44:39 step 31100 loss=3.6887 lr=2.85e-04 tok=4.076B dt=15804 +21:45:55 step 31120 loss=3.8057 lr=2.85e-04 tok=4.079B dt=15880 +21:47:11 step 31140 loss=3.8798 lr=2.85e-04 tok=4.082B dt=15956 +21:48:27 step 31160 loss=3.8504 lr=2.85e-04 tok=4.084B dt=16032 +21:49:43 step 31180 loss=3.8424 lr=2.85e-04 tok=4.087B dt=16107 +21:50:58 step 31200 loss=3.7647 lr=2.85e-04 tok=4.089B dt=16183 +21:51:10 pushed ckpt_w4.pt diff --git a/log_w41.txt b/log_w41.txt new file mode 100644 index 0000000000000000000000000000000000000000..d95c127bb133e5ac6ce44d5d087112af9d409843 --- /dev/null +++ b/log_w41.txt @@ -0,0 +1,540 @@ +launch +14:09:46 name=w41 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +14:09:52 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +14:10:02 params 1.08B +14:10:03 resumed step=82650 seen=10.83B round=191 +14:10:03 stream salt=82650 +14:12:45 start: accum=4, 131072 tokens per optimiser step +14:14:56 step 82651 loss=3.1900 lr=1.01e-04 tok=10.833B dt=131 wait=103 +14:15:01 step 82652 loss=3.1531 lr=1.01e-04 tok=10.833B dt=136 wait=103 +14:15:34 step 82660 loss=3.1801 lr=1.01e-04 tok=10.834B dt=169 wait=104 +14:17:00 step 82680 loss=3.2306 lr=1.01e-04 tok=10.837B dt=255 wait=104 +14:18:27 step 82700 loss=3.2529 lr=1.01e-04 tok=10.840B dt=342 wait=104 +14:19:45 step 82720 loss=3.2908 lr=1.01e-04 tok=10.842B dt=420 wait=105 +14:20:59 step 82740 loss=3.2048 lr=1.01e-04 tok=10.845B dt=495 wait=105 +14:22:14 step 82760 loss=3.0982 lr=1.01e-04 tok=10.848B dt=570 wait=106 +14:23:29 step 82780 loss=3.1815 lr=1.01e-04 tok=10.850B dt=644 wait=106 +14:24:43 step 82800 loss=3.2099 lr=1.01e-04 tok=10.853B dt=719 wait=106 +14:24:58 pushed ckpt_w41.pt,log_w41.txt +14:26:44 waited 106s for global r=192 +14:27:02 global r=192 load 4.262 -> 4.240 +14:28:22 step 82820 loss=3.0549 lr=1.01e-04 tok=10.855B dt=937 wait=107 +14:29:36 step 82840 loss=3.1046 lr=1.01e-04 tok=10.858B dt=1011 wait=107 +14:30:51 step 82860 loss=3.0779 lr=1.01e-04 tok=10.861B dt=1086 wait=107 +14:32:05 step 82880 loss=3.1595 lr=1.01e-04 tok=10.863B dt=1161 wait=108 +14:33:20 step 82900 loss=3.1380 lr=1.01e-04 tok=10.866B dt=1235 wait=108 +14:34:34 step 82920 loss=3.1137 lr=1.01e-04 tok=10.868B dt=1310 wait=108 +14:35:49 step 82940 loss=3.1405 lr=1.01e-04 tok=10.871B dt=1385 wait=109 +14:36:39 pushed ckpt_w41.pt,log_w41.txt +14:37:55 waited 76s for global r=193 +14:38:12 global r=193 load 4.249 -> 4.234 +14:38:56 step 82960 loss=3.1256 lr=1.01e-04 tok=10.874B dt=1571 wait=109 +14:40:10 step 82980 loss=3.1125 lr=1.01e-04 tok=10.876B dt=1646 wait=109 +14:41:25 step 83000 loss=3.2001 lr=1.01e-04 tok=10.879B dt=1720 wait=110 +14:42:40 step 83020 loss=3.1958 lr=1.01e-04 tok=10.882B dt=1795 wait=110 +14:43:55 step 83040 loss=3.1454 lr=1.01e-04 tok=10.884B dt=1870 wait=110 +14:45:09 step 83060 loss=3.1879 lr=1.01e-04 tok=10.887B dt=1945 wait=111 +14:46:24 step 83080 loss=3.0491 lr=1.01e-04 tok=10.889B dt=2019 wait=111 +14:47:38 step 83100 loss=3.0802 lr=1.01e-04 tok=10.892B dt=2094 wait=111 +14:47:50 pushed ckpt_w41.pt,log_w41.txt +14:50:37 waited 166s for global r=194 +14:50:51 global r=194 load 4.253 -> 4.234 +14:52:11 step 83120 loss=3.0935 lr=1.01e-04 tok=10.895B dt=2366 wait=112 +14:53:25 step 83140 loss=3.0857 lr=1.01e-04 tok=10.897B dt=2441 wait=112 +14:54:40 step 83160 loss=3.1006 lr=1.01e-04 tok=10.900B dt=2515 wait=112 +14:55:54 step 83180 loss=3.1815 lr=1.01e-04 tok=10.903B dt=2590 wait=113 +14:57:09 step 83200 loss=3.1787 lr=1.01e-04 tok=10.905B dt=2665 wait=113 +14:58:24 step 83220 loss=3.1501 lr=1.01e-04 tok=10.908B dt=2740 wait=114 +14:59:39 step 83240 loss=3.1903 lr=1.01e-04 tok=10.910B dt=2815 wait=114 +15:00:31 pushed ckpt_w41.pt,log_w41.txt +15:04:17 step 83260 loss=3.1631 lr=1.01e-04 tok=10.913B dt=3092 wait=114 +15:05:31 step 83280 loss=3.1844 lr=1.01e-04 tok=10.916B dt=3167 wait=115 +15:06:46 step 83300 loss=3.0799 lr=1.01e-04 tok=10.918B dt=3241 wait=115 +15:08:00 step 83320 loss=3.0441 lr=1.00e-04 tok=10.921B dt=3316 wait=115 +15:09:15 step 83340 loss=3.0786 lr=1.00e-04 tok=10.924B dt=3390 wait=116 +15:10:29 step 83360 loss=3.0820 lr=1.00e-04 tok=10.926B dt=3465 wait=116 +15:11:44 step 83380 loss=3.1109 lr=1.00e-04 tok=10.929B dt=3539 wait=116 +15:12:58 step 83400 loss=3.0664 lr=1.00e-04 tok=10.931B dt=3614 wait=117 +15:13:14 pushed ckpt_w41.pt,log_w41.txt +15:13:14 waited 0s for global r=195 +15:13:28 global r=195 load 4.260 -> 4.236 +15:14:48 step 83420 loss=3.1228 lr=1.00e-04 tok=10.934B dt=3723 wait=117 +15:16:03 step 83440 loss=3.1347 lr=1.00e-04 tok=10.937B dt=3798 wait=117 +15:17:18 step 83460 loss=3.1861 lr=1.00e-04 tok=10.939B dt=3873 wait=118 +15:18:33 step 83480 loss=3.2109 lr=1.00e-04 tok=10.942B dt=3948 wait=118 +15:19:48 step 83500 loss=3.2052 lr=1.00e-04 tok=10.945B dt=4023 wait=119 +15:21:02 step 83520 loss=3.1134 lr=1.00e-04 tok=10.947B dt=4098 wait=119 +15:22:17 step 83540 loss=3.1412 lr=1.00e-04 tok=10.950B dt=4173 wait=119 +15:23:06 pushed ckpt_w41.pt,log_w41.txt +15:24:22 waited 76s for global r=196 +15:24:34 global r=196 load 4.254 -> 4.231 +15:25:17 step 83560 loss=3.0937 lr=1.00e-04 tok=10.952B dt=4353 wait=120 +15:26:32 step 83580 loss=3.0754 lr=1.00e-04 tok=10.955B dt=4428 wait=120 +15:27:47 step 83600 loss=3.1307 lr=1.00e-04 tok=10.958B dt=4502 wait=120 +15:29:02 step 83620 loss=3.1649 lr=1.00e-04 tok=10.960B dt=4577 wait=121 +15:30:16 step 83640 loss=3.0985 lr=1.00e-04 tok=10.963B dt=4652 wait=121 +15:31:31 step 83660 loss=3.2266 lr=1.00e-04 tok=10.965B dt=4727 wait=122 +15:32:47 step 83680 loss=3.1212 lr=1.00e-04 tok=10.968B dt=4802 wait=122 +15:34:02 step 83700 loss=3.2008 lr=1.00e-04 tok=10.971B dt=4877 wait=123 +15:34:18 pushed ckpt_w41.pt,log_w41.txt +15:38:42 step 83720 loss=3.1808 lr=1.00e-04 tok=10.973B dt=5157 wait=123 +15:39:57 step 83740 loss=3.1179 lr=1.00e-04 tok=10.976B dt=5232 wait=123 +15:41:12 step 83760 loss=3.1076 lr=1.00e-04 tok=10.979B dt=5307 wait=124 +15:42:27 step 83780 loss=3.0549 lr=1.00e-04 tok=10.981B dt=5382 wait=124 +15:43:42 step 83800 loss=3.0322 lr=1.00e-04 tok=10.984B dt=5457 wait=125 +15:44:56 step 83820 loss=3.1232 lr=1.00e-04 tok=10.986B dt=5532 wait=125 +15:46:11 step 83840 loss=3.0937 lr=1.00e-04 tok=10.989B dt=5607 wait=126 +15:47:04 pushed ckpt_w41.pt,log_w41.txt +15:47:04 waited 0s for global r=197 +15:47:20 global r=197 load 4.257 -> 4.236 +15:48:02 step 83860 loss=3.0922 lr=9.99e-05 tok=10.992B dt=5718 wait=126 +15:49:17 step 83880 loss=3.0980 lr=9.99e-05 tok=10.994B dt=5793 wait=126 +15:50:32 step 83900 loss=3.1961 lr=9.99e-05 tok=10.997B dt=5868 wait=127 +15:51:48 step 83920 loss=3.2111 lr=9.99e-05 tok=11.000B dt=5943 wait=127 +15:53:03 step 83940 loss=3.1374 lr=9.99e-05 tok=11.002B dt=6018 wait=128 +15:54:18 step 83960 loss=3.2570 lr=9.98e-05 tok=11.005B dt=6094 wait=128 +15:55:33 step 83980 loss=3.1472 lr=9.98e-05 tok=11.007B dt=6169 wait=129 +15:56:49 step 84000 loss=3.1733 lr=9.98e-05 tok=11.010B dt=6244 wait=129 +15:57:02 pushed ckpt_w41.pt,log_w41.txt +15:59:33 waited 152s for global r=198 +15:59:46 global r=198 load 4.255 -> 4.239 +16:01:05 step 84020 loss=3.0989 lr=9.98e-05 tok=11.013B dt=6500 wait=129 +16:02:20 step 84040 loss=3.1090 lr=9.98e-05 tok=11.015B dt=6575 wait=130 +16:03:35 step 84060 loss=3.0992 lr=9.97e-05 tok=11.018B dt=6650 wait=130 +16:04:49 step 84080 loss=3.1369 lr=9.97e-05 tok=11.021B dt=6725 wait=131 +16:06:04 step 84100 loss=3.1388 lr=9.97e-05 tok=11.023B dt=6800 wait=131 +16:07:20 step 84120 loss=3.1145 lr=9.97e-05 tok=11.026B dt=6875 wait=132 +16:08:35 step 84140 loss=3.1818 lr=9.97e-05 tok=11.028B dt=6950 wait=132 +16:09:29 pushed ckpt_w41.pt,log_w41.txt +16:13:13 step 84160 loss=3.1428 lr=9.96e-05 tok=11.031B dt=7229 wait=132 +16:14:28 step 84180 loss=3.1853 lr=9.96e-05 tok=11.034B dt=7304 wait=133 +16:15:43 step 84200 loss=3.0518 lr=9.96e-05 tok=11.036B dt=7379 wait=133 +16:16:58 step 84220 loss=2.9963 lr=9.96e-05 tok=11.039B dt=7453 wait=134 +16:18:13 step 84240 loss=3.0549 lr=9.96e-05 tok=11.042B dt=7528 wait=134 +16:19:28 step 84260 loss=3.0939 lr=9.95e-05 tok=11.044B dt=7603 wait=135 +16:20:43 step 84280 loss=3.1573 lr=9.95e-05 tok=11.047B dt=7678 wait=135 +16:21:58 step 84300 loss=3.1081 lr=9.95e-05 tok=11.049B dt=7753 wait=136 +16:22:12 pushed ckpt_w41.pt,log_w41.txt +16:22:12 waited 0s for global r=199 +16:22:24 global r=199 load 4.276 -> 4.234 +16:23:46 step 84320 loss=3.1260 lr=9.95e-05 tok=11.052B dt=7862 wait=136 +16:25:01 step 84340 loss=3.1751 lr=9.95e-05 tok=11.055B dt=7937 wait=136 +16:26:17 step 84360 loss=3.1466 lr=9.94e-05 tok=11.057B dt=8012 wait=137 +16:27:32 step 84380 loss=3.2139 lr=9.94e-05 tok=11.060B dt=8087 wait=137 +16:28:47 step 84400 loss=3.1771 lr=9.94e-05 tok=11.062B dt=8163 wait=138 +16:30:02 step 84420 loss=3.0990 lr=9.94e-05 tok=11.065B dt=8238 wait=138 +16:31:17 step 84440 loss=3.1596 lr=9.93e-05 tok=11.068B dt=8313 wait=139 +16:32:08 pushed ckpt_w41.pt,log_w41.txt +16:34:09 waited 121s for global r=200 +16:34:20 global r=200 load 4.248 -> 4.235 +16:35:04 step 84460 loss=3.0439 lr=9.93e-05 tok=11.070B dt=8539 wait=139 +16:36:19 step 84480 loss=3.0573 lr=9.93e-05 tok=11.073B dt=8614 wait=139 +16:37:34 step 84500 loss=3.1267 lr=9.93e-05 tok=11.076B dt=8689 wait=140 +16:38:49 step 84520 loss=3.1597 lr=9.93e-05 tok=11.078B dt=8764 wait=140 +16:40:04 step 84540 loss=3.0850 lr=9.92e-05 tok=11.081B dt=8839 wait=141 +16:41:19 step 84560 loss=3.1908 lr=9.92e-05 tok=11.083B dt=8914 wait=141 +16:42:34 step 84580 loss=3.1317 lr=9.92e-05 tok=11.086B dt=8989 wait=142 +16:43:49 step 84600 loss=3.1660 lr=9.92e-05 tok=11.089B dt=9065 wait=142 +16:44:04 pushed ckpt_w41.pt,log_w41.txt +launch +16:44:57 name=w41 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:45:03 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:45:14 params 1.08B +16:45:14 muon lr=0.005 on 90 matrices, adamw on 38 tensors +16:45:18 resume fail loaded state dict has a different number of parameter groups +16:45:18 stream salt=0 +launch +16:46:32 name=w41 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:46:38 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:46:48 params 1.08B +16:46:48 muon lr=0.005 on 90 matrices, adamw on 38 tensors +16:46:52 optimiser state dropped (loaded state dict has a different number of parameter groups) +16:46:52 resumed step=84450 seen=11.07B round=200 +16:46:52 stream salt=84450 +16:49:38 start: accum=4, 131072 tokens per optimiser step +16:51:29 step 84451 loss=3.1185 lr=9.93e-05 tok=11.069B dt=112 wait=95 +16:51:34 step 84452 loss=3.0721 lr=9.93e-05 tok=11.069B dt=116 wait=95 +16:52:13 step 84460 loss=3.1574 lr=9.93e-05 tok=11.070B dt=156 wait=95 +16:53:51 step 84480 loss=3.2523 lr=9.93e-05 tok=11.073B dt=253 wait=96 +16:55:15 step 84500 loss=3.2265 lr=9.93e-05 tok=11.076B dt=337 wait=96 +16:56:34 step 84520 loss=3.3315 lr=9.93e-05 tok=11.078B dt=417 wait=97 +16:57:54 step 84540 loss=3.2934 lr=9.92e-05 tok=11.081B dt=496 wait=97 +16:59:13 step 84560 loss=3.1736 lr=9.92e-05 tok=11.083B dt=575 wait=97 +17:00:32 step 84580 loss=3.1824 lr=9.92e-05 tok=11.086B dt=654 wait=98 +17:01:51 step 84600 loss=3.2897 lr=9.92e-05 tok=11.089B dt=734 wait=98 +17:02:08 pushed ckpt_w41.pt,log_w41.txt +17:02:08 waited 0s for global r=201 +17:02:25 global r=201 load 4.346 -> 4.246 +17:03:56 step 84620 loss=3.0966 lr=9.92e-05 tok=11.091B dt=859 wait=99 +17:05:15 step 84640 loss=3.1618 lr=9.91e-05 tok=11.094B dt=938 wait=99 +17:06:35 step 84660 loss=3.1896 lr=9.91e-05 tok=11.097B dt=1017 wait=99 +17:07:54 step 84680 loss=3.2488 lr=9.91e-05 tok=11.099B dt=1097 wait=100 +17:09:14 step 84700 loss=3.2444 lr=9.91e-05 tok=11.102B dt=1176 wait=100 +17:10:33 step 84720 loss=3.2386 lr=9.91e-05 tok=11.104B dt=1255 wait=101 +17:11:53 step 84740 loss=3.2534 lr=9.90e-05 tok=11.107B dt=1335 wait=101 +17:12:49 pushed ckpt_w41.pt,log_w41.txt +17:16:37 step 84760 loss=3.2600 lr=9.90e-05 tok=11.110B dt=1620 wait=101 +17:17:57 step 84780 loss=3.2957 lr=9.90e-05 tok=11.112B dt=1699 wait=102 +17:19:16 step 84800 loss=3.2262 lr=9.90e-05 tok=11.115B dt=1779 wait=102 +17:20:36 step 84820 loss=3.1996 lr=9.90e-05 tok=11.118B dt=1858 wait=103 +17:21:55 step 84840 loss=3.2566 lr=9.89e-05 tok=11.120B dt=1938 wait=103 +17:23:15 step 84860 loss=3.3014 lr=9.89e-05 tok=11.123B dt=2017 wait=104 +17:24:34 step 84880 loss=3.3285 lr=9.89e-05 tok=11.125B dt=2097 wait=104 +17:25:54 step 84900 loss=3.3021 lr=9.89e-05 tok=11.128B dt=2176 wait=104 +17:26:08 pushed ckpt_w41.pt,log_w41.txt +17:30:33 step 84920 loss=3.2397 lr=9.89e-05 tok=11.131B dt=2456 wait=105 +17:31:53 step 84940 loss=3.2576 lr=9.88e-05 tok=11.133B dt=2535 wait=105 +17:33:12 step 84960 loss=3.3283 lr=9.88e-05 tok=11.136B dt=2614 wait=106 +17:34:31 step 84980 loss=3.2980 lr=9.88e-05 tok=11.138B dt=2694 wait=106 +17:35:51 step 85000 loss=3.2910 lr=9.88e-05 tok=11.141B dt=2773 wait=106 +17:37:10 step 85020 loss=3.2768 lr=9.88e-05 tok=11.144B dt=2852 wait=107 +17:38:29 step 85040 loss=3.2257 lr=9.87e-05 tok=11.146B dt=2932 wait=107 +17:39:24 pushed ckpt_w41.pt,log_w41.txt +17:43:09 step 85060 loss=3.3196 lr=9.87e-05 tok=11.149B dt=3212 wait=108 +17:44:29 step 85080 loss=3.3078 lr=9.87e-05 tok=11.152B dt=3291 wait=108 +17:45:48 step 85100 loss=3.3464 lr=9.87e-05 tok=11.154B dt=3371 wait=108 +17:47:08 step 85120 loss=3.2894 lr=9.87e-05 tok=11.157B dt=3450 wait=109 +17:48:27 step 85140 loss=3.3245 lr=9.86e-05 tok=11.159B dt=3530 wait=109 +17:49:47 step 85160 loss=3.4327 lr=9.86e-05 tok=11.162B dt=3609 wait=110 +17:51:06 step 85180 loss=3.3855 lr=9.86e-05 tok=11.165B dt=3689 wait=110 +17:52:26 step 85200 loss=3.3719 lr=9.86e-05 tok=11.167B dt=3768 wait=111 +17:52:40 pushed ckpt_w41.pt,log_w41.txt +17:57:08 step 85220 loss=3.4227 lr=9.85e-05 tok=11.170B dt=4050 wait=111 +17:58:27 step 85240 loss=3.2775 lr=9.85e-05 tok=11.173B dt=4129 wait=111 +17:59:46 step 85260 loss=3.3031 lr=9.85e-05 tok=11.175B dt=4209 wait=112 +18:01:06 step 85280 loss=3.3114 lr=9.85e-05 tok=11.178B dt=4288 wait=112 +18:02:25 step 85300 loss=3.2690 lr=9.85e-05 tok=11.180B dt=4368 wait=113 +18:03:44 step 85320 loss=3.2958 lr=9.84e-05 tok=11.183B dt=4447 wait=113 +18:05:04 step 85340 loss=3.3118 lr=9.84e-05 tok=11.186B dt=4526 wait=113 +18:05:59 pushed ckpt_w41.pt,log_w41.txt +18:09:48 step 85360 loss=3.2890 lr=9.84e-05 tok=11.188B dt=4810 wait=114 +18:11:07 step 85380 loss=3.3087 lr=9.84e-05 tok=11.191B dt=4889 wait=114 +18:12:26 step 85400 loss=3.3187 lr=9.84e-05 tok=11.194B dt=4968 wait=115 +18:13:45 step 85420 loss=3.3337 lr=9.83e-05 tok=11.196B dt=5048 wait=115 +18:15:04 step 85440 loss=3.4017 lr=9.83e-05 tok=11.199B dt=5127 wait=115 +18:16:23 step 85460 loss=3.3111 lr=9.83e-05 tok=11.201B dt=5206 wait=116 +18:17:42 step 85480 loss=3.3459 lr=9.83e-05 tok=11.204B dt=5285 wait=116 +launch +18:19:59 name=w41 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:20:04 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:20:13 params 1.08B +18:20:15 resumed step=85500 seen=11.21B round=201 +18:20:15 stream salt=85500 +18:22:14 start: accum=4, 131072 tokens per optimiser step +18:23:26 step 85501 loss=3.2694 lr=9.83e-05 tok=11.207B dt=72 wait=53 +18:23:31 step 85502 loss=3.0840 lr=9.83e-05 tok=11.207B dt=77 wait=53 +18:24:54 step 85520 loss=3.2380 lr=9.82e-05 tok=11.209B dt=160 wait=54 +18:26:10 step 85540 loss=3.2299 lr=9.82e-05 tok=11.212B dt=235 wait=54 +18:27:25 step 85560 loss=3.2642 lr=9.82e-05 tok=11.215B dt=310 wait=54 +18:28:39 step 85580 loss=3.1413 lr=9.82e-05 tok=11.217B dt=385 wait=55 +18:29:54 step 85600 loss=3.2239 lr=9.82e-05 tok=11.220B dt=460 wait=55 +18:31:09 step 85620 loss=3.1384 lr=9.81e-05 tok=11.222B dt=534 wait=55 +18:32:23 step 85640 loss=3.1746 lr=9.81e-05 tok=11.225B dt=609 wait=56 +18:33:16 pushed ckpt_w41.pt,log_w41.txt +18:33:16 waited 0s for global r=202 +18:33:26 global r=202 load 4.265 -> 4.278 +18:34:24 step 85660 loss=3.1990 lr=9.81e-05 tok=11.228B dt=729 wait=56 +18:36:10 step 85680 loss=3.1031 lr=9.81e-05 tok=11.230B dt=835 wait=57 +18:37:56 step 85700 loss=3.1734 lr=9.81e-05 tok=11.233B dt=941 wait=57 +18:39:42 step 85720 loss=3.2046 lr=9.80e-05 tok=11.235B dt=1048 wait=57 +18:41:24 step 85740 loss=3.1117 lr=9.80e-05 tok=11.238B dt=1150 wait=58 +18:43:14 step 85760 loss=3.2418 lr=9.80e-05 tok=11.241B dt=1260 wait=58 +18:45:04 step 85780 loss=3.1276 lr=9.80e-05 tok=11.243B dt=1370 wait=58 +18:46:54 step 85800 loss=3.1787 lr=9.79e-05 tok=11.246B dt=1480 wait=59 +18:47:06 pushed ckpt_w41.pt,log_w41.txt +18:48:22 waited 76s for global r=203 +18:48:30 global r=203 load 4.277 -> 4.259 +18:50:26 step 85820 loss=3.1561 lr=9.79e-05 tok=11.249B dt=1691 wait=59 +launch +18:57:32 name=w41 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:57:36 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:57:47 params 1.08B +18:57:49 resumed step=84450 seen=11.07B round=203 +18:57:49 stream salt=84450 +18:59:37 start: accum=4, 131072 tokens per optimiser step +19:00:40 step 84451 loss=3.1185 lr=9.93e-05 tok=11.069B dt=62 wait=44 +19:00:44 step 84452 loss=3.0748 lr=9.93e-05 tok=11.069B dt=67 wait=45 +19:01:20 step 84460 loss=3.1663 lr=9.93e-05 tok=11.070B dt=103 wait=45 +19:02:43 step 84480 loss=3.2422 lr=9.93e-05 tok=11.073B dt=186 wait=45 +19:03:58 step 84500 loss=3.1808 lr=9.93e-05 tok=11.076B dt=261 wait=46 +19:05:13 step 84520 loss=3.2638 lr=9.93e-05 tok=11.078B dt=336 wait=46 +19:06:28 step 84540 loss=3.2076 lr=9.92e-05 tok=11.081B dt=411 wait=46 +19:07:42 step 84560 loss=3.0735 lr=9.92e-05 tok=11.083B dt=485 wait=47 +19:08:57 step 84580 loss=3.0722 lr=9.92e-05 tok=11.086B dt=559 wait=47 +19:10:11 step 84600 loss=3.1759 lr=9.92e-05 tok=11.089B dt=634 wait=47 +19:10:27 pushed ckpt_w41.pt,log_w41.txt +19:13:13 waited 166s for global r=204 +19:13:23 global r=204 load 4.253 -> 4.250 +19:14:44 step 84620 loss=3.0963 lr=9.92e-05 tok=11.091B dt=906 wait=48 +19:15:58 step 84640 loss=3.1295 lr=9.91e-05 tok=11.094B dt=981 wait=48 +19:17:13 step 84660 loss=3.1295 lr=9.91e-05 tok=11.097B dt=1056 wait=48 +19:18:27 step 84680 loss=3.1706 lr=9.91e-05 tok=11.099B dt=1130 wait=49 +19:19:42 step 84700 loss=3.1556 lr=9.91e-05 tok=11.102B dt=1205 wait=49 +19:20:57 step 84720 loss=3.1382 lr=9.91e-05 tok=11.104B dt=1280 wait=49 +19:22:12 step 84740 loss=3.1458 lr=9.90e-05 tok=11.107B dt=1355 wait=50 +19:23:04 pushed ckpt_w41.pt,log_w41.txt +19:25:04 waited 121s for global r=205 +19:25:11 global r=205 load 4.263 -> 4.250 +19:25:54 step 84760 loss=3.1300 lr=9.90e-05 tok=11.110B dt=1577 wait=50 +19:27:09 step 84780 loss=3.1352 lr=9.90e-05 tok=11.112B dt=1651 wait=50 +19:28:24 step 84800 loss=3.1994 lr=9.90e-05 tok=11.115B dt=1726 wait=51 +19:29:39 step 84820 loss=3.2193 lr=9.90e-05 tok=11.118B dt=1801 wait=51 +19:30:53 step 84840 loss=3.1472 lr=9.89e-05 tok=11.120B dt=1876 wait=51 +19:32:08 step 84860 loss=3.2601 lr=9.89e-05 tok=11.123B dt=1951 wait=52 +19:33:23 step 84880 loss=3.1276 lr=9.89e-05 tok=11.125B dt=2026 wait=52 +launch +19:39:09 name=w41 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:39:14 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:39:23 params 1.08B +19:39:25 resumed step=84450 seen=11.07B round=206 +19:39:25 stream salt=84450 +19:41:17 start: accum=4, 131072 tokens per optimiser step +19:42:24 step 84451 loss=3.1185 lr=9.93e-05 tok=11.069B dt=67 wait=48 +19:42:29 step 84452 loss=3.0748 lr=9.93e-05 tok=11.069B dt=71 wait=48 +19:43:06 step 84460 loss=3.1663 lr=9.93e-05 tok=11.070B dt=109 wait=49 +19:44:27 step 84480 loss=3.2422 lr=9.93e-05 tok=11.073B dt=190 wait=49 +19:45:42 step 84500 loss=3.1808 lr=9.93e-05 tok=11.076B dt=265 wait=49 +19:46:57 step 84520 loss=3.2638 lr=9.93e-05 tok=11.078B dt=339 wait=50 +19:48:12 step 84540 loss=3.2076 lr=9.92e-05 tok=11.081B dt=414 wait=50 +19:49:26 step 84560 loss=3.0735 lr=9.92e-05 tok=11.083B dt=489 wait=50 +19:50:40 step 84580 loss=3.0722 lr=9.92e-05 tok=11.086B dt=563 wait=51 +19:51:55 step 84600 loss=3.1759 lr=9.92e-05 tok=11.089B dt=637 wait=51 +19:52:02 pushed ckpt_w41.pt,log_w41.txt +19:53:48 waited 106s for global r=207 +19:54:02 global r=207 load 4.253 -> 4.240 +19:55:20 step 84620 loss=3.0838 lr=9.92e-05 tok=11.091B dt=843 wait=51 +19:56:35 step 84640 loss=3.1192 lr=9.91e-05 tok=11.094B dt=917 wait=52 +19:57:50 step 84660 loss=3.1208 lr=9.91e-05 tok=11.097B dt=992 wait=52 +19:59:04 step 84680 loss=3.1626 lr=9.91e-05 tok=11.099B dt=1067 wait=53 +20:00:19 step 84700 loss=3.1483 lr=9.91e-05 tok=11.102B dt=1141 wait=53 +20:01:34 step 84720 loss=3.1310 lr=9.91e-05 tok=11.104B dt=1216 wait=53 +20:02:49 step 84740 loss=3.1394 lr=9.90e-05 tok=11.107B dt=1291 wait=54 +20:03:41 pushed ckpt_w41.pt,log_w41.txt +20:05:27 waited 106s for global r=208 +20:05:38 global r=208 load 4.258 -> 4.236 +20:06:20 step 84760 loss=3.1177 lr=9.90e-05 tok=11.110B dt=1503 wait=54 +20:07:35 step 84780 loss=3.1191 lr=9.90e-05 tok=11.112B dt=1578 wait=54 +20:08:50 step 84800 loss=3.1849 lr=9.90e-05 tok=11.115B dt=1652 wait=55 +20:10:05 step 84820 loss=3.2071 lr=9.90e-05 tok=11.118B dt=1727 wait=55 +20:11:19 step 84840 loss=3.1350 lr=9.89e-05 tok=11.120B dt=1802 wait=55 +20:12:34 step 84860 loss=3.2487 lr=9.89e-05 tok=11.123B dt=1877 wait=56 +20:13:49 step 84880 loss=3.1166 lr=9.89e-05 tok=11.125B dt=1952 wait=56 +20:15:04 step 84900 loss=3.1690 lr=9.89e-05 tok=11.128B dt=2026 wait=56 +20:15:19 pushed ckpt_w41.pt,log_w41.txt +20:16:50 waited 91s for global r=209 +20:16:58 global r=209 load 4.255 -> 4.236 +20:18:18 step 84920 loss=3.0845 lr=9.89e-05 tok=11.131B dt=2221 wait=57 +20:19:33 step 84940 loss=3.1027 lr=9.88e-05 tok=11.133B dt=2296 wait=57 +20:20:48 step 84960 loss=3.1048 lr=9.88e-05 tok=11.136B dt=2370 wait=57 +20:22:02 step 84980 loss=3.1771 lr=9.88e-05 tok=11.138B dt=2445 wait=58 +20:23:17 step 85000 loss=3.1530 lr=9.88e-05 tok=11.141B dt=2519 wait=58 +20:24:31 step 85020 loss=3.0978 lr=9.88e-05 tok=11.144B dt=2594 wait=58 +20:25:46 step 85040 loss=3.1236 lr=9.87e-05 tok=11.146B dt=2669 wait=59 +20:26:37 pushed ckpt_w41.pt,log_w41.txt +20:27:53 waited 76s for global r=210 +20:28:02 global r=210 load 4.252 -> 4.240 +20:28:45 step 85060 loss=3.1334 lr=9.87e-05 tok=11.149B dt=2847 wait=59 +20:29:59 step 85080 loss=3.1195 lr=9.87e-05 tok=11.152B dt=2922 wait=60 +20:31:14 step 85100 loss=3.1950 lr=9.87e-05 tok=11.154B dt=2997 wait=60 +20:32:29 step 85120 loss=3.1816 lr=9.87e-05 tok=11.157B dt=3071 wait=60 +20:33:44 step 85140 loss=3.1316 lr=9.86e-05 tok=11.159B dt=3146 wait=61 +20:34:58 step 85160 loss=3.2090 lr=9.86e-05 tok=11.162B dt=3221 wait=61 +20:36:13 step 85180 loss=3.1013 lr=9.86e-05 tok=11.165B dt=3296 wait=61 +20:37:28 step 85200 loss=3.1490 lr=9.86e-05 tok=11.167B dt=3371 wait=62 +20:37:40 pushed ckpt_w41.pt,log_w41.txt +20:39:26 waited 106s for global r=211 +20:39:34 global r=211 load 4.257 -> 4.237 +20:40:56 step 85220 loss=3.1595 lr=9.85e-05 tok=11.170B dt=3578 wait=62 +20:42:10 step 85240 loss=3.1399 lr=9.85e-05 tok=11.173B dt=3653 wait=62 +20:43:25 step 85260 loss=3.1405 lr=9.85e-05 tok=11.175B dt=3727 wait=63 +20:44:39 step 85280 loss=3.2000 lr=9.85e-05 tok=11.178B dt=3802 wait=63 +20:45:54 step 85300 loss=3.1580 lr=9.85e-05 tok=11.180B dt=3877 wait=63 +20:47:09 step 85320 loss=3.1137 lr=9.84e-05 tok=11.183B dt=3951 wait=64 +20:48:23 step 85340 loss=3.1473 lr=9.84e-05 tok=11.186B dt=4026 wait=64 +20:49:12 pushed ckpt_w41.pt,log_w41.txt +20:50:59 waited 106s for global r=212 +20:51:10 global r=212 load 4.252 -> 4.235 +20:51:54 step 85360 loss=3.1019 lr=9.84e-05 tok=11.188B dt=4236 wait=64 +20:53:09 step 85380 loss=3.1253 lr=9.84e-05 tok=11.191B dt=4311 wait=65 +20:54:23 step 85400 loss=3.1746 lr=9.84e-05 tok=11.194B dt=4386 wait=65 +20:55:38 step 85420 loss=3.1961 lr=9.83e-05 tok=11.196B dt=4461 wait=65 +20:56:53 step 85440 loss=3.0883 lr=9.83e-05 tok=11.199B dt=4535 wait=66 +20:58:08 step 85460 loss=3.1951 lr=9.83e-05 tok=11.201B dt=4610 wait=66 +20:59:22 step 85480 loss=3.0937 lr=9.83e-05 tok=11.204B dt=4685 wait=67 +21:00:37 step 85500 loss=3.1424 lr=9.83e-05 tok=11.207B dt=4760 wait=67 +21:00:49 pushed ckpt_w41.pt,log_w41.txt +21:02:50 waited 121s for global r=213 +21:02:58 global r=213 load 4.254 -> 4.238 +21:04:23 step 85520 loss=3.1165 lr=9.82e-05 tok=11.209B dt=4986 wait=67 +21:05:38 step 85540 loss=3.1495 lr=9.82e-05 tok=11.212B dt=5060 wait=68 +21:06:53 step 85560 loss=3.1519 lr=9.82e-05 tok=11.215B dt=5135 wait=68 +21:08:07 step 85580 loss=3.1837 lr=9.82e-05 tok=11.217B dt=5210 wait=68 +21:09:22 step 85600 loss=3.1333 lr=9.82e-05 tok=11.220B dt=5284 wait=69 +21:10:36 step 85620 loss=3.1074 lr=9.81e-05 tok=11.222B dt=5359 wait=69 +21:11:51 step 85640 loss=3.1607 lr=9.81e-05 tok=11.225B dt=5434 wait=69 +21:12:44 pushed ckpt_w41.pt,log_w41.txt +21:16:26 step 85660 loss=3.1227 lr=9.81e-05 tok=11.228B dt=5709 wait=70 +21:17:41 step 85680 loss=3.1964 lr=9.81e-05 tok=11.230B dt=5783 wait=70 +21:18:56 step 85700 loss=3.0827 lr=9.81e-05 tok=11.233B dt=5858 wait=70 +21:20:10 step 85720 loss=3.0710 lr=9.80e-05 tok=11.235B dt=5933 wait=71 +21:21:25 step 85740 loss=3.1420 lr=9.80e-05 tok=11.238B dt=6007 wait=71 +21:22:40 step 85760 loss=3.1195 lr=9.80e-05 tok=11.241B dt=6082 wait=71 +21:23:54 step 85780 loss=3.1532 lr=9.80e-05 tok=11.243B dt=6157 wait=72 +21:25:09 step 85800 loss=3.1057 lr=9.79e-05 tok=11.246B dt=6232 wait=72 +21:25:21 pushed ckpt_w41.pt,log_w41.txt +21:29:42 step 85820 loss=3.0808 lr=9.79e-05 tok=11.249B dt=6505 wait=72 +21:30:57 step 85840 loss=3.1575 lr=9.79e-05 tok=11.251B dt=6580 wait=73 +21:32:12 step 85860 loss=3.1925 lr=9.79e-05 tok=11.254B dt=6654 wait=73 +launch +21:35:20 name=w41 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:35:25 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:35:34 params 1.08B +21:35:36 resumed step=85500 seen=11.21B round=214 +21:35:36 stream salt=85500 +21:37:33 start: accum=4, 131072 tokens per optimiser step +21:38:48 step 85501 loss=3.1105 lr=9.83e-05 tok=11.207B dt=75 wait=55 +21:38:53 step 85502 loss=3.0166 lr=9.83e-05 tok=11.207B dt=81 wait=55 +21:40:19 step 85520 loss=3.2111 lr=9.82e-05 tok=11.209B dt=166 wait=56 +21:41:36 step 85540 loss=3.1951 lr=9.82e-05 tok=11.212B dt=244 wait=56 +21:42:51 step 85560 loss=3.2263 lr=9.82e-05 tok=11.215B dt=319 wait=56 +21:44:06 step 85580 loss=3.1388 lr=9.82e-05 tok=11.217B dt=393 wait=57 +21:45:21 step 85600 loss=3.2056 lr=9.82e-05 tok=11.220B dt=468 wait=57 +21:46:36 step 85620 loss=3.0823 lr=9.81e-05 tok=11.222B dt=543 wait=57 +21:47:51 step 85640 loss=3.0576 lr=9.81e-05 tok=11.225B dt=618 wait=58 +21:48:42 pushed ckpt_w41.pt,log_w41.txt +21:50:28 waited 106s for global r=215 +21:50:44 global r=215 load 4.260 -> 4.245 +21:51:27 step 85660 loss=3.1717 lr=9.81e-05 tok=11.228B dt=835 wait=58 +21:52:42 step 85680 loss=3.0926 lr=9.81e-05 tok=11.230B dt=909 wait=59 +21:53:57 step 85700 loss=3.1636 lr=9.81e-05 tok=11.233B dt=984 wait=59 +21:55:12 step 85720 loss=3.2135 lr=9.80e-05 tok=11.235B dt=1059 wait=59 +21:56:27 step 85740 loss=3.1226 lr=9.80e-05 tok=11.238B dt=1134 wait=60 +21:57:42 step 85760 loss=3.2035 lr=9.80e-05 tok=11.241B dt=1209 wait=60 +21:58:57 step 85780 loss=3.0650 lr=9.80e-05 tok=11.243B dt=1284 wait=61 +22:00:12 step 85800 loss=3.1187 lr=9.79e-05 tok=11.246B dt=1359 wait=61 +22:00:24 pushed ckpt_w41.pt,log_w41.txt +22:01:55 waited 91s for global r=216 +22:02:05 global r=216 load 4.259 -> 4.243 +22:03:27 step 85820 loss=3.1259 lr=9.79e-05 tok=11.249B dt=1554 wait=61 +22:04:42 step 85840 loss=3.1042 lr=9.79e-05 tok=11.251B dt=1629 wait=62 +22:05:57 step 85860 loss=3.1043 lr=9.79e-05 tok=11.254B dt=1704 wait=62 +22:07:11 step 85880 loss=3.1527 lr=9.79e-05 tok=11.256B dt=1779 wait=63 +22:08:26 step 85900 loss=3.1524 lr=9.78e-05 tok=11.259B dt=1853 wait=63 +22:09:41 step 85920 loss=3.1312 lr=9.78e-05 tok=11.262B dt=1928 wait=63 +22:10:56 step 85940 loss=3.1565 lr=9.78e-05 tok=11.264B dt=2003 wait=64 +22:11:46 pushed ckpt_w41.pt,log_w41.txt +22:13:01 waited 76s for global r=217 +22:13:09 global r=217 load 4.257 -> 4.242 +22:13:52 step 85960 loss=3.1011 lr=9.78e-05 tok=11.267B dt=2179 wait=64 +22:15:07 step 85980 loss=3.1114 lr=9.78e-05 tok=11.270B dt=2254 wait=65 +22:16:22 step 86000 loss=3.1898 lr=9.77e-05 tok=11.272B dt=2329 wait=65 +22:17:37 step 86020 loss=3.1854 lr=9.77e-05 tok=11.275B dt=2404 wait=65 +22:18:51 step 86040 loss=3.1075 lr=9.77e-05 tok=11.277B dt=2478 wait=66 +22:20:06 step 86060 loss=3.1638 lr=9.77e-05 tok=11.280B dt=2553 wait=66 +22:21:21 step 86080 loss=3.0805 lr=9.77e-05 tok=11.283B dt=2628 wait=66 +22:22:36 step 86100 loss=3.1126 lr=9.76e-05 tok=11.285B dt=2703 wait=67 +22:22:48 pushed ckpt_w41.pt,log_w41.txt +22:24:20 waited 92s for global r=218 +22:24:27 global r=218 load 4.254 -> 4.238 +22:25:46 step 86120 loss=3.1295 lr=9.76e-05 tok=11.288B dt=2893 wait=67 +22:27:01 step 86140 loss=3.1283 lr=9.76e-05 tok=11.291B dt=2968 wait=68 +22:28:16 step 86160 loss=3.1005 lr=9.76e-05 tok=11.293B dt=3043 wait=68 +22:29:30 step 86180 loss=3.1435 lr=9.76e-05 tok=11.296B dt=3118 wait=68 +22:30:45 step 86200 loss=3.1785 lr=9.75e-05 tok=11.298B dt=3192 wait=69 +22:32:00 step 86220 loss=3.0892 lr=9.75e-05 tok=11.301B dt=3267 wait=69 +22:33:15 step 86240 loss=3.1257 lr=9.75e-05 tok=11.304B dt=3342 wait=70 +22:34:08 pushed ckpt_w41.pt,log_w41.txt +22:35:23 waited 76s for global r=219 +22:35:30 global r=219 load 4.255 -> 4.235 +22:36:13 step 86260 loss=3.1013 lr=9.75e-05 tok=11.306B dt=3520 wait=70 +22:37:28 step 86280 loss=3.1096 lr=9.75e-05 tok=11.309B dt=3595 wait=70 +22:38:43 step 86300 loss=3.1667 lr=9.74e-05 tok=11.312B dt=3670 wait=71 +22:39:58 step 86320 loss=3.2345 lr=9.74e-05 tok=11.314B dt=3745 wait=71 +22:41:13 step 86340 loss=3.1122 lr=9.74e-05 tok=11.317B dt=3820 wait=72 +22:42:28 step 86360 loss=3.1890 lr=9.74e-05 tok=11.319B dt=3895 wait=72 +22:43:43 step 86380 loss=3.0878 lr=9.73e-05 tok=11.322B dt=3970 wait=72 +22:44:58 step 86400 loss=3.1573 lr=9.73e-05 tok=11.325B dt=4045 wait=73 +22:45:10 pushed ckpt_w41.pt,log_w41.txt +22:46:26 waited 76s for global r=220 +22:46:31 global r=220 load 4.262 -> 4.236 +22:47:50 step 86420 loss=3.1266 lr=9.73e-05 tok=11.327B dt=4217 wait=73 +22:49:05 step 86440 loss=3.1076 lr=9.73e-05 tok=11.330B dt=4292 wait=74 +22:50:20 step 86460 loss=3.1299 lr=9.73e-05 tok=11.332B dt=4367 wait=74 +22:51:34 step 86480 loss=3.1358 lr=9.72e-05 tok=11.335B dt=4441 wait=74 +22:52:49 step 86500 loss=3.1299 lr=9.72e-05 tok=11.338B dt=4516 wait=75 +22:54:04 step 86520 loss=3.0961 lr=9.72e-05 tok=11.340B dt=4591 wait=75 +22:55:19 step 86540 loss=3.1429 lr=9.72e-05 tok=11.343B dt=4666 wait=76 +22:56:09 pushed ckpt_w41.pt,log_w41.txt +22:57:39 waited 91s for global r=221 +22:57:44 global r=221 load 4.257 -> 4.232 +22:58:29 step 86560 loss=3.0518 lr=9.72e-05 tok=11.346B dt=4857 wait=76 +22:59:44 step 86580 loss=3.1149 lr=9.71e-05 tok=11.348B dt=4931 wait=76 +23:00:59 step 86600 loss=3.1602 lr=9.71e-05 tok=11.351B dt=5006 wait=77 +23:02:14 step 86620 loss=3.1848 lr=9.71e-05 tok=11.353B dt=5081 wait=77 +23:03:29 step 86640 loss=3.1207 lr=9.71e-05 tok=11.356B dt=5156 wait=77 +23:04:44 step 86660 loss=3.2011 lr=9.71e-05 tok=11.359B dt=5231 wait=78 +23:05:59 step 86680 loss=3.0601 lr=9.70e-05 tok=11.361B dt=5306 wait=78 +23:07:13 step 86700 loss=3.1162 lr=9.70e-05 tok=11.364B dt=5381 wait=79 +23:07:27 pushed ckpt_w41.pt,log_w41.txt +23:08:42 waited 76s for global r=222 +23:08:49 global r=222 load 4.246 -> 4.228 +23:10:15 step 86720 loss=3.1207 lr=9.70e-05 tok=11.367B dt=5562 wait=79 +23:11:30 step 86740 loss=3.1538 lr=9.70e-05 tok=11.369B dt=5637 wait=79 +23:12:45 step 86760 loss=3.1300 lr=9.70e-05 tok=11.372B dt=5712 wait=80 +23:13:59 step 86780 loss=3.1924 lr=9.69e-05 tok=11.374B dt=5787 wait=80 +23:15:15 step 86800 loss=3.1550 lr=9.69e-05 tok=11.377B dt=5862 wait=81 +23:16:29 step 86820 loss=3.1152 lr=9.69e-05 tok=11.380B dt=5937 wait=81 +23:17:44 step 86840 loss=3.1188 lr=9.69e-05 tok=11.382B dt=6012 wait=81 +23:18:34 pushed ckpt_w41.pt,log_w41.txt +23:20:19 waited 106s for global r=223 +23:20:26 global r=223 load 4.246 -> 4.226 +23:21:13 step 86860 loss=3.1027 lr=9.69e-05 tok=11.385B dt=6220 wait=82 +23:22:28 step 86880 loss=3.0928 lr=9.68e-05 tok=11.388B dt=6295 wait=82 +23:23:43 step 86900 loss=3.2100 lr=9.68e-05 tok=11.390B dt=6370 wait=82 +23:24:58 step 86920 loss=3.2166 lr=9.68e-05 tok=11.393B dt=6445 wait=83 +23:26:13 step 86940 loss=3.1283 lr=9.68e-05 tok=11.395B dt=6520 wait=83 +23:27:28 step 86960 loss=3.1790 lr=9.67e-05 tok=11.398B dt=6595 wait=84 +23:28:42 step 86980 loss=3.0741 lr=9.67e-05 tok=11.401B dt=6670 wait=84 +23:29:57 step 87000 loss=3.0859 lr=9.67e-05 tok=11.403B dt=6744 wait=84 +23:30:09 pushed ckpt_w41.pt,log_w41.txt +23:31:40 waited 91s for global r=224 +23:31:52 global r=224 load 4.253 -> 4.227 +23:33:14 step 87020 loss=3.0777 lr=9.67e-05 tok=11.406B dt=6941 wait=85 +23:34:29 step 87040 loss=3.1019 lr=9.67e-05 tok=11.409B dt=7016 wait=85 +23:35:44 step 87060 loss=3.1119 lr=9.66e-05 tok=11.411B dt=7091 wait=86 +23:36:59 step 87080 loss=3.1965 lr=9.66e-05 tok=11.414B dt=7166 wait=86 +23:38:14 step 87100 loss=3.1612 lr=9.66e-05 tok=11.416B dt=7241 wait=86 +23:39:29 step 87120 loss=3.1966 lr=9.66e-05 tok=11.419B dt=7316 wait=87 +23:40:44 step 87140 loss=3.1823 lr=9.66e-05 tok=11.422B dt=7391 wait=87 +23:41:38 pushed ckpt_w41.pt,log_w41.txt +23:43:09 waited 91s for global r=225 +23:43:19 global r=225 load 4.246 -> 4.228 +23:44:02 step 87160 loss=3.0434 lr=9.65e-05 tok=11.424B dt=7589 wait=88 +23:45:17 step 87180 loss=3.0542 lr=9.65e-05 tok=11.427B dt=7664 wait=88 +23:46:32 step 87200 loss=3.1012 lr=9.65e-05 tok=11.429B dt=7739 wait=88 +23:47:47 step 87220 loss=3.1977 lr=9.65e-05 tok=11.432B dt=7814 wait=89 +23:49:01 step 87240 loss=3.1515 lr=9.65e-05 tok=11.435B dt=7889 wait=89 +23:50:16 step 87260 loss=3.2120 lr=9.64e-05 tok=11.437B dt=7964 wait=89 +23:51:31 step 87280 loss=3.1342 lr=9.64e-05 tok=11.440B dt=8039 wait=90 +23:52:47 step 87300 loss=3.1811 lr=9.64e-05 tok=11.443B dt=8114 wait=90 +23:53:00 pushed ckpt_w41.pt,log_w41.txt +23:54:31 waited 91s for global r=226 +23:54:41 global r=226 load 4.249 -> 4.225 +23:56:05 step 87320 loss=3.0858 lr=9.64e-05 tok=11.445B dt=8312 wait=91 +23:57:20 step 87340 loss=3.0756 lr=9.64e-05 tok=11.448B dt=8387 wait=91 +23:58:35 step 87360 loss=3.0737 lr=9.63e-05 tok=11.450B dt=8462 wait=91 +23:59:50 step 87380 loss=3.1264 lr=9.63e-05 tok=11.453B dt=8537 wait=92 +00:01:05 step 87400 loss=3.1564 lr=9.63e-05 tok=11.456B dt=8612 wait=92 +00:02:20 step 87420 loss=3.0895 lr=9.63e-05 tok=11.458B dt=8687 wait=93 +00:03:35 step 87440 loss=3.0946 lr=9.63e-05 tok=11.461B dt=8762 wait=93 +00:04:29 pushed ckpt_w41.pt,log_w41.txt +00:06:00 waited 91s for global r=227 +00:06:11 global r=227 load 4.248 -> 4.228 +00:06:58 step 87460 loss=3.0623 lr=9.62e-05 tok=11.464B dt=8965 wait=93 +00:08:12 step 87480 loss=3.0793 lr=9.62e-05 tok=11.466B dt=9040 wait=94 +00:09:27 step 87500 loss=3.1761 lr=9.62e-05 tok=11.469B dt=9115 wait=94 +00:10:42 step 87520 loss=3.1691 lr=9.62e-05 tok=11.471B dt=9190 wait=95 +00:11:57 step 87540 loss=3.1165 lr=9.61e-05 tok=11.474B dt=9265 wait=95 +00:13:12 step 87560 loss=3.2384 lr=9.61e-05 tok=11.477B dt=9340 wait=95 +00:14:27 step 87580 loss=3.1082 lr=9.61e-05 tok=11.479B dt=9414 wait=96 +00:15:42 step 87600 loss=3.1162 lr=9.61e-05 tok=11.482B dt=9490 wait=96 diff --git a/log_w42.txt b/log_w42.txt new file mode 100644 index 0000000000000000000000000000000000000000..c8b30672c59c6afc9f2e074c48c69fd4483853d4 --- /dev/null +++ b/log_w42.txt @@ -0,0 +1,145 @@ +launch +14:09:54 name=w42 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +14:10:00 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +14:10:10 params 1.08B +14:10:11 resumed step=82650 seen=10.83B round=191 +14:10:11 stream salt=82650 +14:12:59 start: accum=4, 131072 tokens per optimiser step +14:15:10 step 82651 loss=3.2390 lr=1.01e-04 tok=10.833B dt=131 wait=102 +14:15:14 step 82652 loss=3.1024 lr=1.01e-04 tok=10.833B dt=135 wait=102 +14:15:50 step 82660 loss=3.1799 lr=1.01e-04 tok=10.834B dt=171 wait=102 +14:17:19 step 82680 loss=3.2321 lr=1.01e-04 tok=10.837B dt=260 wait=103 +14:18:47 step 82700 loss=3.2106 lr=1.01e-04 tok=10.840B dt=348 wait=103 +14:20:03 step 82720 loss=3.2510 lr=1.01e-04 tok=10.842B dt=424 wait=103 +14:21:18 step 82740 loss=3.1924 lr=1.01e-04 tok=10.845B dt=499 wait=104 +14:22:33 step 82760 loss=3.1114 lr=1.01e-04 tok=10.848B dt=574 wait=104 +14:23:48 step 82780 loss=3.1186 lr=1.01e-04 tok=10.850B dt=649 wait=104 +14:25:03 step 82800 loss=3.1865 lr=1.01e-04 tok=10.853B dt=724 wait=105 +14:25:20 pushed ckpt_w42.pt,log_w42.txt +14:26:36 waited 76s for global r=192 +14:26:57 global r=192 load 4.261 -> 4.240 +14:28:18 step 82820 loss=3.1140 lr=1.01e-04 tok=10.855B dt=919 wait=105 +14:29:33 step 82840 loss=3.1394 lr=1.01e-04 tok=10.858B dt=994 wait=106 +14:30:48 step 82860 loss=3.1268 lr=1.01e-04 tok=10.861B dt=1069 wait=106 +14:32:03 step 82880 loss=3.1947 lr=1.01e-04 tok=10.863B dt=1144 wait=106 +14:33:19 step 82900 loss=3.1467 lr=1.01e-04 tok=10.866B dt=1219 wait=107 +14:34:34 step 82920 loss=3.1173 lr=1.01e-04 tok=10.868B dt=1295 wait=107 +14:35:49 step 82940 loss=3.1913 lr=1.01e-04 tok=10.871B dt=1370 wait=107 +14:36:39 pushed ckpt_w42.pt,log_w42.txt +14:37:55 waited 76s for global r=193 +14:38:08 global r=193 load 4.255 -> 4.234 +14:38:55 step 82960 loss=3.1520 lr=1.01e-04 tok=10.874B dt=1556 wait=108 +14:40:10 step 82980 loss=3.1304 lr=1.01e-04 tok=10.876B dt=1631 wait=108 +14:41:25 step 83000 loss=3.2013 lr=1.01e-04 tok=10.879B dt=1706 wait=108 +14:42:41 step 83020 loss=3.1728 lr=1.01e-04 tok=10.882B dt=1782 wait=109 +14:43:56 step 83040 loss=3.1403 lr=1.01e-04 tok=10.884B dt=1857 wait=109 +14:45:11 step 83060 loss=3.2128 lr=1.01e-04 tok=10.887B dt=1932 wait=109 +14:46:26 step 83080 loss=3.0761 lr=1.01e-04 tok=10.889B dt=2006 wait=110 +14:47:41 step 83100 loss=3.1363 lr=1.01e-04 tok=10.892B dt=2082 wait=110 +14:47:56 pushed ckpt_w42.pt,log_w42.txt +14:50:42 waited 166s for global r=194 +14:50:54 global r=194 load 4.255 -> 4.234 +14:52:16 step 83120 loss=3.0801 lr=1.01e-04 tok=10.895B dt=2357 wait=110 +14:53:31 step 83140 loss=3.1326 lr=1.01e-04 tok=10.897B dt=2432 wait=111 +14:54:46 step 83160 loss=3.1223 lr=1.01e-04 tok=10.900B dt=2507 wait=111 +14:56:01 step 83180 loss=3.1943 lr=1.01e-04 tok=10.903B dt=2582 wait=111 +14:57:16 step 83200 loss=3.1836 lr=1.01e-04 tok=10.905B dt=2657 wait=112 +14:58:32 step 83220 loss=3.1365 lr=1.01e-04 tok=10.908B dt=2732 wait=112 +14:59:47 step 83240 loss=3.1627 lr=1.01e-04 tok=10.910B dt=2808 wait=112 +15:00:38 pushed ckpt_w42.pt,log_w42.txt +15:04:28 step 83260 loss=3.1218 lr=1.01e-04 tok=10.913B dt=3089 wait=112 +15:05:43 step 83280 loss=3.1396 lr=1.01e-04 tok=10.916B dt=3164 wait=113 +15:06:58 step 83300 loss=3.0552 lr=1.01e-04 tok=10.918B dt=3239 wait=113 +15:08:13 step 83320 loss=3.0489 lr=1.00e-04 tok=10.921B dt=3314 wait=113 +15:09:28 step 83340 loss=3.1059 lr=1.00e-04 tok=10.924B dt=3389 wait=114 +15:10:43 step 83360 loss=3.1554 lr=1.00e-04 tok=10.926B dt=3464 wait=114 +15:11:58 step 83380 loss=3.1554 lr=1.00e-04 tok=10.929B dt=3539 wait=114 +15:13:13 step 83400 loss=3.1354 lr=1.00e-04 tok=10.931B dt=3614 wait=115 +15:13:29 pushed ckpt_w42.pt,log_w42.txt +15:13:29 waited 0s for global r=195 +15:13:47 global r=195 load 4.263 -> 4.236 +15:15:09 step 83420 loss=3.1166 lr=1.00e-04 tok=10.934B dt=3730 wait=115 +15:16:24 step 83440 loss=3.1602 lr=1.00e-04 tok=10.937B dt=3805 wait=115 +15:17:40 step 83460 loss=3.1487 lr=1.00e-04 tok=10.939B dt=3880 wait=116 +15:18:55 step 83480 loss=3.2030 lr=1.00e-04 tok=10.942B dt=3956 wait=116 +15:20:10 step 83500 loss=3.1670 lr=1.00e-04 tok=10.945B dt=4031 wait=116 +15:21:25 step 83520 loss=3.1184 lr=1.00e-04 tok=10.947B dt=4106 wait=117 +15:22:40 step 83540 loss=3.1253 lr=1.00e-04 tok=10.950B dt=4181 wait=117 +15:23:29 pushed ckpt_w42.pt,log_w42.txt +15:24:30 waited 61s for global r=196 +15:24:41 global r=196 load 4.249 -> 4.231 +15:25:28 step 83560 loss=3.0858 lr=1.00e-04 tok=10.952B dt=4349 wait=118 +15:26:43 step 83580 loss=3.1044 lr=1.00e-04 tok=10.955B dt=4424 wait=118 +15:27:58 step 83600 loss=3.1270 lr=1.00e-04 tok=10.958B dt=4499 wait=118 +15:29:13 step 83620 loss=3.1942 lr=1.00e-04 tok=10.960B dt=4574 wait=118 +15:30:28 step 83640 loss=3.0928 lr=1.00e-04 tok=10.963B dt=4649 wait=119 +15:31:44 step 83660 loss=3.2106 lr=1.00e-04 tok=10.965B dt=4725 wait=119 +15:32:59 step 83680 loss=3.1131 lr=1.00e-04 tok=10.968B dt=4800 wait=120 +15:34:14 step 83700 loss=3.1476 lr=1.00e-04 tok=10.971B dt=4875 wait=120 +15:34:28 pushed ckpt_w42.pt,log_w42.txt +15:38:54 step 83720 loss=3.1991 lr=1.00e-04 tok=10.973B dt=5155 wait=120 +15:40:09 step 83740 loss=3.1135 lr=1.00e-04 tok=10.976B dt=5230 wait=121 +15:41:24 step 83760 loss=3.0923 lr=1.00e-04 tok=10.979B dt=5305 wait=121 +15:42:39 step 83780 loss=3.0561 lr=1.00e-04 tok=10.981B dt=5380 wait=121 +15:43:54 step 83800 loss=3.0550 lr=1.00e-04 tok=10.984B dt=5455 wait=122 +15:45:09 step 83820 loss=3.1464 lr=1.00e-04 tok=10.986B dt=5530 wait=122 +15:46:24 step 83840 loss=3.1057 lr=1.00e-04 tok=10.989B dt=5605 wait=122 +15:47:14 pushed ckpt_w42.pt,log_w42.txt +15:47:14 waited 0s for global r=197 +15:47:29 global r=197 load 4.252 -> 4.236 +15:48:14 step 83860 loss=3.1178 lr=9.99e-05 tok=10.992B dt=5715 wait=123 +15:49:29 step 83880 loss=3.1375 lr=9.99e-05 tok=10.994B dt=5790 wait=123 +15:50:45 step 83900 loss=3.1522 lr=9.99e-05 tok=10.997B dt=5866 wait=123 +15:52:00 step 83920 loss=3.1877 lr=9.99e-05 tok=11.000B dt=5941 wait=124 +15:53:15 step 83940 loss=3.1089 lr=9.99e-05 tok=11.002B dt=6016 wait=124 +15:54:30 step 83960 loss=3.1867 lr=9.98e-05 tok=11.005B dt=6091 wait=124 +15:55:45 step 83980 loss=3.0682 lr=9.98e-05 tok=11.007B dt=6166 wait=125 +15:57:00 step 84000 loss=3.1146 lr=9.98e-05 tok=11.010B dt=6241 wait=125 +15:57:15 pushed ckpt_w42.pt,log_w42.txt +15:59:32 waited 136s for global r=198 +15:59:50 global r=198 load 4.256 -> 4.239 +16:01:17 step 84020 loss=3.1036 lr=9.98e-05 tok=11.013B dt=6498 wait=125 +16:02:32 step 84040 loss=3.1362 lr=9.98e-05 tok=11.015B dt=6573 wait=126 +16:03:47 step 84060 loss=3.1367 lr=9.97e-05 tok=11.018B dt=6648 wait=126 +16:05:02 step 84080 loss=3.1916 lr=9.97e-05 tok=11.021B dt=6723 wait=126 +16:06:18 step 84100 loss=3.1833 lr=9.97e-05 tok=11.023B dt=6798 wait=127 +16:07:33 step 84120 loss=3.1115 lr=9.97e-05 tok=11.026B dt=6874 wait=127 +16:08:48 step 84140 loss=3.1644 lr=9.97e-05 tok=11.028B dt=6949 wait=127 +16:09:40 pushed ckpt_w42.pt,log_w42.txt +16:13:24 step 84160 loss=3.1383 lr=9.96e-05 tok=11.031B dt=7225 wait=128 +16:14:39 step 84180 loss=3.1438 lr=9.96e-05 tok=11.034B dt=7300 wait=128 +16:15:54 step 84200 loss=3.0737 lr=9.96e-05 tok=11.036B dt=7375 wait=128 +16:17:09 step 84220 loss=3.0359 lr=9.96e-05 tok=11.039B dt=7450 wait=129 +16:18:24 step 84240 loss=3.1054 lr=9.96e-05 tok=11.042B dt=7525 wait=129 +16:19:39 step 84260 loss=3.0889 lr=9.95e-05 tok=11.044B dt=7600 wait=129 +16:20:54 step 84280 loss=3.1484 lr=9.95e-05 tok=11.047B dt=7675 wait=129 +16:22:09 step 84300 loss=3.0875 lr=9.95e-05 tok=11.049B dt=7750 wait=130 +16:22:21 pushed ckpt_w42.pt,log_w42.txt +16:22:21 waited 0s for global r=199 +16:22:37 global r=199 load 4.253 -> 4.234 +16:23:57 step 84320 loss=3.1187 lr=9.95e-05 tok=11.052B dt=7858 wait=130 +16:25:12 step 84340 loss=3.1501 lr=9.95e-05 tok=11.055B dt=7933 wait=130 +16:26:27 step 84360 loss=3.1228 lr=9.94e-05 tok=11.057B dt=8008 wait=131 +16:27:42 step 84380 loss=3.1610 lr=9.94e-05 tok=11.060B dt=8083 wait=131 +16:28:58 step 84400 loss=3.1478 lr=9.94e-05 tok=11.062B dt=8159 wait=131 +16:30:13 step 84420 loss=3.0945 lr=9.94e-05 tok=11.065B dt=8234 wait=132 +16:31:28 step 84440 loss=3.1455 lr=9.93e-05 tok=11.068B dt=8309 wait=132 +16:32:19 pushed ckpt_w42.pt,log_w42.txt +16:34:05 waited 106s for global r=200 +16:34:20 global r=200 load 4.259 -> 4.235 +16:35:02 step 84460 loss=3.1210 lr=9.93e-05 tok=11.070B dt=8523 wait=132 +16:36:17 step 84480 loss=3.0934 lr=9.93e-05 tok=11.073B dt=8598 wait=133 +16:37:32 step 84500 loss=3.1552 lr=9.93e-05 tok=11.076B dt=8673 wait=133 +16:38:47 step 84520 loss=3.1943 lr=9.93e-05 tok=11.078B dt=8748 wait=133 +16:40:02 step 84540 loss=3.1084 lr=9.92e-05 tok=11.081B dt=8823 wait=134 +16:41:17 step 84560 loss=3.1947 lr=9.92e-05 tok=11.083B dt=8898 wait=134 +16:42:32 step 84580 loss=3.1046 lr=9.92e-05 tok=11.086B dt=8973 wait=134 +16:43:48 step 84600 loss=3.1661 lr=9.92e-05 tok=11.089B dt=9049 wait=135 +16:44:00 pushed ckpt_w42.pt,log_w42.txt +16:48:20 step 84620 loss=3.1826 lr=9.92e-05 tok=11.091B dt=9321 wait=135 +16:49:36 step 84640 loss=3.1411 lr=9.91e-05 tok=11.094B dt=9396 wait=135 +16:50:50 step 84660 loss=3.1190 lr=9.91e-05 tok=11.097B dt=9471 wait=136 +16:52:06 step 84680 loss=3.1288 lr=9.91e-05 tok=11.099B dt=9547 wait=136 +16:53:21 step 84700 loss=3.0854 lr=9.91e-05 tok=11.102B dt=9622 wait=136 +16:54:36 step 84720 loss=3.2070 lr=9.91e-05 tok=11.104B dt=9697 wait=137 +16:55:51 step 84740 loss=3.1349 lr=9.90e-05 tok=11.107B dt=9772 wait=137 diff --git a/log_w43.txt b/log_w43.txt new file mode 100644 index 0000000000000000000000000000000000000000..a9b72e2612fe1ea56c7e79c4e54e7eb377f85e8a --- /dev/null +++ b/log_w43.txt @@ -0,0 +1,153 @@ +launch +21:35:37 name=w43 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:35:44 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:35:54 params 1.08B +21:35:56 resumed step=85500 seen=11.21B round=214 +21:35:56 stream salt=85500 +21:37:58 start: accum=4, 131072 tokens per optimiser step +21:39:22 step 85501 loss=3.1914 lr=9.83e-05 tok=11.207B dt=84 wait=46 +21:39:28 step 85502 loss=3.0733 lr=9.83e-05 tok=11.207B dt=89 wait=46 +21:40:59 step 85520 loss=3.2480 lr=9.82e-05 tok=11.209B dt=181 wait=47 +21:42:15 step 85540 loss=3.2227 lr=9.82e-05 tok=11.212B dt=256 wait=48 +21:43:30 step 85560 loss=3.2286 lr=9.82e-05 tok=11.215B dt=332 wait=48 +21:44:45 step 85580 loss=3.0999 lr=9.82e-05 tok=11.217B dt=407 wait=49 +21:46:00 step 85600 loss=3.1806 lr=9.82e-05 tok=11.220B dt=482 wait=49 +21:47:15 step 85620 loss=3.0689 lr=9.81e-05 tok=11.222B dt=557 wait=50 +21:48:30 step 85640 loss=3.1193 lr=9.81e-05 tok=11.225B dt=631 wait=50 +21:49:23 pushed ckpt_w43.pt,log_w43.txt +21:50:24 waited 61s for global r=215 +21:50:43 global r=215 load 4.259 -> 4.245 +21:51:27 step 85660 loss=3.1807 lr=9.81e-05 tok=11.228B dt=809 wait=50 +21:52:43 step 85680 loss=3.1024 lr=9.81e-05 tok=11.230B dt=884 wait=51 +21:53:58 step 85700 loss=3.1605 lr=9.81e-05 tok=11.233B dt=960 wait=51 +21:55:14 step 85720 loss=3.1736 lr=9.80e-05 tok=11.235B dt=1035 wait=52 +21:56:29 step 85740 loss=3.0824 lr=9.80e-05 tok=11.238B dt=1110 wait=52 +21:57:44 step 85760 loss=3.1838 lr=9.80e-05 tok=11.241B dt=1186 wait=53 +21:59:00 step 85780 loss=3.0750 lr=9.80e-05 tok=11.243B dt=1261 wait=53 +22:00:15 step 85800 loss=3.1839 lr=9.79e-05 tok=11.246B dt=1337 wait=54 +22:00:31 pushed ckpt_w43.pt,log_w43.txt +22:01:47 waited 76s for global r=216 +22:02:10 global r=216 load 4.258 -> 4.243 +22:03:31 step 85820 loss=3.1565 lr=9.79e-05 tok=11.249B dt=1532 wait=54 +22:04:46 step 85840 loss=3.1629 lr=9.79e-05 tok=11.251B dt=1608 wait=54 +22:06:02 step 85860 loss=3.0987 lr=9.79e-05 tok=11.254B dt=1683 wait=55 +22:07:17 step 85880 loss=3.1685 lr=9.79e-05 tok=11.256B dt=1758 wait=55 +22:08:32 step 85900 loss=3.1485 lr=9.78e-05 tok=11.259B dt=1834 wait=56 +22:09:47 step 85920 loss=3.1143 lr=9.78e-05 tok=11.262B dt=1909 wait=56 +22:11:03 step 85940 loss=3.1352 lr=9.78e-05 tok=11.264B dt=1984 wait=57 +22:11:52 pushed ckpt_w43.pt,log_w43.txt +22:13:08 waited 76s for global r=217 +22:13:25 global r=217 load 4.261 -> 4.242 +22:14:07 step 85960 loss=3.1319 lr=9.78e-05 tok=11.267B dt=2169 wait=57 +22:15:23 step 85980 loss=3.1476 lr=9.78e-05 tok=11.270B dt=2244 wait=58 +22:16:38 step 86000 loss=3.1856 lr=9.77e-05 tok=11.272B dt=2320 wait=58 +22:17:54 step 86020 loss=3.1703 lr=9.77e-05 tok=11.275B dt=2395 wait=58 +22:19:09 step 86040 loss=3.1241 lr=9.77e-05 tok=11.277B dt=2471 wait=59 +22:20:24 step 86060 loss=3.1576 lr=9.77e-05 tok=11.280B dt=2546 wait=59 +22:21:40 step 86080 loss=3.0608 lr=9.77e-05 tok=11.283B dt=2621 wait=60 +22:22:55 step 86100 loss=3.1583 lr=9.76e-05 tok=11.285B dt=2696 wait=60 +22:23:11 pushed ckpt_w43.pt,log_w43.txt +22:24:13 waited 62s for global r=218 +22:24:19 global r=218 load 4.260 -> 4.238 +22:25:38 step 86120 loss=3.1367 lr=9.76e-05 tok=11.288B dt=2860 wait=61 +22:26:54 step 86140 loss=3.1669 lr=9.76e-05 tok=11.291B dt=2935 wait=61 +22:28:09 step 86160 loss=3.1447 lr=9.76e-05 tok=11.293B dt=3011 wait=62 +22:29:24 step 86180 loss=3.1817 lr=9.76e-05 tok=11.296B dt=3086 wait=62 +22:30:40 step 86200 loss=3.1419 lr=9.75e-05 tok=11.298B dt=3161 wait=62 +22:31:55 step 86220 loss=3.0856 lr=9.75e-05 tok=11.301B dt=3236 wait=63 +22:33:10 step 86240 loss=3.1153 lr=9.75e-05 tok=11.304B dt=3312 wait=63 +22:34:00 pushed ckpt_w43.pt,log_w43.txt +22:35:31 waited 91s for global r=219 +22:35:38 global r=219 load 4.257 -> 4.235 +22:36:23 step 86260 loss=3.1137 lr=9.75e-05 tok=11.306B dt=3505 wait=64 +22:37:39 step 86280 loss=3.1244 lr=9.75e-05 tok=11.309B dt=3580 wait=64 +22:38:54 step 86300 loss=3.2030 lr=9.74e-05 tok=11.312B dt=3656 wait=65 +22:40:09 step 86320 loss=3.1882 lr=9.74e-05 tok=11.314B dt=3731 wait=65 +22:41:25 step 86340 loss=3.1305 lr=9.74e-05 tok=11.317B dt=3806 wait=66 +22:42:40 step 86360 loss=3.2033 lr=9.74e-05 tok=11.319B dt=3882 wait=66 +22:43:55 step 86380 loss=3.0542 lr=9.73e-05 tok=11.322B dt=3957 wait=66 +22:45:10 step 86400 loss=3.1575 lr=9.73e-05 tok=11.325B dt=4032 wait=67 +22:45:25 pushed ckpt_w43.pt,log_w43.txt +22:46:40 waited 76s for global r=220 +22:46:45 global r=220 load 4.263 -> 4.236 +22:48:07 step 86420 loss=3.0927 lr=9.73e-05 tok=11.327B dt=4209 wait=67 +22:49:23 step 86440 loss=3.1173 lr=9.73e-05 tok=11.330B dt=4284 wait=68 +22:50:38 step 86460 loss=3.0642 lr=9.73e-05 tok=11.332B dt=4359 wait=68 +22:51:53 step 86480 loss=3.1401 lr=9.72e-05 tok=11.335B dt=4435 wait=69 +22:53:09 step 86500 loss=3.1615 lr=9.72e-05 tok=11.338B dt=4510 wait=69 +22:54:24 step 86520 loss=3.0936 lr=9.72e-05 tok=11.340B dt=4585 wait=69 +22:55:39 step 86540 loss=3.1594 lr=9.72e-05 tok=11.343B dt=4660 wait=70 +22:56:39 pushed ckpt_w43.pt,log_w43.txt +22:57:39 waited 60s for global r=221 +22:57:49 global r=221 load 4.254 -> 4.232 +22:58:33 step 86560 loss=3.0876 lr=9.72e-05 tok=11.346B dt=4835 wait=70 +22:59:48 step 86580 loss=3.0936 lr=9.71e-05 tok=11.348B dt=4910 wait=71 +23:01:04 step 86600 loss=3.1172 lr=9.71e-05 tok=11.351B dt=4985 wait=71 +23:02:19 step 86620 loss=3.1696 lr=9.71e-05 tok=11.353B dt=5061 wait=72 +23:03:34 step 86640 loss=3.1406 lr=9.71e-05 tok=11.356B dt=5136 wait=72 +23:04:50 step 86660 loss=3.2003 lr=9.71e-05 tok=11.359B dt=5211 wait=73 +23:06:05 step 86680 loss=3.1305 lr=9.70e-05 tok=11.361B dt=5287 wait=73 +23:07:20 step 86700 loss=3.1375 lr=9.70e-05 tok=11.364B dt=5362 wait=73 +23:07:35 pushed ckpt_w43.pt,log_w43.txt +23:08:53 waited 78s for global r=222 +23:09:00 global r=222 load 4.244 -> 4.228 +23:10:24 step 86720 loss=3.0923 lr=9.70e-05 tok=11.367B dt=5546 wait=74 +23:11:40 step 86740 loss=3.1109 lr=9.70e-05 tok=11.369B dt=5621 wait=74 +23:12:55 step 86760 loss=3.1011 lr=9.70e-05 tok=11.372B dt=5697 wait=75 +23:14:11 step 86780 loss=3.1770 lr=9.69e-05 tok=11.374B dt=5772 wait=75 +23:15:26 step 86800 loss=3.1874 lr=9.69e-05 tok=11.377B dt=5848 wait=76 +23:16:41 step 86820 loss=3.1238 lr=9.69e-05 tok=11.380B dt=5923 wait=76 +23:17:57 step 86840 loss=3.1065 lr=9.69e-05 tok=11.382B dt=5998 wait=77 +23:18:56 pushed ckpt_w43.pt,log_w43.txt +23:20:27 waited 91s for global r=223 +23:20:40 global r=223 load 4.243 -> 4.226 +23:21:27 step 86860 loss=3.0979 lr=9.69e-05 tok=11.385B dt=6208 wait=77 +23:22:42 step 86880 loss=3.0682 lr=9.68e-05 tok=11.388B dt=6284 wait=77 +23:23:57 step 86900 loss=3.1601 lr=9.68e-05 tok=11.390B dt=6359 wait=78 +23:25:13 step 86920 loss=3.1813 lr=9.68e-05 tok=11.393B dt=6434 wait=78 +23:26:28 step 86940 loss=3.0980 lr=9.68e-05 tok=11.395B dt=6510 wait=79 +23:27:44 step 86960 loss=3.2009 lr=9.67e-05 tok=11.398B dt=6585 wait=79 +23:28:59 step 86980 loss=3.0569 lr=9.67e-05 tok=11.401B dt=6660 wait=80 +23:30:14 step 87000 loss=3.1011 lr=9.67e-05 tok=11.403B dt=6736 wait=80 +23:30:28 pushed ckpt_w43.pt,log_w43.txt +23:31:43 waited 76s for global r=224 +23:31:56 global r=224 load 4.245 -> 4.227 +23:33:15 step 87020 loss=3.1024 lr=9.67e-05 tok=11.406B dt=6917 wait=81 +23:34:31 step 87040 loss=3.1278 lr=9.67e-05 tok=11.409B dt=6992 wait=81 +23:35:46 step 87060 loss=3.1209 lr=9.66e-05 tok=11.411B dt=7068 wait=81 +23:37:02 step 87080 loss=3.2277 lr=9.66e-05 tok=11.414B dt=7143 wait=82 +23:38:17 step 87100 loss=3.1657 lr=9.66e-05 tok=11.416B dt=7219 wait=82 +23:39:32 step 87120 loss=3.0898 lr=9.66e-05 tok=11.419B dt=7294 wait=83 +23:40:47 step 87140 loss=3.1244 lr=9.66e-05 tok=11.422B dt=7369 wait=83 +23:41:38 pushed ckpt_w43.pt,log_w43.txt +23:43:09 waited 91s for global r=225 +23:43:19 global r=225 load 4.253 -> 4.228 +23:44:00 step 87160 loss=3.0999 lr=9.65e-05 tok=11.424B dt=7562 wait=84 +23:45:16 step 87180 loss=3.1060 lr=9.65e-05 tok=11.427B dt=7637 wait=84 +23:46:31 step 87200 loss=3.1687 lr=9.65e-05 tok=11.429B dt=7713 wait=85 +23:47:47 step 87220 loss=3.2071 lr=9.65e-05 tok=11.432B dt=7788 wait=85 +23:49:02 step 87240 loss=3.1149 lr=9.65e-05 tok=11.435B dt=7864 wait=86 +23:50:17 step 87260 loss=3.2490 lr=9.64e-05 tok=11.437B dt=7939 wait=86 +23:51:33 step 87280 loss=3.0567 lr=9.64e-05 tok=11.440B dt=8014 wait=86 +23:52:48 step 87300 loss=3.0817 lr=9.64e-05 tok=11.443B dt=8089 wait=87 +23:53:00 pushed ckpt_w43.pt,log_w43.txt +23:54:32 waited 92s for global r=226 +23:54:44 global r=226 load 4.245 -> 4.225 +23:56:07 step 87320 loss=3.0959 lr=9.64e-05 tok=11.445B dt=8289 wait=87 +23:57:23 step 87340 loss=3.1395 lr=9.64e-05 tok=11.448B dt=8364 wait=88 +23:58:38 step 87360 loss=3.1459 lr=9.63e-05 tok=11.450B dt=8440 wait=88 +23:59:54 step 87380 loss=3.1986 lr=9.63e-05 tok=11.453B dt=8515 wait=89 +00:01:09 step 87400 loss=3.1526 lr=9.63e-05 tok=11.456B dt=8591 wait=89 +00:02:25 step 87420 loss=3.0822 lr=9.63e-05 tok=11.458B dt=8666 wait=90 +00:03:40 step 87440 loss=3.1007 lr=9.63e-05 tok=11.461B dt=8741 wait=90 +00:04:31 pushed ckpt_w43.pt,log_w43.txt +00:06:01 waited 91s for global r=227 +00:06:11 global r=227 load 4.252 -> 4.228 +00:07:05 step 87460 loss=3.0333 lr=9.62e-05 tok=11.464B dt=8947 wait=90 +00:08:21 step 87480 loss=3.0643 lr=9.62e-05 tok=11.466B dt=9022 wait=91 +00:09:36 step 87500 loss=3.1877 lr=9.62e-05 tok=11.469B dt=9098 wait=91 +00:10:52 step 87520 loss=3.1906 lr=9.62e-05 tok=11.471B dt=9173 wait=92 +00:12:07 step 87540 loss=3.1291 lr=9.61e-05 tok=11.474B dt=9249 wait=92 +00:13:22 step 87560 loss=3.1875 lr=9.61e-05 tok=11.477B dt=9324 wait=93 +00:14:37 step 87580 loss=3.0606 lr=9.61e-05 tok=11.479B dt=9399 wait=93 +00:15:52 step 87600 loss=3.1062 lr=9.61e-05 tok=11.482B dt=9474 wait=93 diff --git a/log_w44.txt b/log_w44.txt new file mode 100644 index 0000000000000000000000000000000000000000..08ff775ede7e149c825e4f4f4ad177e46afea661 --- /dev/null +++ b/log_w44.txt @@ -0,0 +1,327 @@ +launch +18:25:39 name=w44 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:25:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:25:55 params 1.08B +18:25:57 resumed step=85500 seen=11.21B round=202 +18:25:57 stream salt=85500 +18:27:48 start: accum=4, 131072 tokens per optimiser step +18:29:10 step 85501 loss=3.3498 lr=9.83e-05 tok=11.207B dt=82 wait=48 +18:29:15 step 85502 loss=3.1739 lr=9.83e-05 tok=11.207B dt=87 wait=48 +18:30:37 step 85520 loss=3.3012 lr=9.82e-05 tok=11.209B dt=169 wait=48 +18:31:52 step 85540 loss=3.2425 lr=9.82e-05 tok=11.212B dt=244 wait=48 +18:33:07 step 85560 loss=3.2619 lr=9.82e-05 tok=11.215B dt=319 wait=49 +launch +18:34:05 name=w44 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:34:10 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:34:19 params 1.08B +18:34:20 resumed step=85500 seen=11.21B round=202 +18:34:20 stream salt=85500 +18:36:07 start: accum=4, 131072 tokens per optimiser step +18:37:12 step 85501 loss=2.9222 lr=9.83e-05 tok=11.207B dt=66 wait=46 +18:37:17 step 85502 loss=2.7330 lr=9.83e-05 tok=11.207B dt=70 wait=46 +18:38:40 step 85520 loss=2.8931 lr=9.82e-05 tok=11.209B dt=154 wait=46 +18:39:55 step 85540 loss=2.8030 lr=9.82e-05 tok=11.212B dt=229 wait=46 +18:41:10 step 85560 loss=2.8621 lr=9.82e-05 tok=11.215B dt=304 wait=47 +18:42:25 step 85580 loss=2.7780 lr=9.82e-05 tok=11.217B dt=379 wait=47 +18:43:40 step 85600 loss=2.8008 lr=9.82e-05 tok=11.220B dt=453 wait=48 +18:44:55 step 85620 loss=2.7999 lr=9.81e-05 tok=11.222B dt=528 wait=48 +18:46:10 step 85640 loss=2.8253 lr=9.81e-05 tok=11.225B dt=603 wait=48 +18:47:01 pushed ckpt_w44.pt,log_w44.txt +18:48:17 waited 76s for global r=203 +18:48:28 global r=203 load 3.414 -> 3.391 +18:49:13 step 85660 loss=2.7944 lr=9.81e-05 tok=11.228B dt=786 wait=49 +18:50:27 step 85680 loss=2.7393 lr=9.81e-05 tok=11.230B dt=861 wait=49 +launch +18:57:48 name=w44 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:57:53 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:58:02 params 1.08B +18:58:04 resumed step=84450 seen=11.07B round=203 +18:58:04 stream salt=84450 +18:59:50 start: accum=4, 131072 tokens per optimiser step +19:00:56 step 84451 loss=2.9465 lr=9.93e-05 tok=11.069B dt=66 wait=48 +19:01:01 step 84452 loss=2.7023 lr=9.93e-05 tok=11.069B dt=72 wait=48 +19:01:40 step 84460 loss=2.8833 lr=9.93e-05 tok=11.070B dt=110 wait=49 +19:03:05 step 84480 loss=2.7314 lr=9.93e-05 tok=11.073B dt=195 wait=49 +19:04:20 step 84500 loss=2.8162 lr=9.93e-05 tok=11.076B dt=270 wait=50 +19:05:35 step 84520 loss=2.8127 lr=9.93e-05 tok=11.078B dt=345 wait=50 +19:06:50 step 84540 loss=2.8364 lr=9.92e-05 tok=11.081B dt=420 wait=50 +19:08:05 step 84560 loss=2.7047 lr=9.92e-05 tok=11.083B dt=495 wait=51 +19:09:19 step 84580 loss=2.8084 lr=9.92e-05 tok=11.086B dt=569 wait=51 +19:10:34 step 84600 loss=2.7543 lr=9.92e-05 tok=11.089B dt=644 wait=52 +19:10:48 pushed ckpt_w44.pt,log_w44.txt +19:13:19 waited 151s for global r=204 +19:13:30 global r=204 load 3.393 -> 3.381 +19:14:55 step 84620 loss=2.6820 lr=9.92e-05 tok=11.091B dt=905 wait=52 +19:16:10 step 84640 loss=2.7097 lr=9.91e-05 tok=11.094B dt=980 wait=52 +19:17:24 step 84660 loss=2.7674 lr=9.91e-05 tok=11.097B dt=1054 wait=53 +19:18:39 step 84680 loss=2.7780 lr=9.91e-05 tok=11.099B dt=1129 wait=53 +19:19:54 step 84700 loss=2.7278 lr=9.91e-05 tok=11.102B dt=1204 wait=53 +19:21:08 step 84720 loss=2.7173 lr=9.91e-05 tok=11.104B dt=1279 wait=54 +19:22:23 step 84740 loss=2.7530 lr=9.90e-05 tok=11.107B dt=1353 wait=54 +19:23:14 pushed ckpt_w44.pt,log_w44.txt +19:25:00 waited 106s for global r=205 +19:25:07 global r=205 load 3.390 -> 3.383 +19:25:49 step 84760 loss=2.7102 lr=9.90e-05 tok=11.110B dt=1559 wait=54 +19:27:04 step 84780 loss=2.7443 lr=9.90e-05 tok=11.112B dt=1634 wait=55 +19:28:19 step 84800 loss=2.8399 lr=9.90e-05 tok=11.115B dt=1709 wait=55 +19:29:34 step 84820 loss=2.8115 lr=9.90e-05 tok=11.118B dt=1784 wait=55 +19:30:49 step 84840 loss=2.7335 lr=9.89e-05 tok=11.120B dt=1859 wait=56 +19:32:04 step 84860 loss=2.7455 lr=9.89e-05 tok=11.123B dt=1934 wait=56 +19:33:18 step 84880 loss=2.7450 lr=9.89e-05 tok=11.125B dt=2008 wait=56 +19:34:33 step 84900 loss=2.7266 lr=9.89e-05 tok=11.128B dt=2083 wait=57 +launch +19:39:24 name=w44 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/3 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:39:29 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:39:38 params 1.08B +19:39:40 resumed step=84450 seen=11.07B round=206 +19:39:40 stream salt=84450 +19:41:39 start: accum=4, 131072 tokens per optimiser step +19:42:45 step 84451 loss=2.9465 lr=9.93e-05 tok=11.069B dt=66 wait=49 +19:42:50 step 84452 loss=2.7023 lr=9.93e-05 tok=11.069B dt=72 wait=49 +19:43:28 step 84460 loss=2.8833 lr=9.93e-05 tok=11.070B dt=109 wait=49 +19:44:54 step 84480 loss=2.7314 lr=9.93e-05 tok=11.073B dt=196 wait=50 +19:46:10 step 84500 loss=2.8162 lr=9.93e-05 tok=11.076B dt=271 wait=50 +19:47:24 step 84520 loss=2.8127 lr=9.93e-05 tok=11.078B dt=346 wait=50 +19:48:39 step 84540 loss=2.8364 lr=9.92e-05 tok=11.081B dt=421 wait=51 +19:49:54 step 84560 loss=2.7047 lr=9.92e-05 tok=11.083B dt=495 wait=51 +19:51:09 step 84580 loss=2.8084 lr=9.92e-05 tok=11.086B dt=570 wait=51 +19:52:23 step 84600 loss=2.7543 lr=9.92e-05 tok=11.089B dt=644 wait=52 +19:52:30 pushed ckpt_w44.pt,log_w44.txt +19:53:46 waited 76s for global r=207 +19:54:01 global r=207 load 3.393 -> 3.370 +19:55:19 step 84620 loss=2.6690 lr=9.92e-05 tok=11.091B dt=821 wait=52 +19:56:34 step 84640 loss=2.7004 lr=9.91e-05 tok=11.094B dt=895 wait=52 +19:57:49 step 84660 loss=2.7588 lr=9.91e-05 tok=11.097B dt=970 wait=53 +19:59:04 step 84680 loss=2.7698 lr=9.91e-05 tok=11.099B dt=1045 wait=53 +20:00:18 step 84700 loss=2.7210 lr=9.91e-05 tok=11.102B dt=1120 wait=54 +20:01:33 step 84720 loss=2.7099 lr=9.91e-05 tok=11.104B dt=1194 wait=54 +20:02:48 step 84740 loss=2.7458 lr=9.90e-05 tok=11.107B dt=1269 wait=54 +20:03:39 pushed ckpt_w44.pt,log_w44.txt +20:05:40 waited 121s for global r=208 +20:05:50 global r=208 load 3.382 -> 3.366 +20:06:33 step 84760 loss=2.6984 lr=9.90e-05 tok=11.110B dt=1495 wait=55 +20:07:48 step 84780 loss=2.7287 lr=9.90e-05 tok=11.112B dt=1569 wait=55 +20:09:03 step 84800 loss=2.8261 lr=9.90e-05 tok=11.115B dt=1644 wait=55 +20:10:18 step 84820 loss=2.7982 lr=9.90e-05 tok=11.118B dt=1719 wait=56 +20:11:33 step 84840 loss=2.7218 lr=9.89e-05 tok=11.120B dt=1794 wait=56 +20:12:48 step 84860 loss=2.7339 lr=9.89e-05 tok=11.123B dt=1869 wait=56 +20:14:02 step 84880 loss=2.7346 lr=9.89e-05 tok=11.125B dt=1943 wait=57 +20:15:17 step 84900 loss=2.7163 lr=9.89e-05 tok=11.128B dt=2018 wait=57 +20:15:29 pushed ckpt_w44.pt,log_w44.txt +20:16:45 waited 76s for global r=209 +20:16:56 global r=209 load 3.387 -> 3.366 +20:18:18 step 84920 loss=2.7353 lr=9.89e-05 tok=11.131B dt=2199 wait=57 +20:19:33 step 84940 loss=2.7627 lr=9.88e-05 tok=11.133B dt=2274 wait=58 +20:20:48 step 84960 loss=2.7532 lr=9.88e-05 tok=11.136B dt=2349 wait=58 +20:22:03 step 84980 loss=2.7946 lr=9.88e-05 tok=11.138B dt=2424 wait=59 +20:23:18 step 85000 loss=2.6787 lr=9.88e-05 tok=11.141B dt=2499 wait=59 +20:24:32 step 85020 loss=2.7225 lr=9.88e-05 tok=11.144B dt=2573 wait=59 +20:25:47 step 85040 loss=2.7372 lr=9.87e-05 tok=11.146B dt=2648 wait=60 +20:26:37 pushed ckpt_w44.pt,log_w44.txt +20:27:53 waited 76s for global r=210 +20:28:02 global r=210 load 3.389 -> 3.371 +20:28:43 step 85060 loss=2.7220 lr=9.87e-05 tok=11.149B dt=2824 wait=60 +20:29:58 step 85080 loss=2.7012 lr=9.87e-05 tok=11.152B dt=2899 wait=60 +20:31:13 step 85100 loss=2.7511 lr=9.87e-05 tok=11.154B dt=2974 wait=61 +20:32:28 step 85120 loss=2.7879 lr=9.87e-05 tok=11.157B dt=3049 wait=61 +20:33:42 step 85140 loss=2.7288 lr=9.86e-05 tok=11.159B dt=3123 wait=61 +20:34:57 step 85160 loss=2.7506 lr=9.86e-05 tok=11.162B dt=3198 wait=62 +20:36:12 step 85180 loss=2.6943 lr=9.86e-05 tok=11.165B dt=3273 wait=62 +20:37:27 step 85200 loss=2.7292 lr=9.86e-05 tok=11.167B dt=3348 wait=62 +20:37:40 pushed ckpt_w44.pt,log_w44.txt +20:39:26 waited 106s for global r=211 +20:39:38 global r=211 load 3.383 -> 3.367 +20:40:58 step 85220 loss=2.6923 lr=9.85e-05 tok=11.170B dt=3559 wait=63 +20:42:13 step 85240 loss=2.7453 lr=9.85e-05 tok=11.173B dt=3634 wait=63 +20:43:28 step 85260 loss=2.7688 lr=9.85e-05 tok=11.175B dt=3709 wait=64 +20:44:43 step 85280 loss=2.7935 lr=9.85e-05 tok=11.178B dt=3784 wait=64 +20:45:58 step 85300 loss=2.7248 lr=9.85e-05 tok=11.180B dt=3859 wait=64 +20:47:12 step 85320 loss=2.7070 lr=9.84e-05 tok=11.183B dt=3934 wait=65 +20:48:27 step 85340 loss=2.7278 lr=9.84e-05 tok=11.186B dt=4008 wait=65 +20:49:16 pushed ckpt_w44.pt,log_w44.txt +20:51:02 waited 106s for global r=212 +20:51:13 global r=212 load 3.388 -> 3.365 +20:51:58 step 85360 loss=2.6747 lr=9.84e-05 tok=11.188B dt=4219 wait=65 +20:53:13 step 85380 loss=2.7535 lr=9.84e-05 tok=11.191B dt=4294 wait=66 +20:54:28 step 85400 loss=2.7719 lr=9.84e-05 tok=11.194B dt=4369 wait=66 +20:55:43 step 85420 loss=2.8072 lr=9.83e-05 tok=11.196B dt=4444 wait=67 +20:56:58 step 85440 loss=2.7140 lr=9.83e-05 tok=11.199B dt=4519 wait=67 +20:58:13 step 85460 loss=2.7746 lr=9.83e-05 tok=11.201B dt=4594 wait=67 +20:59:28 step 85480 loss=2.7122 lr=9.83e-05 tok=11.204B dt=4669 wait=68 +21:00:42 step 85500 loss=2.7382 lr=9.83e-05 tok=11.207B dt=4744 wait=68 +21:00:54 pushed ckpt_w44.pt,log_w44.txt +21:02:55 waited 121s for global r=213 +21:03:05 global r=213 load 3.394 -> 3.368 +21:04:24 step 85520 loss=2.6755 lr=9.82e-05 tok=11.209B dt=4966 wait=68 +21:05:39 step 85540 loss=2.6928 lr=9.82e-05 tok=11.212B dt=5040 wait=69 +21:06:54 step 85560 loss=2.7691 lr=9.82e-05 tok=11.215B dt=5115 wait=69 +21:08:09 step 85580 loss=2.7576 lr=9.82e-05 tok=11.217B dt=5190 wait=69 +21:09:24 step 85600 loss=2.6569 lr=9.82e-05 tok=11.220B dt=5265 wait=70 +21:10:38 step 85620 loss=2.6523 lr=9.81e-05 tok=11.222B dt=5339 wait=70 +21:11:53 step 85640 loss=2.6999 lr=9.81e-05 tok=11.225B dt=5414 wait=71 +21:12:44 pushed ckpt_w44.pt,log_w44.txt +21:16:30 step 85660 loss=2.6792 lr=9.81e-05 tok=11.228B dt=5691 wait=71 +21:17:44 step 85680 loss=2.6897 lr=9.81e-05 tok=11.230B dt=5766 wait=71 +21:18:59 step 85700 loss=2.6650 lr=9.81e-05 tok=11.233B dt=5840 wait=72 +21:20:14 step 85720 loss=2.6919 lr=9.80e-05 tok=11.235B dt=5915 wait=72 +21:21:29 step 85740 loss=2.6797 lr=9.80e-05 tok=11.238B dt=5990 wait=72 +21:22:44 step 85760 loss=2.7361 lr=9.80e-05 tok=11.241B dt=6065 wait=73 +21:23:59 step 85780 loss=2.7825 lr=9.80e-05 tok=11.243B dt=6140 wait=73 +21:25:14 step 85800 loss=2.7432 lr=9.79e-05 tok=11.246B dt=6215 wait=74 +21:25:28 pushed ckpt_w44.pt,log_w44.txt +21:29:50 step 85820 loss=2.7397 lr=9.79e-05 tok=11.249B dt=6491 wait=74 +21:31:05 step 85840 loss=2.8058 lr=9.79e-05 tok=11.251B dt=6566 wait=74 +21:32:20 step 85860 loss=2.7999 lr=9.79e-05 tok=11.254B dt=6641 wait=75 +launch +21:35:28 name=w44 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:35:33 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:35:42 params 1.08B +21:35:44 resumed step=85500 seen=11.21B round=214 +21:35:44 stream salt=85500 +21:37:37 start: accum=4, 131072 tokens per optimiser step +21:38:39 step 85501 loss=3.1669 lr=9.83e-05 tok=11.207B dt=63 wait=45 +21:38:44 step 85502 loss=3.0964 lr=9.83e-05 tok=11.207B dt=67 wait=45 +21:40:07 step 85520 loss=3.2526 lr=9.82e-05 tok=11.209B dt=150 wait=45 +21:41:26 step 85540 loss=3.1972 lr=9.82e-05 tok=11.212B dt=230 wait=46 +21:42:41 step 85560 loss=3.2405 lr=9.82e-05 tok=11.215B dt=305 wait=46 +21:43:56 step 85580 loss=3.1375 lr=9.82e-05 tok=11.217B dt=380 wait=47 +21:45:11 step 85600 loss=3.2045 lr=9.82e-05 tok=11.220B dt=454 wait=47 +21:46:26 step 85620 loss=3.0963 lr=9.81e-05 tok=11.222B dt=529 wait=47 +21:47:40 step 85640 loss=3.0992 lr=9.81e-05 tok=11.225B dt=604 wait=48 +21:48:32 pushed ckpt_w44.pt,log_w44.txt +21:50:17 waited 106s for global r=215 +21:50:34 global r=215 load 4.273 -> 4.245 +21:51:16 step 85660 loss=3.1580 lr=9.81e-05 tok=11.228B dt=820 wait=48 +21:52:31 step 85680 loss=3.1198 lr=9.81e-05 tok=11.230B dt=894 wait=49 +21:53:46 step 85700 loss=3.1120 lr=9.81e-05 tok=11.233B dt=969 wait=49 +21:55:01 step 85720 loss=3.1709 lr=9.80e-05 tok=11.235B dt=1044 wait=49 +21:56:16 step 85740 loss=3.0931 lr=9.80e-05 tok=11.238B dt=1119 wait=50 +21:57:31 step 85760 loss=3.1697 lr=9.80e-05 tok=11.241B dt=1194 wait=50 +21:58:46 step 85780 loss=3.1021 lr=9.80e-05 tok=11.243B dt=1269 wait=51 +22:00:00 step 85800 loss=3.1382 lr=9.79e-05 tok=11.246B dt=1344 wait=51 +22:00:14 pushed ckpt_w44.pt,log_w44.txt +22:02:00 waited 106s for global r=216 +22:02:12 global r=216 load 4.257 -> 4.243 +22:03:34 step 85820 loss=3.0585 lr=9.79e-05 tok=11.249B dt=1557 wait=51 +22:04:48 step 85840 loss=3.1067 lr=9.79e-05 tok=11.251B dt=1632 wait=52 +22:06:03 step 85860 loss=3.1066 lr=9.79e-05 tok=11.254B dt=1707 wait=52 +22:07:18 step 85880 loss=3.1664 lr=9.79e-05 tok=11.256B dt=1781 wait=53 +22:08:33 step 85900 loss=3.2152 lr=9.78e-05 tok=11.259B dt=1856 wait=53 +22:09:48 step 85920 loss=3.1131 lr=9.78e-05 tok=11.262B dt=1931 wait=53 +22:11:03 step 85940 loss=3.1499 lr=9.78e-05 tok=11.264B dt=2006 wait=54 +22:11:54 pushed ckpt_w44.pt,log_w44.txt +22:13:10 waited 76s for global r=217 +22:13:20 global r=217 load 4.256 -> 4.242 +22:14:06 step 85960 loss=3.0868 lr=9.78e-05 tok=11.267B dt=2190 wait=54 +22:15:21 step 85980 loss=3.1076 lr=9.78e-05 tok=11.270B dt=2265 wait=55 +22:16:36 step 86000 loss=3.1580 lr=9.77e-05 tok=11.272B dt=2340 wait=55 +22:17:51 step 86020 loss=3.1743 lr=9.77e-05 tok=11.275B dt=2415 wait=55 +22:19:06 step 86040 loss=3.1465 lr=9.77e-05 tok=11.277B dt=2489 wait=56 +22:20:21 step 86060 loss=3.2063 lr=9.77e-05 tok=11.280B dt=2564 wait=56 +22:21:36 step 86080 loss=3.1000 lr=9.77e-05 tok=11.283B dt=2639 wait=57 +22:22:51 step 86100 loss=3.1225 lr=9.76e-05 tok=11.285B dt=2714 wait=57 +22:23:02 pushed ckpt_w44.pt,log_w44.txt +22:24:18 waited 76s for global r=218 +22:24:24 global r=218 load 4.253 -> 4.238 +22:25:45 step 86120 loss=3.1064 lr=9.76e-05 tok=11.288B dt=2889 wait=57 +22:27:00 step 86140 loss=3.1504 lr=9.76e-05 tok=11.291B dt=2964 wait=58 +22:28:15 step 86160 loss=3.1402 lr=9.76e-05 tok=11.293B dt=3039 wait=58 +22:29:30 step 86180 loss=3.1583 lr=9.76e-05 tok=11.296B dt=3113 wait=59 +22:30:45 step 86200 loss=3.1718 lr=9.75e-05 tok=11.298B dt=3188 wait=59 +22:32:00 step 86220 loss=3.1301 lr=9.75e-05 tok=11.301B dt=3263 wait=59 +22:33:15 step 86240 loss=3.1288 lr=9.75e-05 tok=11.304B dt=3338 wait=60 +22:34:11 pushed ckpt_w44.pt,log_w44.txt +22:35:27 waited 76s for global r=219 +22:35:34 global r=219 load 4.257 -> 4.235 +22:36:18 step 86260 loss=3.1193 lr=9.75e-05 tok=11.306B dt=3522 wait=60 +22:37:33 step 86280 loss=3.0933 lr=9.75e-05 tok=11.309B dt=3597 wait=61 +22:38:48 step 86300 loss=3.1836 lr=9.74e-05 tok=11.312B dt=3672 wait=61 +22:40:03 step 86320 loss=3.1386 lr=9.74e-05 tok=11.314B dt=3746 wait=61 +22:41:18 step 86340 loss=3.0680 lr=9.74e-05 tok=11.317B dt=3821 wait=62 +22:42:32 step 86360 loss=3.1866 lr=9.74e-05 tok=11.319B dt=3896 wait=62 +22:43:47 step 86380 loss=3.0594 lr=9.73e-05 tok=11.322B dt=3971 wait=63 +22:45:02 step 86400 loss=3.1204 lr=9.73e-05 tok=11.325B dt=4046 wait=63 +22:45:16 pushed ckpt_w44.pt,log_w44.txt +22:46:32 waited 76s for global r=220 +22:46:37 global r=220 load 4.247 -> 4.236 +22:47:55 step 86420 loss=3.0835 lr=9.73e-05 tok=11.327B dt=4219 wait=63 +22:49:10 step 86440 loss=3.1035 lr=9.73e-05 tok=11.330B dt=4294 wait=64 +22:50:25 step 86460 loss=3.1340 lr=9.73e-05 tok=11.332B dt=4369 wait=64 +22:51:40 step 86480 loss=3.1966 lr=9.72e-05 tok=11.335B dt=4444 wait=65 +22:52:55 step 86500 loss=3.2009 lr=9.72e-05 tok=11.338B dt=4519 wait=65 +22:54:10 step 86520 loss=3.1107 lr=9.72e-05 tok=11.340B dt=4594 wait=65 +22:55:25 step 86540 loss=3.1202 lr=9.72e-05 tok=11.343B dt=4668 wait=66 +22:56:19 pushed ckpt_w44.pt,log_w44.txt +22:57:35 waited 76s for global r=221 +22:57:42 global r=221 load 4.247 -> 4.232 +22:58:25 step 86560 loss=3.0646 lr=9.72e-05 tok=11.346B dt=4848 wait=66 +22:59:40 step 86580 loss=3.0755 lr=9.71e-05 tok=11.348B dt=4923 wait=67 +23:00:54 step 86600 loss=3.1456 lr=9.71e-05 tok=11.351B dt=4998 wait=67 +23:02:10 step 86620 loss=3.1926 lr=9.71e-05 tok=11.353B dt=5073 wait=68 +23:03:24 step 86640 loss=3.1212 lr=9.71e-05 tok=11.356B dt=5148 wait=68 +23:04:39 step 86660 loss=3.1954 lr=9.71e-05 tok=11.359B dt=5223 wait=68 +23:05:54 step 86680 loss=3.0720 lr=9.70e-05 tok=11.361B dt=5297 wait=69 +23:07:09 step 86700 loss=3.1315 lr=9.70e-05 tok=11.364B dt=5372 wait=69 +23:07:20 pushed ckpt_w44.pt,log_w44.txt +23:08:51 waited 91s for global r=222 +23:09:00 global r=222 load 4.250 -> 4.228 +23:10:23 step 86720 loss=3.1027 lr=9.70e-05 tok=11.367B dt=5567 wait=69 +23:11:38 step 86740 loss=3.1367 lr=9.70e-05 tok=11.369B dt=5642 wait=70 +23:12:53 step 86760 loss=3.0841 lr=9.70e-05 tok=11.372B dt=5716 wait=70 +23:14:08 step 86780 loss=3.1551 lr=9.69e-05 tok=11.374B dt=5791 wait=71 +23:15:23 step 86800 loss=3.1695 lr=9.69e-05 tok=11.377B dt=5866 wait=71 +23:16:38 step 86820 loss=3.1122 lr=9.69e-05 tok=11.380B dt=5941 wait=71 +23:17:53 step 86840 loss=3.1587 lr=9.69e-05 tok=11.382B dt=6016 wait=72 +23:18:43 pushed ckpt_w44.pt,log_w44.txt +23:20:14 waited 91s for global r=223 +23:20:25 global r=223 load 4.250 -> 4.226 +23:21:08 step 86860 loss=3.0995 lr=9.69e-05 tok=11.385B dt=6212 wait=72 +23:22:23 step 86880 loss=3.0707 lr=9.68e-05 tok=11.388B dt=6287 wait=73 +23:23:38 step 86900 loss=3.1072 lr=9.68e-05 tok=11.390B dt=6361 wait=73 +23:24:53 step 86920 loss=3.1634 lr=9.68e-05 tok=11.393B dt=6436 wait=73 +23:26:08 step 86940 loss=3.1107 lr=9.68e-05 tok=11.395B dt=6511 wait=74 +23:27:22 step 86960 loss=3.1911 lr=9.67e-05 tok=11.398B dt=6586 wait=74 +23:28:37 step 86980 loss=3.0839 lr=9.67e-05 tok=11.401B dt=6661 wait=75 +23:29:52 step 87000 loss=3.1178 lr=9.67e-05 tok=11.403B dt=6736 wait=75 +23:30:07 pushed ckpt_w44.pt,log_w44.txt +23:31:52 waited 106s for global r=224 +23:32:02 global r=224 load 4.246 -> 4.227 +23:33:21 step 87020 loss=3.1178 lr=9.67e-05 tok=11.406B dt=6945 wait=75 +23:34:36 step 87040 loss=3.1515 lr=9.67e-05 tok=11.409B dt=7020 wait=76 +23:35:51 step 87060 loss=3.1172 lr=9.66e-05 tok=11.411B dt=7095 wait=76 +23:37:06 step 87080 loss=3.1524 lr=9.66e-05 tok=11.414B dt=7169 wait=77 +23:38:21 step 87100 loss=3.1381 lr=9.66e-05 tok=11.416B dt=7245 wait=77 +23:39:36 step 87120 loss=3.1103 lr=9.66e-05 tok=11.419B dt=7320 wait=77 +23:40:51 step 87140 loss=3.1453 lr=9.66e-05 tok=11.422B dt=7394 wait=78 +23:41:40 pushed ckpt_w44.pt,log_w44.txt +23:43:11 waited 91s for global r=225 +23:43:19 global r=225 load 4.245 -> 4.228 +23:44:01 step 87160 loss=3.1079 lr=9.65e-05 tok=11.424B dt=7584 wait=78 +23:45:15 step 87180 loss=3.0735 lr=9.65e-05 tok=11.427B dt=7659 wait=79 +23:46:30 step 87200 loss=3.1613 lr=9.65e-05 tok=11.429B dt=7733 wait=79 +23:47:45 step 87220 loss=3.2023 lr=9.65e-05 tok=11.432B dt=7809 wait=79 +23:49:00 step 87240 loss=3.1262 lr=9.65e-05 tok=11.435B dt=7883 wait=80 +23:50:15 step 87260 loss=3.1799 lr=9.64e-05 tok=11.437B dt=7958 wait=80 +23:51:30 step 87280 loss=3.0140 lr=9.64e-05 tok=11.440B dt=8033 wait=81 +23:52:45 step 87300 loss=3.1348 lr=9.64e-05 tok=11.443B dt=8108 wait=81 +23:52:58 pushed ckpt_w44.pt,log_w44.txt +23:54:29 waited 91s for global r=226 +23:54:43 global r=226 load 4.247 -> 4.225 +23:56:06 step 87320 loss=3.1015 lr=9.64e-05 tok=11.445B dt=8310 wait=81 +23:57:21 step 87340 loss=3.1291 lr=9.64e-05 tok=11.448B dt=8384 wait=82 +23:58:36 step 87360 loss=3.0995 lr=9.63e-05 tok=11.450B dt=8459 wait=82 +23:59:51 step 87380 loss=3.1706 lr=9.63e-05 tok=11.453B dt=8534 wait=83 +00:01:06 step 87400 loss=3.1947 lr=9.63e-05 tok=11.456B dt=8609 wait=83 +00:02:21 step 87420 loss=3.1277 lr=9.63e-05 tok=11.458B dt=8684 wait=83 +00:03:36 step 87440 loss=3.1635 lr=9.63e-05 tok=11.461B dt=8759 wait=84 +00:04:28 pushed ckpt_w44.pt,log_w44.txt +00:05:58 waited 91s for global r=227 +00:06:12 global r=227 load 4.248 -> 4.228 +00:06:53 step 87460 loss=3.1005 lr=9.62e-05 tok=11.464B dt=8956 wait=84 +00:08:08 step 87480 loss=3.1023 lr=9.62e-05 tok=11.466B dt=9031 wait=85 +00:09:23 step 87500 loss=3.1754 lr=9.62e-05 tok=11.469B dt=9106 wait=85 +00:10:38 step 87520 loss=3.1720 lr=9.62e-05 tok=11.471B dt=9181 wait=86 +00:11:53 step 87540 loss=3.1037 lr=9.61e-05 tok=11.474B dt=9256 wait=86 +00:13:07 step 87560 loss=3.2045 lr=9.61e-05 tok=11.477B dt=9331 wait=86 +00:14:22 step 87580 loss=3.1040 lr=9.61e-05 tok=11.479B dt=9406 wait=87 +00:15:37 step 87600 loss=3.1601 lr=9.61e-05 tok=11.482B dt=9481 wait=87 diff --git a/log_w45.txt b/log_w45.txt new file mode 100644 index 0000000000000000000000000000000000000000..4bc201e99f52cd6ecf0921d8b6ea654c6f2e16b3 --- /dev/null +++ b/log_w45.txt @@ -0,0 +1,153 @@ +launch +21:35:44 name=w45 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 3/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:35:50 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:35:59 params 1.08B +21:36:01 resumed step=85500 seen=11.21B round=214 +21:36:01 stream salt=85500 +21:37:55 start: accum=4, 131072 tokens per optimiser step +21:39:20 step 85501 loss=3.0505 lr=9.83e-05 tok=11.207B dt=85 wait=53 +21:39:25 step 85502 loss=2.9604 lr=9.83e-05 tok=11.207B dt=91 wait=53 +21:40:53 step 85520 loss=3.2322 lr=9.82e-05 tok=11.209B dt=178 wait=53 +21:42:08 step 85540 loss=3.2091 lr=9.82e-05 tok=11.212B dt=253 wait=54 +21:43:23 step 85560 loss=3.2737 lr=9.82e-05 tok=11.215B dt=329 wait=54 +21:44:38 step 85580 loss=3.0918 lr=9.82e-05 tok=11.217B dt=404 wait=54 +21:45:53 step 85600 loss=3.1723 lr=9.82e-05 tok=11.220B dt=479 wait=55 +21:47:08 step 85620 loss=3.0841 lr=9.81e-05 tok=11.222B dt=554 wait=55 +21:48:23 step 85640 loss=3.1171 lr=9.81e-05 tok=11.225B dt=628 wait=56 +21:49:14 pushed ckpt_w45.pt,log_w45.txt +21:50:15 waited 61s for global r=215 +21:50:30 global r=215 load 4.264 -> 4.245 +21:51:13 step 85660 loss=3.1777 lr=9.81e-05 tok=11.228B dt=798 wait=56 +21:52:28 step 85680 loss=3.0892 lr=9.81e-05 tok=11.230B dt=873 wait=56 +21:53:43 step 85700 loss=3.1269 lr=9.81e-05 tok=11.233B dt=948 wait=57 +21:54:58 step 85720 loss=3.1866 lr=9.80e-05 tok=11.235B dt=1023 wait=57 +21:56:13 step 85740 loss=3.1056 lr=9.80e-05 tok=11.238B dt=1098 wait=58 +21:57:28 step 85760 loss=3.2059 lr=9.80e-05 tok=11.241B dt=1174 wait=58 +21:58:43 step 85780 loss=3.0784 lr=9.80e-05 tok=11.243B dt=1249 wait=58 +21:59:59 step 85800 loss=3.1720 lr=9.79e-05 tok=11.246B dt=1324 wait=59 +22:00:12 pushed ckpt_w45.pt,log_w45.txt +22:01:57 waited 106s for global r=216 +22:02:10 global r=216 load 4.262 -> 4.243 +22:03:30 step 85820 loss=3.1261 lr=9.79e-05 tok=11.249B dt=1535 wait=59 +22:04:45 step 85840 loss=3.1059 lr=9.79e-05 tok=11.251B dt=1610 wait=59 +22:06:00 step 85860 loss=3.1232 lr=9.79e-05 tok=11.254B dt=1685 wait=60 +22:07:15 step 85880 loss=3.1701 lr=9.79e-05 tok=11.256B dt=1760 wait=60 +22:08:30 step 85900 loss=3.1679 lr=9.78e-05 tok=11.259B dt=1835 wait=61 +22:09:45 step 85920 loss=3.1167 lr=9.78e-05 tok=11.262B dt=1910 wait=61 +22:11:00 step 85940 loss=3.1250 lr=9.78e-05 tok=11.264B dt=1985 wait=61 +22:11:49 pushed ckpt_w45.pt,log_w45.txt +22:13:05 waited 76s for global r=217 +22:13:15 global r=217 load 4.261 -> 4.242 +22:14:01 step 85960 loss=3.0844 lr=9.78e-05 tok=11.267B dt=2166 wait=62 +22:15:16 step 85980 loss=3.0923 lr=9.78e-05 tok=11.270B dt=2241 wait=62 +22:16:31 step 86000 loss=3.1563 lr=9.77e-05 tok=11.272B dt=2316 wait=63 +22:17:46 step 86020 loss=3.1838 lr=9.77e-05 tok=11.275B dt=2391 wait=63 +22:19:01 step 86040 loss=3.1099 lr=9.77e-05 tok=11.277B dt=2466 wait=63 +22:20:16 step 86060 loss=3.1747 lr=9.77e-05 tok=11.280B dt=2541 wait=64 +22:21:31 step 86080 loss=3.0598 lr=9.77e-05 tok=11.283B dt=2616 wait=64 +22:22:46 step 86100 loss=3.1236 lr=9.76e-05 tok=11.285B dt=2691 wait=64 +22:23:00 pushed ckpt_w45.pt,log_w45.txt +22:24:16 waited 76s for global r=218 +22:24:23 global r=218 load 4.266 -> 4.238 +22:25:46 step 86120 loss=3.1041 lr=9.76e-05 tok=11.288B dt=2871 wait=65 +22:27:01 step 86140 loss=3.1181 lr=9.76e-05 tok=11.291B dt=2946 wait=65 +22:28:16 step 86160 loss=3.1440 lr=9.76e-05 tok=11.293B dt=3021 wait=65 +22:29:31 step 86180 loss=3.1864 lr=9.76e-05 tok=11.296B dt=3097 wait=66 +22:30:47 step 86200 loss=3.1601 lr=9.75e-05 tok=11.298B dt=3172 wait=66 +22:32:01 step 86220 loss=3.0993 lr=9.75e-05 tok=11.301B dt=3247 wait=67 +22:33:16 step 86240 loss=3.1146 lr=9.75e-05 tok=11.304B dt=3322 wait=67 +22:34:08 pushed ckpt_w45.pt,log_w45.txt +22:35:23 waited 76s for global r=219 +22:35:30 global r=219 load 4.250 -> 4.235 +22:36:11 step 86260 loss=3.0827 lr=9.75e-05 tok=11.306B dt=3496 wait=67 +22:37:26 step 86280 loss=3.1112 lr=9.75e-05 tok=11.309B dt=3571 wait=68 +22:38:41 step 86300 loss=3.1513 lr=9.74e-05 tok=11.312B dt=3646 wait=68 +22:39:56 step 86320 loss=3.2054 lr=9.74e-05 tok=11.314B dt=3721 wait=68 +22:41:11 step 86340 loss=3.1233 lr=9.74e-05 tok=11.317B dt=3796 wait=69 +22:42:25 step 86360 loss=3.2145 lr=9.74e-05 tok=11.319B dt=3871 wait=69 +22:43:40 step 86380 loss=3.0457 lr=9.73e-05 tok=11.322B dt=3945 wait=70 +22:44:55 step 86400 loss=3.1208 lr=9.73e-05 tok=11.325B dt=4020 wait=70 +22:45:08 pushed ckpt_w45.pt,log_w45.txt +22:46:39 waited 91s for global r=220 +22:46:44 global r=220 load 4.250 -> 4.236 +22:48:04 step 86420 loss=3.0906 lr=9.73e-05 tok=11.327B dt=4209 wait=70 +22:49:19 step 86440 loss=3.1211 lr=9.73e-05 tok=11.330B dt=4284 wait=71 +22:50:34 step 86460 loss=3.0913 lr=9.73e-05 tok=11.332B dt=4359 wait=71 +22:51:49 step 86480 loss=3.1637 lr=9.72e-05 tok=11.335B dt=4434 wait=71 +22:53:04 step 86500 loss=3.1363 lr=9.72e-05 tok=11.338B dt=4509 wait=72 +22:54:19 step 86520 loss=3.0854 lr=9.72e-05 tok=11.340B dt=4584 wait=72 +22:55:34 step 86540 loss=3.1523 lr=9.72e-05 tok=11.343B dt=4659 wait=73 +22:56:27 pushed ckpt_w45.pt,log_w45.txt +22:57:43 waited 76s for global r=221 +22:57:51 global r=221 load 4.254 -> 4.232 +22:58:32 step 86560 loss=3.1261 lr=9.72e-05 tok=11.346B dt=4837 wait=73 +22:59:47 step 86580 loss=3.1446 lr=9.71e-05 tok=11.348B dt=4912 wait=73 +23:01:02 step 86600 loss=3.2097 lr=9.71e-05 tok=11.351B dt=4988 wait=74 +23:02:18 step 86620 loss=3.2469 lr=9.71e-05 tok=11.353B dt=5063 wait=74 +23:03:33 step 86640 loss=3.1191 lr=9.71e-05 tok=11.356B dt=5138 wait=74 +23:04:47 step 86660 loss=3.1732 lr=9.71e-05 tok=11.359B dt=5212 wait=75 +23:06:02 step 86680 loss=3.0659 lr=9.70e-05 tok=11.361B dt=5287 wait=75 +23:07:17 step 86700 loss=3.1322 lr=9.70e-05 tok=11.364B dt=5362 wait=76 +23:07:29 pushed ckpt_w45.pt,log_w45.txt +23:08:45 waited 76s for global r=222 +23:08:55 global r=222 load 4.249 -> 4.228 +23:10:17 step 86720 loss=3.0905 lr=9.70e-05 tok=11.367B dt=5542 wait=76 +23:11:32 step 86740 loss=3.1485 lr=9.70e-05 tok=11.369B dt=5617 wait=76 +23:12:47 step 86760 loss=3.1029 lr=9.70e-05 tok=11.372B dt=5692 wait=77 +23:14:02 step 86780 loss=3.1525 lr=9.69e-05 tok=11.374B dt=5767 wait=77 +23:15:17 step 86800 loss=3.1137 lr=9.69e-05 tok=11.377B dt=5842 wait=77 +23:16:31 step 86820 loss=3.0840 lr=9.69e-05 tok=11.380B dt=5917 wait=78 +23:17:46 step 86840 loss=3.1314 lr=9.69e-05 tok=11.382B dt=5991 wait=78 +23:18:36 pushed ckpt_w45.pt,log_w45.txt +23:20:22 waited 106s for global r=223 +23:20:33 global r=223 load 4.245 -> 4.226 +23:21:15 step 86860 loss=3.1231 lr=9.69e-05 tok=11.385B dt=6200 wait=79 +23:22:30 step 86880 loss=3.1399 lr=9.68e-05 tok=11.388B dt=6275 wait=79 +23:23:45 step 86900 loss=3.1422 lr=9.68e-05 tok=11.390B dt=6350 wait=79 +23:25:00 step 86920 loss=3.1876 lr=9.68e-05 tok=11.393B dt=6425 wait=80 +23:26:15 step 86940 loss=3.1008 lr=9.68e-05 tok=11.395B dt=6500 wait=80 +23:27:30 step 86960 loss=3.1789 lr=9.67e-05 tok=11.398B dt=6575 wait=80 +23:28:45 step 86980 loss=3.0362 lr=9.67e-05 tok=11.401B dt=6650 wait=81 +23:30:00 step 87000 loss=3.0937 lr=9.67e-05 tok=11.403B dt=6725 wait=81 +23:30:12 pushed ckpt_w45.pt,log_w45.txt +23:31:43 waited 91s for global r=224 +23:31:53 global r=224 load 4.244 -> 4.227 +23:33:16 step 87020 loss=3.0633 lr=9.67e-05 tok=11.406B dt=6921 wait=82 +23:34:31 step 87040 loss=3.1102 lr=9.67e-05 tok=11.409B dt=6996 wait=82 +23:35:46 step 87060 loss=3.1303 lr=9.66e-05 tok=11.411B dt=7071 wait=82 +23:37:01 step 87080 loss=3.2101 lr=9.66e-05 tok=11.414B dt=7146 wait=83 +23:38:16 step 87100 loss=3.1480 lr=9.66e-05 tok=11.416B dt=7222 wait=83 +23:39:32 step 87120 loss=3.1274 lr=9.66e-05 tok=11.419B dt=7297 wait=83 +23:40:47 step 87140 loss=3.1482 lr=9.66e-05 tok=11.422B dt=7372 wait=84 +23:41:38 pushed ckpt_w45.pt,log_w45.txt +23:43:09 waited 91s for global r=225 +23:43:19 global r=225 load 4.248 -> 4.228 +23:44:04 step 87160 loss=3.0842 lr=9.65e-05 tok=11.424B dt=7569 wait=84 +23:45:19 step 87180 loss=3.0962 lr=9.65e-05 tok=11.427B dt=7644 wait=85 +23:46:34 step 87200 loss=3.1365 lr=9.65e-05 tok=11.429B dt=7719 wait=85 +23:47:49 step 87220 loss=3.1570 lr=9.65e-05 tok=11.432B dt=7794 wait=85 +23:49:04 step 87240 loss=3.1319 lr=9.65e-05 tok=11.435B dt=7869 wait=86 +23:50:19 step 87260 loss=3.2482 lr=9.64e-05 tok=11.437B dt=7944 wait=86 +23:51:34 step 87280 loss=3.1089 lr=9.64e-05 tok=11.440B dt=8019 wait=86 +23:52:49 step 87300 loss=3.1219 lr=9.64e-05 tok=11.443B dt=8094 wait=87 +23:53:01 pushed ckpt_w45.pt,log_w45.txt +23:54:32 waited 91s for global r=226 +23:54:46 global r=226 load 4.241 -> 4.225 +23:56:09 step 87320 loss=3.0419 lr=9.64e-05 tok=11.445B dt=8294 wait=87 +23:57:24 step 87340 loss=3.0815 lr=9.64e-05 tok=11.448B dt=8369 wait=88 +23:58:39 step 87360 loss=3.0904 lr=9.63e-05 tok=11.450B dt=8444 wait=88 +23:59:54 step 87380 loss=3.1801 lr=9.63e-05 tok=11.453B dt=8519 wait=88 +00:01:09 step 87400 loss=3.1608 lr=9.63e-05 tok=11.456B dt=8594 wait=89 +00:02:24 step 87420 loss=3.1343 lr=9.63e-05 tok=11.458B dt=8670 wait=89 +00:03:40 step 87440 loss=3.1462 lr=9.63e-05 tok=11.461B dt=8745 wait=90 +00:04:29 pushed ckpt_w45.pt,log_w45.txt +00:06:00 waited 91s for global r=227 +00:06:18 global r=227 load 4.238 -> 4.228 +00:07:08 step 87460 loss=3.1112 lr=9.62e-05 tok=11.464B dt=8953 wait=90 +00:08:23 step 87480 loss=3.0975 lr=9.62e-05 tok=11.466B dt=9028 wait=90 +00:09:38 step 87500 loss=3.1273 lr=9.62e-05 tok=11.469B dt=9103 wait=91 +00:10:53 step 87520 loss=3.1466 lr=9.62e-05 tok=11.471B dt=9178 wait=91 +00:12:07 step 87540 loss=3.0854 lr=9.61e-05 tok=11.474B dt=9253 wait=91 +00:13:22 step 87560 loss=3.2163 lr=9.61e-05 tok=11.477B dt=9328 wait=92 +00:14:37 step 87580 loss=3.0760 lr=9.61e-05 tok=11.479B dt=9403 wait=92 +00:15:53 step 87600 loss=3.1756 lr=9.61e-05 tok=11.482B dt=9478 wait=93 diff --git a/log_w5.txt b/log_w5.txt new file mode 100644 index 0000000000000000000000000000000000000000..8ebfd673b88e3bb8133113529571839ad9280be3 --- /dev/null +++ b/log_w5.txt @@ -0,0 +1,38 @@ +launch +18:40:41 name=w5 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:40:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:40:56 params 1.08B +18:40:57 resumed step=28800 seen=3.77B round=14 +18:40:57 stream salt=28800 +18:43:12 start: accum=4, 131072 tokens per optimiser step +18:46:03 step 28820 loss=3.7794 lr=2.87e-04 tok=3.777B dt=171 +18:47:37 step 28840 loss=3.9305 lr=2.87e-04 tok=3.780B dt=265 +18:48:53 step 28860 loss=3.8295 lr=2.87e-04 tok=3.783B dt=340 +18:50:09 step 28880 loss=3.8898 lr=2.87e-04 tok=3.785B dt=416 +18:51:25 step 28900 loss=3.7435 lr=2.87e-04 tok=3.788B dt=492 +18:52:41 step 28920 loss=3.7931 lr=2.87e-04 tok=3.791B dt=568 +18:53:57 step 28940 loss=3.8192 lr=2.87e-04 tok=3.793B dt=644 +18:55:13 step 28960 loss=3.8208 lr=2.87e-04 tok=3.796B dt=720 +18:56:29 step 28980 loss=3.7537 lr=2.87e-04 tok=3.798B dt=796 +18:57:45 step 29000 loss=3.8416 lr=2.87e-04 tok=3.801B dt=872 +18:59:01 step 29020 loss=3.7883 lr=2.87e-04 tok=3.804B dt=948 +19:00:16 step 29040 loss=3.7320 lr=2.87e-04 tok=3.806B dt=1024 +19:01:32 step 29060 loss=3.7562 lr=2.87e-04 tok=3.809B dt=1100 +19:02:48 step 29080 loss=3.7514 lr=2.87e-04 tok=3.812B dt=1176 +19:04:04 step 29100 loss=3.8191 lr=2.87e-04 tok=3.814B dt=1252 +19:05:20 step 29120 loss=3.8805 lr=2.87e-04 tok=3.817B dt=1328 +19:06:36 step 29140 loss=3.8489 lr=2.87e-04 tok=3.819B dt=1404 +19:07:52 step 29160 loss=3.8081 lr=2.87e-04 tok=3.822B dt=1480 +19:09:08 step 29180 loss=3.8947 lr=2.87e-04 tok=3.825B dt=1556 +19:10:25 step 29200 loss=3.8082 lr=2.87e-04 tok=3.827B dt=1632 +19:11:41 step 29220 loss=3.7903 lr=2.86e-04 tok=3.830B dt=1708 +19:12:57 step 29240 loss=3.7119 lr=2.86e-04 tok=3.833B dt=1784 +19:14:13 step 29260 loss=3.7798 lr=2.86e-04 tok=3.835B dt=1860 +19:15:29 step 29280 loss=3.7023 lr=2.86e-04 tok=3.838B dt=1936 +19:16:44 step 29300 loss=3.8437 lr=2.86e-04 tok=3.840B dt=2012 +19:18:00 step 29320 loss=3.7657 lr=2.86e-04 tok=3.843B dt=2088 +19:19:16 step 29340 loss=3.7144 lr=2.86e-04 tok=3.846B dt=2164 +19:20:32 step 29360 loss=3.8139 lr=2.86e-04 tok=3.848B dt=2240 +19:21:48 step 29380 loss=3.6732 lr=2.86e-04 tok=3.851B dt=2316 +19:23:04 step 29400 loss=3.8126 lr=2.86e-04 tok=3.854B dt=2392 +19:23:17 pushed ckpt_w5.pt diff --git a/log_w51.txt b/log_w51.txt new file mode 100644 index 0000000000000000000000000000000000000000..aa2b0c08528e22d6830e07d99210562dc8dbf850 --- /dev/null +++ b/log_w51.txt @@ -0,0 +1,289 @@ +launch +16:02:40 name=w51 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:02:47 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:02:57 params 1.08B +16:02:59 resumed step=87600 seen=11.48B round=229 +16:02:59 stream salt=87600 +16:05:34 start: accum=4, 131072 tokens per optimiser step +16:07:54 step 87601 loss=3.1526 lr=9.61e-05 tok=11.482B dt=140 wait=105 +16:07:59 step 87602 loss=3.0413 lr=9.61e-05 tok=11.482B dt=145 wait=106 +16:09:33 step 87620 loss=3.2104 lr=9.61e-05 tok=11.485B dt=239 wait=106 +16:11:06 step 87640 loss=3.1767 lr=9.60e-05 tok=11.487B dt=333 wait=107 +16:12:22 step 87660 loss=3.2190 lr=9.60e-05 tok=11.490B dt=408 wait=107 +16:13:37 step 87680 loss=3.1134 lr=9.60e-05 tok=11.492B dt=483 wait=107 +16:14:52 step 87700 loss=3.1626 lr=9.60e-05 tok=11.495B dt=558 wait=108 +16:16:07 step 87720 loss=3.0630 lr=9.60e-05 tok=11.498B dt=633 wait=108 +16:17:22 step 87740 loss=3.1016 lr=9.59e-05 tok=11.500B dt=708 wait=109 +16:18:19 pushed ckpt_w51.pt,log_w51.txt +16:19:35 waited 76s for global r=230 +16:19:54 global r=230 load 4.254 -> 4.231 +16:20:36 step 87760 loss=3.1766 lr=9.59e-05 tok=11.503B dt=902 wait=109 +16:21:52 step 87780 loss=3.0846 lr=9.59e-05 tok=11.506B dt=978 wait=109 +16:23:07 step 87800 loss=3.1601 lr=9.59e-05 tok=11.508B dt=1054 wait=110 +16:24:23 step 87820 loss=3.1898 lr=9.59e-05 tok=11.511B dt=1129 wait=110 +16:25:39 step 87840 loss=3.1090 lr=9.58e-05 tok=11.513B dt=1205 wait=111 +16:26:54 step 87860 loss=3.2010 lr=9.58e-05 tok=11.516B dt=1280 wait=111 +16:28:09 step 87880 loss=3.0699 lr=9.58e-05 tok=11.519B dt=1356 wait=112 +16:29:25 step 87900 loss=3.1229 lr=9.58e-05 tok=11.521B dt=1431 wait=112 +16:29:38 pushed ckpt_w51.pt,log_w51.txt +16:31:10 waited 93s for global r=231 +16:31:24 global r=231 load 4.256 -> 4.232 +16:32:46 step 87920 loss=3.1013 lr=9.58e-05 tok=11.524B dt=1632 wait=113 +16:34:02 step 87940 loss=3.1245 lr=9.57e-05 tok=11.526B dt=1708 wait=113 +16:35:17 step 87960 loss=3.0819 lr=9.57e-05 tok=11.529B dt=1783 wait=114 +16:36:33 step 87980 loss=3.1319 lr=9.57e-05 tok=11.532B dt=1859 wait=114 +16:37:49 step 88000 loss=3.1521 lr=9.57e-05 tok=11.534B dt=1935 wait=115 +16:39:04 step 88020 loss=3.1393 lr=9.56e-05 tok=11.537B dt=2010 wait=115 +16:40:19 step 88040 loss=3.1567 lr=9.56e-05 tok=11.540B dt=2086 wait=115 +16:41:09 pushed ckpt_w51.pt,log_w51.txt +16:44:11 waited 182s for global r=232 +16:44:22 global r=232 load 4.246 -> 4.227 +16:45:04 step 88060 loss=3.0855 lr=9.56e-05 tok=11.542B dt=2370 wait=116 +16:46:19 step 88080 loss=3.0622 lr=9.56e-05 tok=11.545B dt=2445 wait=116 +16:47:34 step 88100 loss=3.1021 lr=9.56e-05 tok=11.547B dt=2521 wait=117 +16:48:50 step 88120 loss=3.1642 lr=9.55e-05 tok=11.550B dt=2596 wait=117 +16:50:06 step 88140 loss=3.0998 lr=9.55e-05 tok=11.553B dt=2672 wait=118 +16:51:21 step 88160 loss=3.1818 lr=9.55e-05 tok=11.555B dt=2747 wait=118 +16:52:36 step 88180 loss=3.0913 lr=9.55e-05 tok=11.558B dt=2823 wait=119 +16:53:51 step 88200 loss=3.1838 lr=9.55e-05 tok=11.561B dt=2898 wait=119 +16:54:06 pushed ckpt_w51.pt,log_w51.txt +16:58:28 step 88220 loss=3.1980 lr=9.54e-05 tok=11.563B dt=3174 wait=119 +16:59:43 step 88240 loss=3.0926 lr=9.54e-05 tok=11.566B dt=3250 wait=120 +17:00:59 step 88260 loss=3.0340 lr=9.54e-05 tok=11.568B dt=3325 wait=120 +17:02:14 step 88280 loss=3.0459 lr=9.54e-05 tok=11.571B dt=3401 wait=121 +17:03:30 step 88300 loss=3.0287 lr=9.54e-05 tok=11.574B dt=3476 wait=121 +17:04:45 step 88320 loss=3.1558 lr=9.53e-05 tok=11.576B dt=3552 wait=122 +17:06:01 step 88340 loss=3.1319 lr=9.53e-05 tok=11.579B dt=3627 wait=122 +17:06:50 pushed ckpt_w51.pt,log_w51.txt +17:06:50 waited 0s for global r=233 +17:07:02 global r=233 load 4.256 -> 4.228 +17:07:48 step 88360 loss=3.1109 lr=9.53e-05 tok=11.582B dt=3735 wait=123 +17:09:04 step 88380 loss=3.0669 lr=9.53e-05 tok=11.584B dt=3810 wait=123 +17:10:19 step 88400 loss=3.1224 lr=9.53e-05 tok=11.587B dt=3885 wait=123 +17:11:34 step 88420 loss=3.1427 lr=9.52e-05 tok=11.589B dt=3960 wait=124 +17:12:49 step 88440 loss=3.0950 lr=9.52e-05 tok=11.592B dt=4036 wait=124 +17:14:04 step 88460 loss=3.1981 lr=9.52e-05 tok=11.595B dt=4111 wait=125 +17:15:20 step 88480 loss=3.1074 lr=9.52e-05 tok=11.597B dt=4186 wait=125 +17:16:35 step 88500 loss=3.1292 lr=9.51e-05 tok=11.600B dt=4261 wait=125 +17:16:48 pushed ckpt_w51.pt,log_w51.txt +17:18:19 waited 91s for global r=234 +17:18:28 global r=234 load 4.245 -> 4.230 +17:19:49 step 88520 loss=3.0786 lr=9.51e-05 tok=11.602B dt=4455 wait=126 +17:21:05 step 88540 loss=3.0993 lr=9.51e-05 tok=11.605B dt=4531 wait=126 +17:22:20 step 88560 loss=3.1115 lr=9.51e-05 tok=11.608B dt=4606 wait=127 +17:23:35 step 88580 loss=3.1673 lr=9.51e-05 tok=11.610B dt=4682 wait=127 +17:24:51 step 88600 loss=3.1249 lr=9.50e-05 tok=11.613B dt=4757 wait=128 +17:26:07 step 88620 loss=3.1004 lr=9.50e-05 tok=11.616B dt=4833 wait=128 +17:27:22 step 88640 loss=3.1308 lr=9.50e-05 tok=11.618B dt=4908 wait=129 +17:28:13 pushed ckpt_w51.pt,log_w51.txt +17:31:57 step 88660 loss=3.1127 lr=9.50e-05 tok=11.621B dt=5183 wait=129 +17:33:12 step 88680 loss=3.1733 lr=9.50e-05 tok=11.623B dt=5258 wait=129 +17:34:28 step 88700 loss=3.1050 lr=9.49e-05 tok=11.626B dt=5334 wait=130 +17:35:44 step 88720 loss=3.0284 lr=9.49e-05 tok=11.629B dt=5410 wait=130 +17:36:59 step 88740 loss=3.0527 lr=9.49e-05 tok=11.631B dt=5486 wait=131 +17:38:15 step 88760 loss=3.0831 lr=9.49e-05 tok=11.634B dt=5561 wait=131 +17:39:30 step 88780 loss=3.1282 lr=9.49e-05 tok=11.637B dt=5636 wait=132 +17:40:45 step 88800 loss=3.0960 lr=9.48e-05 tok=11.639B dt=5711 wait=132 +17:40:57 pushed ckpt_w51.pt,log_w51.txt +17:40:58 waited 0s for global r=235 +17:41:11 global r=235 load 4.257 -> 4.231 +17:42:30 step 88820 loss=3.1116 lr=9.48e-05 tok=11.642B dt=5816 wait=132 +17:43:45 step 88840 loss=3.1488 lr=9.48e-05 tok=11.644B dt=5891 wait=133 +17:45:00 step 88860 loss=3.1517 lr=9.48e-05 tok=11.647B dt=5967 wait=133 +17:46:15 step 88880 loss=3.1644 lr=9.48e-05 tok=11.650B dt=6041 wait=133 +17:47:30 step 88900 loss=3.1209 lr=9.47e-05 tok=11.652B dt=6117 wait=134 +17:48:45 step 88920 loss=3.0872 lr=9.47e-05 tok=11.655B dt=6191 wait=134 +17:50:00 step 88940 loss=3.1532 lr=9.47e-05 tok=11.658B dt=6267 wait=135 +17:50:53 pushed ckpt_w51.pt,log_w51.txt +17:52:09 waited 76s for global r=236 +17:52:28 global r=236 load 4.244 -> 4.225 +17:53:12 step 88960 loss=3.0931 lr=9.47e-05 tok=11.660B dt=6459 wait=135 +17:54:28 step 88980 loss=3.1153 lr=9.46e-05 tok=11.663B dt=6534 wait=135 +17:55:43 step 89000 loss=3.1688 lr=9.46e-05 tok=11.665B dt=6610 wait=136 +17:56:59 step 89020 loss=3.1685 lr=9.46e-05 tok=11.668B dt=6685 wait=136 +17:58:14 step 89040 loss=3.0824 lr=9.46e-05 tok=11.671B dt=6761 wait=137 +17:59:30 step 89060 loss=3.1724 lr=9.46e-05 tok=11.673B dt=6836 wait=137 +18:00:45 step 89080 loss=3.0300 lr=9.45e-05 tok=11.676B dt=6911 wait=138 +18:02:00 step 89100 loss=3.1427 lr=9.45e-05 tok=11.679B dt=6986 wait=138 +18:02:14 pushed ckpt_w51.pt,log_w51.txt +18:06:37 step 89120 loss=3.1706 lr=9.45e-05 tok=11.681B dt=7263 wait=139 +18:07:52 step 89140 loss=3.0202 lr=9.45e-05 tok=11.684B dt=7338 wait=139 +18:09:07 step 89160 loss=3.0865 lr=9.45e-05 tok=11.686B dt=7413 wait=139 +18:10:22 step 89180 loss=3.0903 lr=9.44e-05 tok=11.689B dt=7488 wait=140 +18:11:37 step 89200 loss=3.0390 lr=9.44e-05 tok=11.692B dt=7563 wait=140 +18:12:52 step 89220 loss=3.1720 lr=9.44e-05 tok=11.694B dt=7638 wait=140 +18:14:08 step 89240 loss=3.1219 lr=9.44e-05 tok=11.697B dt=7714 wait=141 +18:15:03 pushed ckpt_w51.pt,log_w51.txt +18:15:03 waited 0s for global r=237 +18:15:13 global r=237 load 4.253 -> 4.232 +18:15:58 step 89260 loss=3.0985 lr=9.44e-05 tok=11.699B dt=7824 wait=141 +18:17:13 step 89280 loss=3.1106 lr=9.43e-05 tok=11.702B dt=7900 wait=142 +18:18:29 step 89300 loss=3.1539 lr=9.43e-05 tok=11.705B dt=7975 wait=142 +18:19:45 step 89320 loss=3.1941 lr=9.43e-05 tok=11.707B dt=8051 wait=143 +18:21:00 step 89340 loss=3.1000 lr=9.43e-05 tok=11.710B dt=8127 wait=143 +18:22:15 step 89360 loss=3.1948 lr=9.43e-05 tok=11.713B dt=8202 wait=144 +18:23:30 step 89380 loss=3.0253 lr=9.42e-05 tok=11.715B dt=8277 wait=144 +18:24:45 step 89400 loss=3.0854 lr=9.42e-05 tok=11.718B dt=8352 wait=144 +18:24:57 pushed ckpt_w51.pt,log_w51.txt +18:26:28 waited 91s for global r=238 +18:26:38 global r=238 load 4.253 -> 4.231 +18:28:01 step 89420 loss=3.0570 lr=9.42e-05 tok=11.720B dt=8547 wait=145 +18:29:16 step 89440 loss=3.0892 lr=9.42e-05 tok=11.723B dt=8623 wait=145 +18:30:32 step 89460 loss=3.1140 lr=9.41e-05 tok=11.726B dt=8698 wait=146 +18:31:48 step 89480 loss=3.1542 lr=9.41e-05 tok=11.728B dt=8774 wait=146 +18:33:03 step 89500 loss=3.1473 lr=9.41e-05 tok=11.731B dt=8849 wait=147 +18:34:18 step 89520 loss=3.0982 lr=9.41e-05 tok=11.734B dt=8925 wait=147 +18:35:33 step 89540 loss=3.1487 lr=9.41e-05 tok=11.736B dt=9000 wait=148 +18:36:25 pushed ckpt_w51.pt,log_w51.txt +18:40:10 step 89560 loss=3.1276 lr=9.40e-05 tok=11.739B dt=9276 wait=148 +18:41:25 step 89580 loss=3.1376 lr=9.40e-05 tok=11.741B dt=9352 wait=148 +18:42:41 step 89600 loss=3.0717 lr=9.40e-05 tok=11.744B dt=9427 wait=149 +18:43:56 step 89620 loss=3.0644 lr=9.40e-05 tok=11.747B dt=9502 wait=149 +18:45:11 step 89640 loss=3.1076 lr=9.40e-05 tok=11.749B dt=9578 wait=150 +18:46:27 step 89660 loss=3.1146 lr=9.39e-05 tok=11.752B dt=9653 wait=150 +18:47:42 step 89680 loss=3.1826 lr=9.39e-05 tok=11.755B dt=9729 wait=151 +18:48:58 step 89700 loss=3.1222 lr=9.39e-05 tok=11.757B dt=9804 wait=151 +18:49:12 pushed ckpt_w51.pt,log_w51.txt +18:49:12 waited 0s for global r=239 +18:49:21 global r=239 load 4.254 -> 4.230 +18:50:45 step 89720 loss=3.0689 lr=9.39e-05 tok=11.760B dt=9911 wait=152 +18:52:00 step 89740 loss=3.1133 lr=9.39e-05 tok=11.762B dt=9987 wait=152 +18:53:16 step 89760 loss=3.1076 lr=9.38e-05 tok=11.765B dt=10062 wait=152 +18:54:31 step 89780 loss=3.1530 lr=9.38e-05 tok=11.768B dt=10137 wait=153 +18:55:46 step 89800 loss=3.1371 lr=9.38e-05 tok=11.770B dt=10212 wait=153 +18:57:01 step 89820 loss=3.0967 lr=9.38e-05 tok=11.773B dt=10287 wait=154 +18:58:16 step 89840 loss=3.1722 lr=9.38e-05 tok=11.776B dt=10363 wait=154 +18:59:09 pushed ckpt_w51.pt,log_w51.txt +19:00:27 waited 79s for global r=240 +19:00:36 global r=240 load 4.240 -> 4.227 +19:01:18 step 89860 loss=3.1420 lr=9.37e-05 tok=11.778B dt=10545 wait=154 +19:02:34 step 89880 loss=3.1049 lr=9.37e-05 tok=11.781B dt=10620 wait=155 +19:03:49 step 89900 loss=3.1031 lr=9.37e-05 tok=11.783B dt=10695 wait=155 +19:05:05 step 89920 loss=3.1287 lr=9.37e-05 tok=11.786B dt=10771 wait=156 +19:06:20 step 89940 loss=3.0597 lr=9.36e-05 tok=11.789B dt=10847 wait=156 +19:07:36 step 89960 loss=3.1592 lr=9.36e-05 tok=11.791B dt=10922 wait=157 +19:08:51 step 89980 loss=3.0760 lr=9.36e-05 tok=11.794B dt=10998 wait=157 +19:10:07 step 90000 loss=3.1529 lr=9.36e-05 tok=11.796B dt=11073 wait=158 +19:10:24 pushed ckpt_w51.pt,log_w51.txt +19:14:49 step 90020 loss=3.1475 lr=9.36e-05 tok=11.799B dt=11355 wait=158 +19:16:05 step 90040 loss=3.1240 lr=9.35e-05 tok=11.802B dt=11431 wait=159 +19:17:20 step 90060 loss=3.0653 lr=9.35e-05 tok=11.804B dt=11507 wait=159 +19:18:36 step 90080 loss=3.0709 lr=9.35e-05 tok=11.807B dt=11582 wait=159 +19:19:50 step 90100 loss=3.0322 lr=9.35e-05 tok=11.810B dt=11657 wait=160 +19:21:06 step 90120 loss=3.1548 lr=9.35e-05 tok=11.812B dt=11732 wait=160 +19:22:21 step 90140 loss=3.0994 lr=9.34e-05 tok=11.815B dt=11808 wait=161 +19:23:13 pushed ckpt_w51.pt,log_w51.txt +19:23:13 waited 0s for global r=241 +19:23:30 global r=241 load 4.237 -> 4.224 +19:24:16 step 90160 loss=3.1149 lr=9.34e-05 tok=11.817B dt=11922 wait=161 +19:25:31 step 90180 loss=3.1028 lr=9.34e-05 tok=11.820B dt=11998 wait=162 +19:26:47 step 90200 loss=3.1702 lr=9.34e-05 tok=11.823B dt=12073 wait=162 +19:28:02 step 90220 loss=3.1847 lr=9.34e-05 tok=11.825B dt=12149 wait=163 +19:29:18 step 90240 loss=3.0643 lr=9.33e-05 tok=11.828B dt=12224 wait=163 +19:30:33 step 90260 loss=3.1919 lr=9.33e-05 tok=11.831B dt=12299 wait=164 +19:31:49 step 90280 loss=3.0806 lr=9.33e-05 tok=11.833B dt=12375 wait=164 +19:33:05 step 90300 loss=3.1072 lr=9.33e-05 tok=11.836B dt=12451 wait=165 +19:33:25 pushed ckpt_w51.pt,log_w51.txt +19:34:41 waited 76s for global r=242 +19:34:49 global r=242 load 4.236 -> 4.224 +19:36:14 step 90320 loss=3.0809 lr=9.32e-05 tok=11.838B dt=12641 wait=165 +19:37:30 step 90340 loss=3.1112 lr=9.32e-05 tok=11.841B dt=12716 wait=165 +19:38:45 step 90360 loss=3.1404 lr=9.32e-05 tok=11.844B dt=12792 wait=166 +19:40:01 step 90380 loss=3.1753 lr=9.32e-05 tok=11.846B dt=12867 wait=166 +19:41:17 step 90400 loss=3.1290 lr=9.32e-05 tok=11.849B dt=12943 wait=167 +19:42:32 step 90420 loss=3.0886 lr=9.31e-05 tok=11.852B dt=13018 wait=167 +19:43:48 step 90440 loss=3.1621 lr=9.31e-05 tok=11.854B dt=13094 wait=168 +19:44:39 pushed ckpt_w51.pt,log_w51.txt +19:48:22 step 90460 loss=3.1252 lr=9.31e-05 tok=11.857B dt=13369 wait=168 +19:49:40 step 90480 loss=3.1004 lr=9.31e-05 tok=11.859B dt=13446 wait=169 +19:50:55 step 90500 loss=3.0306 lr=9.31e-05 tok=11.862B dt=13521 wait=169 +19:52:10 step 90520 loss=3.0046 lr=9.30e-05 tok=11.865B dt=13596 wait=170 +19:53:25 step 90540 loss=3.0933 lr=9.30e-05 tok=11.867B dt=13671 wait=170 +19:54:40 step 90560 loss=3.1262 lr=9.30e-05 tok=11.870B dt=13746 wait=171 +19:55:56 step 90580 loss=3.1757 lr=9.30e-05 tok=11.873B dt=13822 wait=171 +19:57:11 step 90600 loss=3.0634 lr=9.30e-05 tok=11.875B dt=13897 wait=171 +19:57:26 pushed ckpt_w51.pt,log_w51.txt +19:57:26 waited 0s for global r=243 +19:57:36 global r=243 load 4.257 -> 4.225 +19:59:02 step 90620 loss=3.0402 lr=9.29e-05 tok=11.878B dt=14008 wait=172 +20:00:17 step 90640 loss=3.1048 lr=9.29e-05 tok=11.880B dt=14083 wait=172 +20:01:33 step 90660 loss=3.1176 lr=9.29e-05 tok=11.883B dt=14159 wait=173 +launch +20:01:57 name=w51 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:02:04 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:02:16 params 1.08B +20:02:23 resumed step=90600 seen=11.88B round=243 +20:02:23 stream salt=90600 +20:04:30 start: accum=4, 131072 tokens per optimiser step +20:05:51 step 90601 loss=2.8761 lr=9.30e-05 tok=11.875B dt=81 wait=60 +20:05:57 step 90602 loss=3.0622 lr=9.30e-05 tok=11.875B dt=87 wait=60 +20:07:38 step 90620 loss=3.2143 lr=9.29e-05 tok=11.878B dt=188 wait=61 +20:09:01 step 90640 loss=3.3609 lr=9.29e-05 tok=11.880B dt=271 wait=61 +20:10:17 step 90660 loss=3.3241 lr=9.29e-05 tok=11.883B dt=347 wait=62 +20:11:32 step 90680 loss=3.3799 lr=9.29e-05 tok=11.886B dt=422 wait=62 +20:12:48 step 90700 loss=3.2269 lr=9.29e-05 tok=11.888B dt=498 wait=62 +20:14:03 step 90720 loss=3.2980 lr=9.28e-05 tok=11.891B dt=573 wait=63 +20:15:19 step 90740 loss=3.2957 lr=9.28e-05 tok=11.893B dt=649 wait=63 +20:16:15 pushed ckpt_w51.pt,log_w51.txt +20:17:01 waited 46s for global r=244 +20:17:17 global r=244 load 4.255 -> 4.228 +20:18:00 step 90760 loss=3.2889 lr=9.28e-05 tok=11.896B dt=810 wait=64 +20:19:16 step 90780 loss=3.2096 lr=9.28e-05 tok=11.899B dt=886 wait=64 +20:20:32 step 90800 loss=3.2711 lr=9.27e-05 tok=11.901B dt=962 wait=64 +20:21:47 step 90820 loss=3.2360 lr=9.27e-05 tok=11.904B dt=1037 wait=65 +20:23:03 step 90840 loss=3.3902 lr=9.27e-05 tok=11.907B dt=1113 wait=65 +20:24:19 step 90860 loss=3.3558 lr=9.27e-05 tok=11.909B dt=1189 wait=66 +20:25:35 step 90880 loss=3.3677 lr=9.27e-05 tok=11.912B dt=1265 wait=66 +20:26:51 step 90900 loss=3.3225 lr=9.26e-05 tok=11.914B dt=1341 wait=67 +20:27:06 pushed ckpt_w51.pt,log_w51.txt +20:28:52 waited 106s for global r=245 +20:29:07 global r=245 load 4.241 -> 4.225 +20:30:29 step 90920 loss=3.3175 lr=9.26e-05 tok=11.917B dt=1559 wait=67 +20:31:45 step 90940 loss=3.2593 lr=9.26e-05 tok=11.920B dt=1635 wait=68 +20:33:00 step 90960 loss=3.3098 lr=9.26e-05 tok=11.922B dt=1710 wait=68 +20:34:16 step 90980 loss=3.2876 lr=9.26e-05 tok=11.925B dt=1786 wait=69 +20:35:32 step 91000 loss=3.3888 lr=9.25e-05 tok=11.928B dt=1862 wait=69 +20:36:48 step 91020 loss=3.3520 lr=9.25e-05 tok=11.930B dt=1938 wait=70 +20:38:04 step 91040 loss=3.2875 lr=9.25e-05 tok=11.933B dt=2014 wait=70 +20:39:00 pushed ckpt_w51.pt,log_w51.txt +20:41:16 waited 136s for global r=246 +20:41:51 global r=246 load 4.240 -> 4.220 +20:42:35 step 91060 loss=3.3722 lr=9.25e-05 tok=11.935B dt=2285 wait=71 +20:43:51 step 91080 loss=3.2080 lr=9.25e-05 tok=11.938B dt=2361 wait=71 +20:45:07 step 91100 loss=3.2449 lr=9.24e-05 tok=11.941B dt=2437 wait=72 +20:46:23 step 91120 loss=3.2290 lr=9.24e-05 tok=11.943B dt=2513 wait=72 +20:47:39 step 91140 loss=3.3428 lr=9.24e-05 tok=11.946B dt=2589 wait=73 +20:48:55 step 91160 loss=3.3773 lr=9.24e-05 tok=11.949B dt=2665 wait=73 +20:50:11 step 91180 loss=3.4024 lr=9.23e-05 tok=11.951B dt=2741 wait=74 +20:51:27 step 91200 loss=3.3427 lr=9.23e-05 tok=11.954B dt=2817 wait=74 +20:51:39 pushed ckpt_w51.pt,log_w51.txt +20:53:56 waited 138s for global r=247 +20:54:14 global r=247 load 4.238 -> 4.224 +20:55:35 step 91220 loss=3.2955 lr=9.23e-05 tok=11.956B dt=3065 wait=74 +20:56:51 step 91240 loss=3.2090 lr=9.23e-05 tok=11.959B dt=3141 wait=75 +20:58:06 step 91260 loss=3.2235 lr=9.23e-05 tok=11.962B dt=3216 wait=75 +20:59:22 step 91280 loss=3.2809 lr=9.22e-05 tok=11.964B dt=3292 wait=76 +21:00:38 step 91300 loss=3.3818 lr=9.22e-05 tok=11.967B dt=3368 wait=76 +21:01:54 step 91320 loss=3.3309 lr=9.22e-05 tok=11.969B dt=3444 wait=77 +21:03:10 step 91340 loss=3.2959 lr=9.22e-05 tok=11.972B dt=3520 wait=77 +21:04:04 pushed ckpt_w51.pt,log_w51.txt +21:05:35 waited 92s for global r=248 +21:05:49 global r=248 load 4.247 -> 4.227 +21:06:37 step 91360 loss=3.3688 lr=9.22e-05 tok=11.975B dt=3727 wait=78 +21:07:53 step 91380 loss=3.3045 lr=9.21e-05 tok=11.977B dt=3803 wait=78 +21:09:08 step 91400 loss=3.3142 lr=9.21e-05 tok=11.980B dt=3878 wait=79 +21:10:24 step 91420 loss=3.2625 lr=9.21e-05 tok=11.983B dt=3954 wait=79 +21:11:40 step 91440 loss=3.3118 lr=9.21e-05 tok=11.985B dt=4030 wait=80 +21:12:56 step 91460 loss=3.3202 lr=9.21e-05 tok=11.988B dt=4106 wait=80 +21:14:11 step 91480 loss=3.3601 lr=9.20e-05 tok=11.990B dt=4181 wait=81 +21:15:27 step 91500 loss=3.3074 lr=9.20e-05 tok=11.993B dt=4257 wait=81 +21:15:46 pushed ckpt_w51.pt,log_w51.txt +21:17:32 waited 106s for global r=249 +21:17:43 global r=249 load 4.238 -> 4.226 +21:19:06 step 91520 loss=3.2828 lr=9.20e-05 tok=11.996B dt=4476 wait=82 +21:20:22 step 91540 loss=3.2140 lr=9.20e-05 tok=11.998B dt=4552 wait=82 +21:21:37 step 91560 loss=3.2435 lr=9.19e-05 tok=12.001B dt=4627 wait=83 +21:22:53 step 91580 loss=3.3018 lr=9.19e-05 tok=12.004B dt=4703 wait=83 +21:24:09 step 91600 loss=3.3698 lr=9.19e-05 tok=12.006B dt=4779 wait=84 +21:25:25 step 91620 loss=3.3363 lr=9.19e-05 tok=12.009B dt=4854 wait=84 +21:26:40 step 91640 loss=3.2936 lr=9.19e-05 tok=12.011B dt=4930 wait=84 diff --git a/log_w52.txt b/log_w52.txt new file mode 100644 index 0000000000000000000000000000000000000000..d72dfeee65dfd4a53fc2b2c214892ab6db67973e --- /dev/null +++ b/log_w52.txt @@ -0,0 +1,473 @@ +launch +16:02:47 name=w52 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:02:53 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:03:03 params 1.08B +16:03:04 resumed step=87600 seen=11.48B round=229 +16:03:04 stream salt=87600 +16:05:39 start: accum=4, 131072 tokens per optimiser step +16:07:50 step 87601 loss=3.1857 lr=9.61e-05 tok=11.482B dt=131 wait=102 +16:07:55 step 87602 loss=3.0491 lr=9.61e-05 tok=11.482B dt=135 wait=102 +16:09:19 step 87620 loss=3.2162 lr=9.61e-05 tok=11.485B dt=219 wait=102 +16:10:45 step 87640 loss=3.2023 lr=9.60e-05 tok=11.487B dt=305 wait=103 +16:12:00 step 87660 loss=3.2273 lr=9.60e-05 tok=11.490B dt=381 wait=103 +16:13:16 step 87680 loss=3.1103 lr=9.60e-05 tok=11.492B dt=456 wait=104 +16:14:31 step 87700 loss=3.1410 lr=9.60e-05 tok=11.495B dt=532 wait=104 +16:15:47 step 87720 loss=3.0817 lr=9.60e-05 tok=11.498B dt=607 wait=105 +16:17:02 step 87740 loss=3.0976 lr=9.59e-05 tok=11.500B dt=683 wait=105 +16:17:55 pushed ckpt_w52.pt,log_w52.txt +16:19:41 waited 106s for global r=230 +16:19:59 global r=230 load 4.251 -> 4.231 +16:20:40 step 87760 loss=3.1612 lr=9.59e-05 tok=11.503B dt=900 wait=105 +16:21:55 step 87780 loss=3.0489 lr=9.59e-05 tok=11.506B dt=976 wait=106 +16:23:11 step 87800 loss=3.1403 lr=9.59e-05 tok=11.508B dt=1051 wait=106 +16:24:26 step 87820 loss=3.1746 lr=9.59e-05 tok=11.511B dt=1127 wait=107 +16:25:42 step 87840 loss=3.1173 lr=9.58e-05 tok=11.513B dt=1202 wait=107 +16:26:57 step 87860 loss=3.2291 lr=9.58e-05 tok=11.516B dt=1278 wait=107 +16:28:13 step 87880 loss=3.0637 lr=9.58e-05 tok=11.519B dt=1354 wait=108 +16:29:29 step 87900 loss=3.1511 lr=9.58e-05 tok=11.521B dt=1430 wait=108 +16:29:41 pushed ckpt_w52.pt,log_w52.txt +16:31:12 waited 91s for global r=231 +16:31:24 global r=231 load 4.249 -> 4.232 +16:32:48 step 87920 loss=3.1286 lr=9.58e-05 tok=11.524B dt=1628 wait=109 +16:34:03 step 87940 loss=3.1412 lr=9.57e-05 tok=11.526B dt=1704 wait=109 +16:35:19 step 87960 loss=3.1132 lr=9.57e-05 tok=11.529B dt=1779 wait=110 +16:36:34 step 87980 loss=3.1113 lr=9.57e-05 tok=11.532B dt=1855 wait=110 +16:37:49 step 88000 loss=3.1108 lr=9.57e-05 tok=11.534B dt=1930 wait=110 +16:39:05 step 88020 loss=3.1011 lr=9.56e-05 tok=11.537B dt=2006 wait=111 +16:40:21 step 88040 loss=3.1491 lr=9.56e-05 tok=11.540B dt=2081 wait=111 +16:41:10 pushed ckpt_w52.pt,log_w52.txt +16:44:12 waited 181s for global r=232 +16:44:23 global r=232 load 4.248 -> 4.227 +16:45:06 step 88060 loss=3.1221 lr=9.56e-05 tok=11.542B dt=2367 wait=112 +16:46:22 step 88080 loss=3.0840 lr=9.56e-05 tok=11.545B dt=2442 wait=112 +16:47:37 step 88100 loss=3.1460 lr=9.56e-05 tok=11.547B dt=2518 wait=112 +16:48:53 step 88120 loss=3.1403 lr=9.55e-05 tok=11.550B dt=2593 wait=113 +16:50:08 step 88140 loss=3.0572 lr=9.55e-05 tok=11.553B dt=2669 wait=113 +16:51:24 step 88160 loss=3.1905 lr=9.55e-05 tok=11.555B dt=2744 wait=114 +16:52:39 step 88180 loss=3.0974 lr=9.55e-05 tok=11.558B dt=2820 wait=114 +16:53:55 step 88200 loss=3.1700 lr=9.55e-05 tok=11.561B dt=2896 wait=114 +16:54:08 pushed ckpt_w52.pt,log_w52.txt +16:58:32 step 88220 loss=3.1994 lr=9.54e-05 tok=11.563B dt=3172 wait=115 +16:59:47 step 88240 loss=3.0854 lr=9.54e-05 tok=11.566B dt=3248 wait=115 +17:01:02 step 88260 loss=3.1011 lr=9.54e-05 tok=11.568B dt=3323 wait=116 +17:02:18 step 88280 loss=3.0471 lr=9.54e-05 tok=11.571B dt=3398 wait=116 +17:03:33 step 88300 loss=3.0522 lr=9.54e-05 tok=11.574B dt=3474 wait=116 +17:04:48 step 88320 loss=3.1505 lr=9.53e-05 tok=11.576B dt=3549 wait=117 +17:06:04 step 88340 loss=3.1675 lr=9.53e-05 tok=11.579B dt=3625 wait=117 +17:06:54 pushed ckpt_w52.pt,log_w52.txt +17:06:54 waited 0s for global r=233 +17:07:02 global r=233 load 4.244 -> 4.228 +17:07:50 step 88360 loss=3.1543 lr=9.53e-05 tok=11.582B dt=3730 wait=118 +17:09:06 step 88380 loss=3.1554 lr=9.53e-05 tok=11.584B dt=3806 wait=118 +17:10:21 step 88400 loss=3.1298 lr=9.53e-05 tok=11.587B dt=3882 wait=119 +17:11:37 step 88420 loss=3.1661 lr=9.52e-05 tok=11.589B dt=3957 wait=119 +17:12:52 step 88440 loss=3.0556 lr=9.52e-05 tok=11.592B dt=4032 wait=119 +17:14:07 step 88460 loss=3.1718 lr=9.52e-05 tok=11.595B dt=4108 wait=120 +17:15:23 step 88480 loss=3.0489 lr=9.52e-05 tok=11.597B dt=4183 wait=120 +17:16:38 step 88500 loss=3.1309 lr=9.51e-05 tok=11.600B dt=4259 wait=121 +17:16:50 pushed ckpt_w52.pt,log_w52.txt +17:18:21 waited 91s for global r=234 +17:18:29 global r=234 load 4.248 -> 4.230 +17:19:51 step 88520 loss=3.0913 lr=9.51e-05 tok=11.602B dt=4452 wait=121 +17:21:07 step 88540 loss=3.1324 lr=9.51e-05 tok=11.605B dt=4527 wait=121 +17:22:23 step 88560 loss=3.1368 lr=9.51e-05 tok=11.608B dt=4603 wait=122 +17:23:38 step 88580 loss=3.1737 lr=9.51e-05 tok=11.610B dt=4679 wait=122 +17:24:53 step 88600 loss=3.1319 lr=9.50e-05 tok=11.613B dt=4754 wait=123 +17:26:09 step 88620 loss=3.0898 lr=9.50e-05 tok=11.616B dt=4829 wait=123 +17:27:24 step 88640 loss=3.1159 lr=9.50e-05 tok=11.618B dt=4905 wait=123 +17:28:14 pushed ckpt_w52.pt,log_w52.txt +17:32:00 step 88660 loss=3.0932 lr=9.50e-05 tok=11.621B dt=5181 wait=124 +17:33:16 step 88680 loss=3.1814 lr=9.50e-05 tok=11.623B dt=5257 wait=124 +17:34:31 step 88700 loss=3.0928 lr=9.49e-05 tok=11.626B dt=5332 wait=125 +17:35:47 step 88720 loss=3.0584 lr=9.49e-05 tok=11.629B dt=5407 wait=125 +17:37:02 step 88740 loss=3.0863 lr=9.49e-05 tok=11.631B dt=5483 wait=126 +17:38:17 step 88760 loss=3.0909 lr=9.49e-05 tok=11.634B dt=5558 wait=126 +17:39:33 step 88780 loss=3.0995 lr=9.49e-05 tok=11.637B dt=5633 wait=126 +17:40:48 step 88800 loss=3.1339 lr=9.48e-05 tok=11.639B dt=5709 wait=127 +17:41:01 pushed ckpt_w52.pt,log_w52.txt +17:41:02 waited 0s for global r=235 +17:41:14 global r=235 load 4.251 -> 4.231 +17:42:35 step 88820 loss=3.1191 lr=9.48e-05 tok=11.642B dt=5815 wait=127 +17:43:50 step 88840 loss=3.1223 lr=9.48e-05 tok=11.644B dt=5891 wait=128 +17:45:06 step 88860 loss=3.1062 lr=9.48e-05 tok=11.647B dt=5967 wait=128 +17:46:21 step 88880 loss=3.1478 lr=9.48e-05 tok=11.650B dt=6042 wait=128 +17:47:37 step 88900 loss=3.1561 lr=9.47e-05 tok=11.652B dt=6117 wait=129 +17:48:52 step 88920 loss=3.1291 lr=9.47e-05 tok=11.655B dt=6193 wait=129 +17:50:08 step 88940 loss=3.1413 lr=9.47e-05 tok=11.658B dt=6269 wait=130 +17:50:59 pushed ckpt_w52.pt,log_w52.txt +17:52:15 waited 76s for global r=236 +17:52:23 global r=236 load 4.246 -> 4.225 +17:53:07 step 88960 loss=3.1213 lr=9.47e-05 tok=11.660B dt=6448 wait=130 +17:54:23 step 88980 loss=3.0930 lr=9.46e-05 tok=11.663B dt=6523 wait=130 +17:55:38 step 89000 loss=3.1801 lr=9.46e-05 tok=11.665B dt=6599 wait=131 +17:56:54 step 89020 loss=3.1803 lr=9.46e-05 tok=11.668B dt=6675 wait=131 +17:58:10 step 89040 loss=3.1059 lr=9.46e-05 tok=11.671B dt=6750 wait=132 +17:59:25 step 89060 loss=3.1730 lr=9.46e-05 tok=11.673B dt=6826 wait=132 +18:00:40 step 89080 loss=3.0638 lr=9.45e-05 tok=11.676B dt=6901 wait=133 +18:01:56 step 89100 loss=3.1251 lr=9.45e-05 tok=11.679B dt=6977 wait=133 +18:02:09 pushed ckpt_w52.pt,log_w52.txt +18:06:33 step 89120 loss=3.1629 lr=9.45e-05 tok=11.681B dt=7254 wait=133 +18:07:49 step 89140 loss=3.0795 lr=9.45e-05 tok=11.684B dt=7329 wait=134 +18:09:04 step 89160 loss=3.0805 lr=9.45e-05 tok=11.686B dt=7405 wait=134 +18:10:19 step 89180 loss=3.0796 lr=9.44e-05 tok=11.689B dt=7480 wait=135 +18:11:35 step 89200 loss=3.0507 lr=9.44e-05 tok=11.692B dt=7555 wait=135 +18:12:50 step 89220 loss=3.1528 lr=9.44e-05 tok=11.694B dt=7631 wait=135 +18:14:06 step 89240 loss=3.1018 lr=9.44e-05 tok=11.697B dt=7706 wait=136 +18:14:57 pushed ckpt_w52.pt,log_w52.txt +18:14:57 waited 0s for global r=237 +18:15:05 global r=237 load 4.259 -> 4.232 +18:15:51 step 89260 loss=3.0935 lr=9.44e-05 tok=11.699B dt=7812 wait=136 +18:17:07 step 89280 loss=3.0853 lr=9.43e-05 tok=11.702B dt=7888 wait=137 +18:18:22 step 89300 loss=3.1440 lr=9.43e-05 tok=11.705B dt=7963 wait=137 +18:19:38 step 89320 loss=3.1840 lr=9.43e-05 tok=11.707B dt=8039 wait=137 +18:20:53 step 89340 loss=3.1105 lr=9.43e-05 tok=11.710B dt=8114 wait=138 +18:22:09 step 89360 loss=3.1439 lr=9.43e-05 tok=11.713B dt=8189 wait=138 +18:23:24 step 89380 loss=3.0632 lr=9.42e-05 tok=11.715B dt=8265 wait=139 +18:24:40 step 89400 loss=3.1029 lr=9.42e-05 tok=11.718B dt=8340 wait=139 +18:24:52 pushed ckpt_w52.pt,log_w52.txt +18:26:22 waited 91s for global r=238 +18:26:30 global r=238 load 4.249 -> 4.231 +18:27:52 step 89420 loss=3.0665 lr=9.42e-05 tok=11.720B dt=8532 wait=139 +18:29:07 step 89440 loss=3.1126 lr=9.42e-05 tok=11.723B dt=8608 wait=140 +18:30:23 step 89460 loss=3.0946 lr=9.41e-05 tok=11.726B dt=8683 wait=140 +18:31:38 step 89480 loss=3.1937 lr=9.41e-05 tok=11.728B dt=8759 wait=141 +18:32:54 step 89500 loss=3.1311 lr=9.41e-05 tok=11.731B dt=8835 wait=141 +18:34:09 step 89520 loss=3.0683 lr=9.41e-05 tok=11.734B dt=8910 wait=141 +18:35:25 step 89540 loss=3.1155 lr=9.41e-05 tok=11.736B dt=8985 wait=142 +18:36:18 pushed ckpt_w52.pt,log_w52.txt +18:40:04 step 89560 loss=3.0811 lr=9.40e-05 tok=11.739B dt=9265 wait=142 +18:41:20 step 89580 loss=3.1305 lr=9.40e-05 tok=11.741B dt=9341 wait=143 +18:42:35 step 89600 loss=3.1015 lr=9.40e-05 tok=11.744B dt=9416 wait=143 +18:43:51 step 89620 loss=3.0570 lr=9.40e-05 tok=11.747B dt=9492 wait=143 +18:45:06 step 89640 loss=3.0920 lr=9.40e-05 tok=11.749B dt=9567 wait=144 +18:46:22 step 89660 loss=3.1164 lr=9.39e-05 tok=11.752B dt=9642 wait=144 +18:47:37 step 89680 loss=3.1500 lr=9.39e-05 tok=11.755B dt=9718 wait=145 +18:48:53 step 89700 loss=3.1451 lr=9.39e-05 tok=11.757B dt=9793 wait=145 +18:49:05 pushed ckpt_w52.pt,log_w52.txt +18:49:05 waited 0s for global r=239 +18:49:13 global r=239 load 4.258 -> 4.230 +18:50:33 step 89720 loss=3.0735 lr=9.39e-05 tok=11.760B dt=9894 wait=146 +18:51:49 step 89740 loss=3.0941 lr=9.39e-05 tok=11.762B dt=9969 wait=146 +18:53:04 step 89760 loss=3.0943 lr=9.38e-05 tok=11.765B dt=10045 wait=146 +18:54:19 step 89780 loss=3.1241 lr=9.38e-05 tok=11.768B dt=10120 wait=147 +18:55:35 step 89800 loss=3.1397 lr=9.38e-05 tok=11.770B dt=10196 wait=147 +18:56:50 step 89820 loss=3.1214 lr=9.38e-05 tok=11.773B dt=10271 wait=148 +18:58:06 step 89840 loss=3.1875 lr=9.38e-05 tok=11.776B dt=10347 wait=148 +18:58:56 pushed ckpt_w52.pt,log_w52.txt +19:00:26 waited 91s for global r=240 +19:00:35 global r=240 load 4.244 -> 4.227 +19:01:18 step 89860 loss=3.1181 lr=9.37e-05 tok=11.778B dt=10539 wait=148 +19:02:33 step 89880 loss=3.0965 lr=9.37e-05 tok=11.781B dt=10614 wait=149 +19:03:49 step 89900 loss=3.1333 lr=9.37e-05 tok=11.783B dt=10690 wait=149 +19:05:04 step 89920 loss=3.1354 lr=9.37e-05 tok=11.786B dt=10765 wait=150 +19:06:20 step 89940 loss=3.0798 lr=9.36e-05 tok=11.789B dt=10840 wait=150 +19:07:35 step 89960 loss=3.1466 lr=9.36e-05 tok=11.791B dt=10916 wait=150 +19:08:51 step 89980 loss=3.0735 lr=9.36e-05 tok=11.794B dt=10991 wait=151 +19:10:06 step 90000 loss=3.1311 lr=9.36e-05 tok=11.796B dt=11067 wait=151 +19:10:18 pushed ckpt_w52.pt,log_w52.txt +19:14:42 step 90020 loss=3.1702 lr=9.36e-05 tok=11.799B dt=11342 wait=152 +19:15:57 step 90040 loss=3.1226 lr=9.35e-05 tok=11.802B dt=11418 wait=152 +19:17:12 step 90060 loss=3.1153 lr=9.35e-05 tok=11.804B dt=11493 wait=153 +19:18:28 step 90080 loss=3.0946 lr=9.35e-05 tok=11.807B dt=11569 wait=153 +19:19:43 step 90100 loss=2.9931 lr=9.35e-05 tok=11.810B dt=11644 wait=153 +19:20:58 step 90120 loss=3.1412 lr=9.35e-05 tok=11.812B dt=11719 wait=154 +19:22:14 step 90140 loss=3.1151 lr=9.34e-05 tok=11.815B dt=11795 wait=154 +19:23:04 pushed ckpt_w52.pt,log_w52.txt +19:23:04 waited 0s for global r=241 +19:23:18 global r=241 load 4.262 -> 4.224 +19:24:02 step 90160 loss=3.1120 lr=9.34e-05 tok=11.817B dt=11902 wait=155 +19:25:17 step 90180 loss=3.0937 lr=9.34e-05 tok=11.820B dt=11978 wait=155 +19:26:33 step 90200 loss=3.1770 lr=9.34e-05 tok=11.823B dt=12054 wait=155 +19:27:49 step 90220 loss=3.1628 lr=9.34e-05 tok=11.825B dt=12129 wait=156 +19:29:04 step 90240 loss=3.1300 lr=9.33e-05 tok=11.828B dt=12205 wait=156 +19:30:20 step 90260 loss=3.1553 lr=9.33e-05 tok=11.831B dt=12280 wait=157 +19:31:35 step 90280 loss=3.0334 lr=9.33e-05 tok=11.833B dt=12356 wait=157 +19:32:51 step 90300 loss=3.0631 lr=9.33e-05 tok=11.836B dt=12431 wait=157 +19:33:03 pushed ckpt_w52.pt,log_w52.txt +19:34:34 waited 91s for global r=242 +19:34:41 global r=242 load 4.249 -> 4.224 +19:36:02 step 90320 loss=3.0770 lr=9.32e-05 tok=11.838B dt=12623 wait=158 +19:37:18 step 90340 loss=3.1217 lr=9.32e-05 tok=11.841B dt=12698 wait=158 +19:38:33 step 90360 loss=3.1153 lr=9.32e-05 tok=11.844B dt=12774 wait=159 +19:39:49 step 90380 loss=3.2121 lr=9.32e-05 tok=11.846B dt=12850 wait=159 +19:41:04 step 90400 loss=3.1407 lr=9.32e-05 tok=11.849B dt=12925 wait=160 +19:42:20 step 90420 loss=3.0779 lr=9.31e-05 tok=11.852B dt=13000 wait=160 +19:43:35 step 90440 loss=3.1417 lr=9.31e-05 tok=11.854B dt=13076 wait=160 +19:44:28 pushed ckpt_w52.pt,log_w52.txt +19:48:12 step 90460 loss=3.1198 lr=9.31e-05 tok=11.857B dt=13352 wait=161 +19:49:27 step 90480 loss=3.1474 lr=9.31e-05 tok=11.859B dt=13428 wait=161 +19:50:42 step 90500 loss=3.0840 lr=9.31e-05 tok=11.862B dt=13503 wait=162 +19:51:58 step 90520 loss=3.0612 lr=9.30e-05 tok=11.865B dt=13579 wait=162 +19:53:14 step 90540 loss=3.1116 lr=9.30e-05 tok=11.867B dt=13655 wait=162 +19:54:30 step 90560 loss=3.1252 lr=9.30e-05 tok=11.870B dt=13730 wait=163 +19:55:45 step 90580 loss=3.1318 lr=9.30e-05 tok=11.873B dt=13806 wait=163 +19:57:00 step 90600 loss=3.0823 lr=9.30e-05 tok=11.875B dt=13881 wait=164 +19:57:13 pushed ckpt_w52.pt,log_w52.txt +19:57:13 waited 0s for global r=243 +19:57:20 global r=243 load 4.242 -> 4.225 +19:58:43 step 90620 loss=3.0620 lr=9.29e-05 tok=11.878B dt=13983 wait=164 +19:59:58 step 90640 loss=3.1031 lr=9.29e-05 tok=11.880B dt=14059 wait=164 +20:01:14 step 90660 loss=3.0911 lr=9.29e-05 tok=11.883B dt=14134 wait=165 +launch +20:02:13 name=w52 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:02:18 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:02:27 params 1.08B +20:02:31 resumed step=90600 seen=11.88B round=243 +20:02:31 stream salt=90600 +20:04:30 start: accum=4, 131072 tokens per optimiser step +20:05:36 step 90601 loss=3.0204 lr=9.30e-05 tok=11.875B dt=65 wait=50 +20:05:40 step 90602 loss=3.1252 lr=9.30e-05 tok=11.875B dt=70 wait=50 +20:07:05 step 90620 loss=3.2035 lr=9.29e-05 tok=11.878B dt=155 wait=50 +20:08:29 step 90640 loss=3.3700 lr=9.29e-05 tok=11.880B dt=239 wait=50 +20:09:45 step 90660 loss=3.2995 lr=9.29e-05 tok=11.883B dt=315 wait=51 +20:11:01 step 90680 loss=3.3406 lr=9.29e-05 tok=11.886B dt=390 wait=51 +20:12:16 step 90700 loss=3.2076 lr=9.29e-05 tok=11.888B dt=466 wait=52 +20:13:32 step 90720 loss=3.3069 lr=9.28e-05 tok=11.891B dt=542 wait=52 +20:14:48 step 90740 loss=3.3656 lr=9.28e-05 tok=11.893B dt=618 wait=52 +20:15:41 pushed ckpt_w52.pt,log_w52.txt +20:17:11 waited 91s for global r=244 +20:17:26 global r=244 load 4.243 -> 4.228 +20:18:10 step 90760 loss=3.3112 lr=9.28e-05 tok=11.896B dt=819 wait=53 +20:19:25 step 90780 loss=3.2280 lr=9.28e-05 tok=11.899B dt=895 wait=53 +20:20:41 step 90800 loss=3.2914 lr=9.27e-05 tok=11.901B dt=971 wait=54 +20:21:57 step 90820 loss=3.2382 lr=9.27e-05 tok=11.904B dt=1047 wait=54 +20:23:13 step 90840 loss=3.3571 lr=9.27e-05 tok=11.907B dt=1122 wait=54 +20:24:28 step 90860 loss=3.3390 lr=9.27e-05 tok=11.909B dt=1198 wait=55 +20:25:44 step 90880 loss=3.3395 lr=9.27e-05 tok=11.912B dt=1274 wait=55 +20:27:00 step 90900 loss=3.2927 lr=9.26e-05 tok=11.914B dt=1349 wait=56 +20:27:17 pushed ckpt_w52.pt,log_w52.txt +20:28:47 waited 91s for global r=245 +20:28:57 global r=245 load 4.253 -> 4.225 +20:30:17 step 90920 loss=3.2701 lr=9.26e-05 tok=11.917B dt=1547 wait=56 +20:31:32 step 90940 loss=3.2330 lr=9.26e-05 tok=11.920B dt=1622 wait=56 +20:32:48 step 90960 loss=3.2784 lr=9.26e-05 tok=11.922B dt=1698 wait=57 +20:34:04 step 90980 loss=3.3475 lr=9.26e-05 tok=11.925B dt=1774 wait=57 +20:35:20 step 91000 loss=3.4063 lr=9.25e-05 tok=11.928B dt=1850 wait=58 +20:36:36 step 91020 loss=3.3166 lr=9.25e-05 tok=11.930B dt=1926 wait=58 +20:37:51 step 91040 loss=3.2620 lr=9.25e-05 tok=11.933B dt=2001 wait=58 +20:38:41 pushed ckpt_w52.pt,log_w52.txt +20:41:27 waited 166s for global r=246 +20:41:55 global r=246 load 4.239 -> 4.220 +20:42:38 step 91060 loss=3.3312 lr=9.25e-05 tok=11.935B dt=2288 wait=59 +20:43:54 step 91080 loss=3.2291 lr=9.25e-05 tok=11.938B dt=2363 wait=59 +20:45:09 step 91100 loss=3.2917 lr=9.24e-05 tok=11.941B dt=2439 wait=60 +20:46:25 step 91120 loss=3.2710 lr=9.24e-05 tok=11.943B dt=2515 wait=60 +20:47:41 step 91140 loss=3.3460 lr=9.24e-05 tok=11.946B dt=2591 wait=60 +20:48:57 step 91160 loss=3.3537 lr=9.24e-05 tok=11.949B dt=2667 wait=61 +20:50:13 step 91180 loss=3.4100 lr=9.23e-05 tok=11.951B dt=2742 wait=61 +20:51:28 step 91200 loss=3.3090 lr=9.23e-05 tok=11.954B dt=2818 wait=61 +20:51:43 pushed ckpt_w52.pt,log_w52.txt +20:53:59 waited 136s for global r=247 +20:54:09 global r=247 load 4.249 -> 4.224 +20:55:31 step 91220 loss=3.2746 lr=9.23e-05 tok=11.956B dt=3061 wait=62 +20:56:47 step 91240 loss=3.2031 lr=9.23e-05 tok=11.959B dt=3136 wait=62 +20:58:02 step 91260 loss=3.2537 lr=9.23e-05 tok=11.962B dt=3212 wait=63 +20:59:18 step 91280 loss=3.3176 lr=9.22e-05 tok=11.964B dt=3288 wait=63 +21:00:34 step 91300 loss=3.4336 lr=9.22e-05 tok=11.967B dt=3364 wait=63 +21:01:50 step 91320 loss=3.3628 lr=9.22e-05 tok=11.969B dt=3440 wait=64 +21:03:06 step 91340 loss=3.2880 lr=9.22e-05 tok=11.972B dt=3516 wait=64 +21:03:56 pushed ckpt_w52.pt,log_w52.txt +21:05:41 waited 106s for global r=248 +21:05:52 global r=248 load 4.246 -> 4.227 +21:06:36 step 91360 loss=3.3470 lr=9.22e-05 tok=11.975B dt=3726 wait=64 +21:07:52 step 91380 loss=3.2559 lr=9.21e-05 tok=11.977B dt=3801 wait=65 +21:09:07 step 91400 loss=3.3001 lr=9.21e-05 tok=11.980B dt=3877 wait=65 +21:10:23 step 91420 loss=3.2321 lr=9.21e-05 tok=11.983B dt=3953 wait=66 +21:11:39 step 91440 loss=3.3479 lr=9.21e-05 tok=11.985B dt=4028 wait=66 +21:12:54 step 91460 loss=3.3706 lr=9.21e-05 tok=11.988B dt=4104 wait=66 +21:14:10 step 91480 loss=3.4021 lr=9.20e-05 tok=11.990B dt=4180 wait=67 +21:15:26 step 91500 loss=3.2688 lr=9.20e-05 tok=11.993B dt=4255 wait=67 +21:15:40 pushed ckpt_w52.pt,log_w52.txt +21:17:25 waited 106s for global r=249 +21:17:34 global r=249 load 4.242 -> 4.226 +21:18:55 step 91520 loss=3.2550 lr=9.20e-05 tok=11.996B dt=4465 wait=67 +21:20:11 step 91540 loss=3.2124 lr=9.20e-05 tok=11.998B dt=4540 wait=68 +21:21:26 step 91560 loss=3.2538 lr=9.19e-05 tok=12.001B dt=4616 wait=68 +21:22:42 step 91580 loss=3.2761 lr=9.19e-05 tok=12.004B dt=4691 wait=69 +21:23:58 step 91600 loss=3.4114 lr=9.19e-05 tok=12.006B dt=4767 wait=69 +21:25:13 step 91620 loss=3.3484 lr=9.19e-05 tok=12.009B dt=4843 wait=69 +21:26:29 step 91640 loss=3.2790 lr=9.19e-05 tok=12.011B dt=4919 wait=70 +21:27:19 pushed ckpt_w52.pt,log_w52.txt +21:31:05 step 91660 loss=3.3950 lr=9.18e-05 tok=12.014B dt=5195 wait=70 +21:32:21 step 91680 loss=3.4601 lr=9.18e-05 tok=12.017B dt=5271 wait=70 +21:33:37 step 91700 loss=3.3947 lr=9.18e-05 tok=12.019B dt=5347 wait=71 +21:34:53 step 91720 loss=3.4022 lr=9.18e-05 tok=12.022B dt=5422 wait=71 +21:36:08 step 91740 loss=3.4087 lr=9.18e-05 tok=12.025B dt=5498 wait=72 +21:37:24 step 91760 loss=3.2664 lr=9.17e-05 tok=12.027B dt=5574 wait=72 +21:38:40 step 91780 loss=3.2968 lr=9.17e-05 tok=12.030B dt=5650 wait=72 +21:39:56 step 91800 loss=3.3414 lr=9.17e-05 tok=12.032B dt=5725 wait=73 +21:40:08 pushed ckpt_w52.pt,log_w52.txt +21:44:28 step 91820 loss=3.2463 lr=9.17e-05 tok=12.035B dt=5998 wait=73 +21:45:44 step 91840 loss=3.2635 lr=9.17e-05 tok=12.038B dt=6074 wait=74 +21:47:00 step 91860 loss=3.3691 lr=9.16e-05 tok=12.040B dt=6150 wait=74 +21:48:16 step 91880 loss=3.2766 lr=9.16e-05 tok=12.043B dt=6225 wait=74 +21:49:32 step 91900 loss=3.2940 lr=9.16e-05 tok=12.046B dt=6301 wait=75 +21:50:47 step 91920 loss=3.2486 lr=9.16e-05 tok=12.048B dt=6377 wait=75 +21:52:03 step 91940 loss=3.3893 lr=9.15e-05 tok=12.051B dt=6453 wait=75 +21:52:56 pushed ckpt_w52.pt,log_w52.txt +21:56:40 step 91960 loss=3.3727 lr=9.15e-05 tok=12.053B dt=6730 wait=76 +21:57:55 step 91980 loss=3.3182 lr=9.15e-05 tok=12.056B dt=6805 wait=76 +21:59:11 step 92000 loss=3.2693 lr=9.15e-05 tok=12.059B dt=6881 wait=77 +22:00:27 step 92020 loss=3.3192 lr=9.15e-05 tok=12.061B dt=6957 wait=77 +22:01:43 step 92040 loss=3.2240 lr=9.14e-05 tok=12.064B dt=7032 wait=77 +22:02:59 step 92060 loss=3.3054 lr=9.14e-05 tok=12.066B dt=7108 wait=78 +22:04:14 step 92080 loss=3.3077 lr=9.14e-05 tok=12.069B dt=7184 wait=78 +22:05:30 step 92100 loss=3.2714 lr=9.14e-05 tok=12.072B dt=7260 wait=78 +22:05:41 pushed ckpt_w52.pt,log_w52.txt +22:06:12 waited 30s for global r=250 +22:06:18 global r=250 load 4.257 -> 4.238 +22:07:39 step 92120 loss=3.2801 lr=9.14e-05 tok=12.074B dt=7389 wait=79 +22:08:55 step 92140 loss=3.2938 lr=9.13e-05 tok=12.077B dt=7464 wait=79 +22:10:10 step 92160 loss=3.3014 lr=9.13e-05 tok=12.080B dt=7540 wait=80 +22:11:26 step 92180 loss=3.3463 lr=9.13e-05 tok=12.082B dt=7616 wait=80 +22:12:42 step 92200 loss=3.4082 lr=9.13e-05 tok=12.085B dt=7692 wait=80 +22:13:58 step 92220 loss=3.3391 lr=9.13e-05 tok=12.087B dt=7768 wait=81 +22:15:13 step 92240 loss=3.2453 lr=9.12e-05 tok=12.090B dt=7843 wait=81 +22:16:05 pushed ckpt_w52.pt,log_w52.txt +22:19:54 step 92260 loss=3.3512 lr=9.12e-05 tok=12.093B dt=8123 wait=81 +22:21:10 step 92280 loss=3.4481 lr=9.12e-05 tok=12.095B dt=8199 wait=82 +22:22:26 step 92300 loss=3.4297 lr=9.12e-05 tok=12.098B dt=8275 wait=82 +22:23:42 step 92320 loss=3.4187 lr=9.12e-05 tok=12.101B dt=8352 wait=83 +22:24:58 step 92340 loss=3.4268 lr=9.11e-05 tok=12.103B dt=8428 wait=83 +22:26:14 step 92360 loss=3.2682 lr=9.11e-05 tok=12.106B dt=8504 wait=84 +22:27:30 step 92380 loss=3.3204 lr=9.11e-05 tok=12.108B dt=8580 wait=84 +22:28:45 step 92400 loss=3.3152 lr=9.11e-05 tok=12.111B dt=8655 wait=84 +22:28:59 pushed ckpt_w52.pt,log_w52.txt +22:28:59 waited 0s for global r=251 +22:29:05 global r=251 load 4.251 -> 4.237 +22:30:26 step 92420 loss=3.2541 lr=9.10e-05 tok=12.114B dt=8756 wait=85 +22:31:42 step 92440 loss=3.2285 lr=9.10e-05 tok=12.116B dt=8832 wait=85 +22:32:58 step 92460 loss=3.2599 lr=9.10e-05 tok=12.119B dt=8907 wait=85 +22:34:14 step 92480 loss=3.3204 lr=9.10e-05 tok=12.122B dt=8983 wait=86 +22:35:29 step 92500 loss=3.3888 lr=9.10e-05 tok=12.124B dt=9059 wait=86 +22:36:46 step 92520 loss=3.3643 lr=9.09e-05 tok=12.127B dt=9135 wait=87 +22:38:01 step 92540 loss=3.2803 lr=9.09e-05 tok=12.129B dt=9211 wait=87 +22:38:51 pushed ckpt_w52.pt,log_w52.txt +22:40:22 waited 91s for global r=252 +22:40:28 global r=252 load 4.237 -> 4.228 +22:41:11 step 92560 loss=3.3739 lr=9.09e-05 tok=12.132B dt=9401 wait=87 +22:42:27 step 92580 loss=3.2404 lr=9.09e-05 tok=12.135B dt=9476 wait=88 +22:43:42 step 92600 loss=3.2583 lr=9.09e-05 tok=12.137B dt=9552 wait=88 +22:44:58 step 92620 loss=3.2182 lr=9.08e-05 tok=12.140B dt=9628 wait=89 +22:46:14 step 92640 loss=3.3638 lr=9.08e-05 tok=12.143B dt=9704 wait=89 +22:47:30 step 92660 loss=3.3382 lr=9.08e-05 tok=12.145B dt=9779 wait=89 +22:48:45 step 92680 loss=3.3842 lr=9.08e-05 tok=12.148B dt=9855 wait=90 +22:50:01 step 92700 loss=3.2938 lr=9.08e-05 tok=12.150B dt=9931 wait=90 +22:50:20 pushed ckpt_w52.pt,log_w52.txt +22:54:43 step 92720 loss=3.4002 lr=9.07e-05 tok=12.153B dt=10213 wait=90 +22:55:59 step 92740 loss=3.4459 lr=9.07e-05 tok=12.156B dt=10289 wait=91 +22:57:15 step 92760 loss=3.4827 lr=9.07e-05 tok=12.158B dt=10365 wait=91 +22:58:31 step 92780 loss=3.3562 lr=9.07e-05 tok=12.161B dt=10441 wait=92 +22:59:47 step 92800 loss=3.3209 lr=9.06e-05 tok=12.163B dt=10517 wait=92 +23:01:03 step 92820 loss=3.3173 lr=9.06e-05 tok=12.166B dt=10592 wait=92 +23:02:18 step 92840 loss=3.2479 lr=9.06e-05 tok=12.169B dt=10668 wait=93 +23:03:08 pushed ckpt_w52.pt,log_w52.txt +23:03:08 waited 0s for global r=253 +23:03:15 global r=253 load 4.251 -> 4.223 +23:03:58 step 92860 loss=3.3182 lr=9.06e-05 tok=12.171B dt=10768 wait=93 +23:05:14 step 92880 loss=3.2236 lr=9.06e-05 tok=12.174B dt=10843 wait=93 +23:06:29 step 92900 loss=3.2374 lr=9.05e-05 tok=12.177B dt=10919 wait=94 +23:07:45 step 92920 loss=3.2474 lr=9.05e-05 tok=12.179B dt=10995 wait=94 +23:09:00 step 92940 loss=3.3755 lr=9.05e-05 tok=12.182B dt=11070 wait=95 +23:10:16 step 92960 loss=3.3260 lr=9.05e-05 tok=12.184B dt=11146 wait=95 +23:11:32 step 92980 loss=3.4035 lr=9.05e-05 tok=12.187B dt=11222 wait=95 +23:12:48 step 93000 loss=3.3181 lr=9.04e-05 tok=12.190B dt=11297 wait=96 +23:12:59 pushed ckpt_w52.pt,log_w52.txt +23:14:30 waited 91s for global r=254 +23:14:40 global r=254 load 4.240 -> 4.223 +23:15:58 step 93020 loss=3.3092 lr=9.04e-05 tok=12.192B dt=11488 wait=96 +23:17:14 step 93040 loss=3.2117 lr=9.04e-05 tok=12.195B dt=11564 wait=96 +23:18:30 step 93060 loss=3.2517 lr=9.04e-05 tok=12.198B dt=11640 wait=97 +23:19:46 step 93080 loss=3.3039 lr=9.04e-05 tok=12.200B dt=11715 wait=97 +23:21:01 step 93100 loss=3.4021 lr=9.03e-05 tok=12.203B dt=11791 wait=98 +23:22:17 step 93120 loss=3.3755 lr=9.03e-05 tok=12.205B dt=11867 wait=98 +23:23:33 step 93140 loss=3.2908 lr=9.03e-05 tok=12.208B dt=11943 wait=98 +23:24:22 pushed ckpt_w52.pt,log_w52.txt +23:28:07 step 93160 loss=3.3965 lr=9.03e-05 tok=12.211B dt=12216 wait=99 +23:29:22 step 93180 loss=3.4793 lr=9.02e-05 tok=12.213B dt=12292 wait=99 +23:30:38 step 93200 loss=3.4413 lr=9.02e-05 tok=12.216B dt=12368 wait=100 +23:31:54 step 93220 loss=3.3303 lr=9.02e-05 tok=12.219B dt=12444 wait=100 +23:33:10 step 93240 loss=3.3996 lr=9.02e-05 tok=12.221B dt=12519 wait=100 +23:34:26 step 93260 loss=3.2921 lr=9.02e-05 tok=12.224B dt=12595 wait=101 +23:35:42 step 93280 loss=3.3387 lr=9.01e-05 tok=12.226B dt=12671 wait=101 +23:36:57 step 93300 loss=3.3291 lr=9.01e-05 tok=12.229B dt=12747 wait=101 +23:37:09 pushed ckpt_w52.pt,log_w52.txt +23:37:09 waited 0s for global r=255 +23:37:19 global r=255 load 4.246 -> 4.222 +23:38:39 step 93320 loss=3.3053 lr=9.01e-05 tok=12.232B dt=12848 wait=102 +23:39:54 step 93340 loss=3.2367 lr=9.01e-05 tok=12.234B dt=12924 wait=102 +23:41:10 step 93360 loss=3.2384 lr=9.01e-05 tok=12.237B dt=13000 wait=103 +23:42:26 step 93380 loss=3.2690 lr=9.00e-05 tok=12.240B dt=13076 wait=103 +23:43:42 step 93400 loss=3.3446 lr=9.00e-05 tok=12.242B dt=13151 wait=103 +23:44:57 step 93420 loss=3.3333 lr=9.00e-05 tok=12.245B dt=13227 wait=104 +23:46:13 step 93440 loss=3.2839 lr=9.00e-05 tok=12.247B dt=13303 wait=104 +23:47:03 pushed ckpt_w52.pt,log_w52.txt +23:48:49 waited 106s for global r=256 +23:48:55 global r=256 load 4.235 -> 4.221 +23:49:42 step 93460 loss=3.3740 lr=9.00e-05 tok=12.250B dt=13512 wait=105 +23:50:58 step 93480 loss=3.2759 lr=8.99e-05 tok=12.253B dt=13587 wait=105 +23:52:13 step 93500 loss=3.2988 lr=8.99e-05 tok=12.255B dt=13663 wait=105 +23:53:29 step 93520 loss=3.2553 lr=8.99e-05 tok=12.258B dt=13739 wait=106 +23:54:45 step 93540 loss=3.3701 lr=8.99e-05 tok=12.260B dt=13815 wait=106 +23:56:01 step 93560 loss=3.3307 lr=8.98e-05 tok=12.263B dt=13890 wait=106 +23:57:16 step 93580 loss=3.3148 lr=8.98e-05 tok=12.266B dt=13966 wait=107 +23:58:32 step 93600 loss=3.3121 lr=8.98e-05 tok=12.268B dt=14042 wait=107 +23:58:44 pushed ckpt_w52.pt,log_w52.txt +00:03:07 step 93620 loss=3.3707 lr=8.98e-05 tok=12.271B dt=14317 wait=108 +00:04:23 step 93640 loss=3.4204 lr=8.98e-05 tok=12.274B dt=14393 wait=108 +00:05:39 step 93660 loss=3.5013 lr=8.97e-05 tok=12.276B dt=14469 wait=108 +00:06:55 step 93680 loss=3.3874 lr=8.97e-05 tok=12.279B dt=14544 wait=109 +00:08:11 step 93700 loss=3.3269 lr=8.97e-05 tok=12.281B dt=14620 wait=109 +00:09:26 step 93720 loss=3.3287 lr=8.97e-05 tok=12.284B dt=14696 wait=110 +00:10:42 step 93740 loss=3.2653 lr=8.97e-05 tok=12.287B dt=14771 wait=110 +00:11:34 pushed ckpt_w52.pt,log_w52.txt +00:11:34 waited 0s for global r=257 +00:11:40 global r=257 load 4.250 -> 4.220 +00:12:24 step 93760 loss=3.2940 lr=8.96e-05 tok=12.289B dt=14874 wait=110 +00:13:40 step 93780 loss=3.2366 lr=8.96e-05 tok=12.292B dt=14950 wait=111 +00:14:55 step 93800 loss=3.2563 lr=8.96e-05 tok=12.295B dt=15025 wait=111 +00:16:11 step 93820 loss=3.2323 lr=8.96e-05 tok=12.297B dt=15101 wait=111 +00:17:27 step 93840 loss=3.3602 lr=8.95e-05 tok=12.300B dt=15177 wait=112 +00:18:43 step 93860 loss=3.3262 lr=8.95e-05 tok=12.302B dt=15253 wait=112 +00:19:59 step 93880 loss=3.3686 lr=8.95e-05 tok=12.305B dt=15329 wait=113 +00:21:15 step 93900 loss=3.2963 lr=8.95e-05 tok=12.308B dt=15404 wait=113 +00:21:30 pushed ckpt_w52.pt,log_w52.txt +00:23:00 waited 91s for global r=258 +00:23:07 global r=258 load 4.242 -> 4.222 +00:24:27 step 93920 loss=3.3018 lr=8.95e-05 tok=12.310B dt=15597 wait=113 +00:25:43 step 93940 loss=3.2533 lr=8.94e-05 tok=12.313B dt=15673 wait=114 +00:26:59 step 93960 loss=3.2443 lr=8.94e-05 tok=12.316B dt=15748 wait=114 +00:28:14 step 93980 loss=3.2736 lr=8.94e-05 tok=12.318B dt=15824 wait=114 +00:29:30 step 94000 loss=3.3665 lr=8.94e-05 tok=12.321B dt=15900 wait=115 +00:30:46 step 94020 loss=3.3237 lr=8.94e-05 tok=12.323B dt=15975 wait=115 +00:32:02 step 94040 loss=3.2996 lr=8.93e-05 tok=12.326B dt=16051 wait=116 +00:32:54 pushed ckpt_w52.pt,log_w52.txt +00:36:39 step 94060 loss=3.3916 lr=8.93e-05 tok=12.329B dt=16329 wait=116 +00:37:55 step 94080 loss=3.4658 lr=8.93e-05 tok=12.331B dt=16405 wait=116 +00:39:11 step 94100 loss=3.4146 lr=8.93e-05 tok=12.334B dt=16481 wait=117 +00:40:27 step 94120 loss=3.3932 lr=8.93e-05 tok=12.336B dt=16557 wait=117 +00:41:43 step 94140 loss=3.3999 lr=8.92e-05 tok=12.339B dt=16632 wait=117 +00:42:58 step 94160 loss=3.2616 lr=8.92e-05 tok=12.342B dt=16708 wait=118 +00:44:14 step 94180 loss=3.3469 lr=8.92e-05 tok=12.344B dt=16784 wait=118 +00:45:30 step 94200 loss=3.3098 lr=8.92e-05 tok=12.347B dt=16860 wait=119 +00:45:45 pushed ckpt_w52.pt,log_w52.txt +00:45:45 waited 0s for global r=259 +00:45:51 global r=259 load 4.250 -> 4.221 +00:47:14 step 94220 loss=3.2746 lr=8.91e-05 tok=12.350B dt=16963 wait=119 +00:48:29 step 94240 loss=3.2220 lr=8.91e-05 tok=12.352B dt=17039 wait=119 +00:49:45 step 94260 loss=3.2805 lr=8.91e-05 tok=12.355B dt=17115 wait=120 +00:51:01 step 94280 loss=3.3179 lr=8.91e-05 tok=12.357B dt=17191 wait=120 +00:52:17 step 94300 loss=3.3972 lr=8.91e-05 tok=12.360B dt=17266 wait=121 +00:53:32 step 94320 loss=3.3632 lr=8.90e-05 tok=12.363B dt=17342 wait=121 +00:54:48 step 94340 loss=3.2747 lr=8.90e-05 tok=12.365B dt=17418 wait=121 +00:55:38 pushed ckpt_w52.pt,log_w52.txt +00:56:54 waited 76s for global r=260 +00:57:00 global r=260 load 4.229 -> 4.218 +00:57:42 step 94360 loss=3.3585 lr=8.90e-05 tok=12.368B dt=17592 wait=122 +00:58:58 step 94380 loss=3.2435 lr=8.90e-05 tok=12.371B dt=17667 wait=122 +01:00:13 step 94400 loss=3.2761 lr=8.90e-05 tok=12.373B dt=17743 wait=122 +01:01:29 step 94420 loss=3.2421 lr=8.89e-05 tok=12.376B dt=17819 wait=123 +01:02:45 step 94440 loss=3.3693 lr=8.89e-05 tok=12.378B dt=17895 wait=123 +01:04:01 step 94460 loss=3.3967 lr=8.89e-05 tok=12.381B dt=17971 wait=124 +01:05:17 step 94480 loss=3.3617 lr=8.89e-05 tok=12.384B dt=18046 wait=124 +01:06:32 step 94500 loss=3.2693 lr=8.89e-05 tok=12.386B dt=18122 wait=124 diff --git a/log_w53.txt b/log_w53.txt new file mode 100644 index 0000000000000000000000000000000000000000..ad56f821b8e4a2bacd552d603ce1cf3f959dc984 --- /dev/null +++ b/log_w53.txt @@ -0,0 +1,289 @@ +launch +16:02:55 name=w53 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:03:01 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:03:12 params 1.08B +16:03:14 resumed step=87600 seen=11.48B round=229 +16:03:14 stream salt=87600 +16:05:54 start: accum=4, 131072 tokens per optimiser step +16:07:58 step 87601 loss=3.1801 lr=9.61e-05 tok=11.482B dt=124 wait=93 +16:08:03 step 87602 loss=3.0529 lr=9.61e-05 tok=11.482B dt=128 wait=93 +16:09:32 step 87620 loss=3.2189 lr=9.61e-05 tok=11.485B dt=218 wait=93 +16:10:57 step 87640 loss=3.2369 lr=9.60e-05 tok=11.487B dt=303 wait=94 +16:12:13 step 87660 loss=3.2207 lr=9.60e-05 tok=11.490B dt=378 wait=94 +16:13:28 step 87680 loss=3.1202 lr=9.60e-05 tok=11.492B dt=453 wait=95 +16:14:43 step 87700 loss=3.1895 lr=9.60e-05 tok=11.495B dt=529 wait=95 +16:15:58 step 87720 loss=3.0836 lr=9.60e-05 tok=11.498B dt=604 wait=96 +16:17:14 step 87740 loss=3.0799 lr=9.59e-05 tok=11.500B dt=679 wait=96 +16:18:06 pushed ckpt_w53.pt,log_w53.txt +16:19:37 waited 91s for global r=230 +16:19:52 global r=230 load 4.256 -> 4.231 +16:20:36 step 87760 loss=3.1610 lr=9.59e-05 tok=11.503B dt=882 wait=97 +16:21:51 step 87780 loss=3.0917 lr=9.59e-05 tok=11.506B dt=957 wait=97 +16:23:06 step 87800 loss=3.1411 lr=9.59e-05 tok=11.508B dt=1032 wait=98 +16:24:22 step 87820 loss=3.1828 lr=9.59e-05 tok=11.511B dt=1107 wait=98 +16:25:37 step 87840 loss=3.1076 lr=9.58e-05 tok=11.513B dt=1182 wait=98 +16:26:52 step 87860 loss=3.1809 lr=9.58e-05 tok=11.516B dt=1258 wait=99 +16:28:07 step 87880 loss=3.0554 lr=9.58e-05 tok=11.519B dt=1333 wait=99 +16:29:23 step 87900 loss=3.1494 lr=9.58e-05 tok=11.521B dt=1408 wait=100 +16:29:38 pushed ckpt_w53.pt,log_w53.txt +16:31:23 waited 106s for global r=231 +16:31:36 global r=231 load 4.249 -> 4.232 +16:32:56 step 87920 loss=3.1261 lr=9.58e-05 tok=11.524B dt=1622 wait=100 +16:34:11 step 87940 loss=3.1334 lr=9.57e-05 tok=11.526B dt=1697 wait=101 +16:35:27 step 87960 loss=3.1443 lr=9.57e-05 tok=11.529B dt=1772 wait=101 +16:36:42 step 87980 loss=3.1542 lr=9.57e-05 tok=11.532B dt=1848 wait=102 +16:37:57 step 88000 loss=3.1184 lr=9.57e-05 tok=11.534B dt=1923 wait=102 +16:39:13 step 88020 loss=3.0887 lr=9.56e-05 tok=11.537B dt=1998 wait=103 +16:40:28 step 88040 loss=3.1230 lr=9.56e-05 tok=11.540B dt=2073 wait=103 +16:41:29 pushed ckpt_w53.pt,log_w53.txt +16:44:16 waited 166s for global r=232 +16:44:25 global r=232 load 4.245 -> 4.227 +16:45:09 step 88060 loss=3.0887 lr=9.56e-05 tok=11.542B dt=2355 wait=104 +16:46:24 step 88080 loss=3.1065 lr=9.56e-05 tok=11.545B dt=2430 wait=104 +16:47:40 step 88100 loss=3.1807 lr=9.56e-05 tok=11.547B dt=2505 wait=104 +16:48:55 step 88120 loss=3.1857 lr=9.55e-05 tok=11.550B dt=2580 wait=105 +16:50:10 step 88140 loss=3.0792 lr=9.55e-05 tok=11.553B dt=2656 wait=105 +16:51:25 step 88160 loss=3.1821 lr=9.55e-05 tok=11.555B dt=2731 wait=106 +16:52:40 step 88180 loss=3.0998 lr=9.55e-05 tok=11.558B dt=2806 wait=106 +16:53:56 step 88200 loss=3.1426 lr=9.55e-05 tok=11.561B dt=2881 wait=107 +16:54:11 pushed ckpt_w53.pt,log_w53.txt +16:58:34 step 88220 loss=3.1483 lr=9.54e-05 tok=11.563B dt=3160 wait=107 +16:59:50 step 88240 loss=3.0403 lr=9.54e-05 tok=11.566B dt=3235 wait=108 +17:01:05 step 88260 loss=3.0543 lr=9.54e-05 tok=11.568B dt=3310 wait=108 +17:02:20 step 88280 loss=3.0822 lr=9.54e-05 tok=11.571B dt=3385 wait=109 +17:03:35 step 88300 loss=3.0495 lr=9.54e-05 tok=11.574B dt=3460 wait=109 +17:04:50 step 88320 loss=3.1365 lr=9.53e-05 tok=11.576B dt=3536 wait=109 +17:06:05 step 88340 loss=3.1242 lr=9.53e-05 tok=11.579B dt=3611 wait=110 +17:06:56 pushed ckpt_w53.pt,log_w53.txt +17:06:57 waited 0s for global r=233 +17:07:05 global r=233 load 4.254 -> 4.228 +17:07:48 step 88360 loss=3.1308 lr=9.53e-05 tok=11.582B dt=3714 wait=110 +17:09:04 step 88380 loss=3.1110 lr=9.53e-05 tok=11.584B dt=3789 wait=111 +17:10:19 step 88400 loss=3.1606 lr=9.53e-05 tok=11.587B dt=3864 wait=111 +17:11:34 step 88420 loss=3.1942 lr=9.52e-05 tok=11.589B dt=3940 wait=112 +17:12:49 step 88440 loss=3.1064 lr=9.52e-05 tok=11.592B dt=4015 wait=112 +17:14:05 step 88460 loss=3.1905 lr=9.52e-05 tok=11.595B dt=4090 wait=113 +17:15:20 step 88480 loss=3.0650 lr=9.52e-05 tok=11.597B dt=4165 wait=113 +17:16:35 step 88500 loss=3.1044 lr=9.51e-05 tok=11.600B dt=4241 wait=114 +17:16:48 pushed ckpt_w53.pt,log_w53.txt +17:18:19 waited 91s for global r=234 +17:18:29 global r=234 load 4.245 -> 4.230 +17:19:48 step 88520 loss=3.1093 lr=9.51e-05 tok=11.602B dt=4433 wait=114 +17:21:03 step 88540 loss=3.1453 lr=9.51e-05 tok=11.605B dt=4509 wait=115 +17:22:19 step 88560 loss=3.0916 lr=9.51e-05 tok=11.608B dt=4584 wait=115 +17:23:34 step 88580 loss=3.1729 lr=9.51e-05 tok=11.610B dt=4659 wait=115 +17:24:49 step 88600 loss=3.1766 lr=9.50e-05 tok=11.613B dt=4735 wait=116 +17:26:04 step 88620 loss=3.1348 lr=9.50e-05 tok=11.616B dt=4810 wait=116 +17:27:20 step 88640 loss=3.1479 lr=9.50e-05 tok=11.618B dt=4885 wait=117 +17:28:10 pushed ckpt_w53.pt,log_w53.txt +17:31:52 step 88660 loss=3.1017 lr=9.50e-05 tok=11.621B dt=5158 wait=117 +17:33:07 step 88680 loss=3.1259 lr=9.50e-05 tok=11.623B dt=5233 wait=118 +17:34:22 step 88700 loss=3.0633 lr=9.49e-05 tok=11.626B dt=5308 wait=118 +17:35:37 step 88720 loss=3.0544 lr=9.49e-05 tok=11.629B dt=5383 wait=119 +17:36:53 step 88740 loss=3.1121 lr=9.49e-05 tok=11.631B dt=5458 wait=119 +17:38:08 step 88760 loss=3.1058 lr=9.49e-05 tok=11.634B dt=5533 wait=120 +17:39:23 step 88780 loss=3.1122 lr=9.49e-05 tok=11.637B dt=5608 wait=120 +17:40:38 step 88800 loss=3.0823 lr=9.48e-05 tok=11.639B dt=5684 wait=120 +17:40:53 pushed ckpt_w53.pt,log_w53.txt +17:40:53 waited 0s for global r=235 +17:41:04 global r=235 load 4.257 -> 4.231 +17:42:27 step 88820 loss=3.1071 lr=9.48e-05 tok=11.642B dt=5793 wait=121 +17:43:42 step 88840 loss=3.1118 lr=9.48e-05 tok=11.644B dt=5868 wait=121 +17:44:58 step 88860 loss=3.1525 lr=9.48e-05 tok=11.647B dt=5943 wait=122 +17:46:13 step 88880 loss=3.2051 lr=9.48e-05 tok=11.650B dt=6018 wait=122 +17:47:28 step 88900 loss=3.1822 lr=9.47e-05 tok=11.652B dt=6094 wait=123 +17:48:43 step 88920 loss=3.1381 lr=9.47e-05 tok=11.655B dt=6169 wait=123 +17:49:59 step 88940 loss=3.1675 lr=9.47e-05 tok=11.658B dt=6244 wait=124 +17:50:49 pushed ckpt_w53.pt,log_w53.txt +17:52:19 waited 91s for global r=236 +17:52:33 global r=236 load 4.243 -> 4.225 +17:53:15 step 88960 loss=3.0906 lr=9.47e-05 tok=11.660B dt=6441 wait=124 +17:54:30 step 88980 loss=3.0975 lr=9.46e-05 tok=11.663B dt=6516 wait=125 +17:55:46 step 89000 loss=3.1365 lr=9.46e-05 tok=11.665B dt=6591 wait=125 +17:57:01 step 89020 loss=3.1652 lr=9.46e-05 tok=11.668B dt=6666 wait=125 +17:58:16 step 89040 loss=3.0869 lr=9.46e-05 tok=11.671B dt=6742 wait=126 +17:59:31 step 89060 loss=3.1868 lr=9.46e-05 tok=11.673B dt=6817 wait=126 +18:00:46 step 89080 loss=3.0521 lr=9.45e-05 tok=11.676B dt=6892 wait=127 +18:02:02 step 89100 loss=3.1110 lr=9.45e-05 tok=11.679B dt=6967 wait=127 +18:02:42 pushed ckpt_w53.pt,log_w53.txt +18:07:02 step 89120 loss=3.1600 lr=9.45e-05 tok=11.681B dt=7268 wait=128 +18:08:17 step 89140 loss=3.0577 lr=9.45e-05 tok=11.684B dt=7343 wait=128 +18:09:32 step 89160 loss=3.0977 lr=9.45e-05 tok=11.686B dt=7418 wait=129 +18:10:48 step 89180 loss=3.1223 lr=9.44e-05 tok=11.689B dt=7493 wait=129 +18:12:03 step 89200 loss=3.0690 lr=9.44e-05 tok=11.692B dt=7569 wait=130 +18:13:18 step 89220 loss=3.1788 lr=9.44e-05 tok=11.694B dt=7644 wait=130 +18:14:33 step 89240 loss=3.1203 lr=9.44e-05 tok=11.697B dt=7719 wait=131 +18:15:24 pushed ckpt_w53.pt,log_w53.txt +18:15:25 waited 0s for global r=237 +18:15:35 global r=237 load 4.255 -> 4.232 +18:16:16 step 89260 loss=3.0881 lr=9.44e-05 tok=11.699B dt=7822 wait=131 +18:17:31 step 89280 loss=3.0493 lr=9.43e-05 tok=11.702B dt=7897 wait=131 +18:18:47 step 89300 loss=3.1491 lr=9.43e-05 tok=11.705B dt=7972 wait=132 +18:20:02 step 89320 loss=3.2138 lr=9.43e-05 tok=11.707B dt=8047 wait=132 +18:21:17 step 89340 loss=3.1191 lr=9.43e-05 tok=11.710B dt=8123 wait=133 +18:22:32 step 89360 loss=3.1829 lr=9.43e-05 tok=11.713B dt=8198 wait=133 +18:23:47 step 89380 loss=3.0567 lr=9.42e-05 tok=11.715B dt=8273 wait=134 +18:25:03 step 89400 loss=3.0929 lr=9.42e-05 tok=11.718B dt=8348 wait=134 +18:25:17 pushed ckpt_w53.pt,log_w53.txt +18:26:17 waited 60s for global r=238 +18:26:25 global r=238 load 4.251 -> 4.231 +18:27:48 step 89420 loss=3.0914 lr=9.42e-05 tok=11.720B dt=8513 wait=135 +18:29:03 step 89440 loss=3.1416 lr=9.42e-05 tok=11.723B dt=8588 wait=135 +18:30:18 step 89460 loss=3.1332 lr=9.41e-05 tok=11.726B dt=8664 wait=135 +18:31:33 step 89480 loss=3.1987 lr=9.41e-05 tok=11.728B dt=8739 wait=136 +18:32:49 step 89500 loss=3.1151 lr=9.41e-05 tok=11.731B dt=8814 wait=136 +18:34:04 step 89520 loss=3.0787 lr=9.41e-05 tok=11.734B dt=8890 wait=137 +18:35:19 step 89540 loss=3.1090 lr=9.41e-05 tok=11.736B dt=8965 wait=137 +18:36:11 pushed ckpt_w53.pt,log_w53.txt +18:39:55 step 89560 loss=3.0897 lr=9.40e-05 tok=11.739B dt=9240 wait=138 +18:41:10 step 89580 loss=3.1758 lr=9.40e-05 tok=11.741B dt=9316 wait=138 +18:42:26 step 89600 loss=3.0745 lr=9.40e-05 tok=11.744B dt=9391 wait=139 +18:43:41 step 89620 loss=3.0816 lr=9.40e-05 tok=11.747B dt=9466 wait=139 +18:44:56 step 89640 loss=3.0998 lr=9.40e-05 tok=11.749B dt=9541 wait=140 +18:46:11 step 89660 loss=3.0879 lr=9.39e-05 tok=11.752B dt=9616 wait=140 +18:47:26 step 89680 loss=3.1300 lr=9.39e-05 tok=11.755B dt=9692 wait=141 +18:48:41 step 89700 loss=3.0994 lr=9.39e-05 tok=11.757B dt=9767 wait=141 +18:48:55 pushed ckpt_w53.pt,log_w53.txt +18:48:56 waited 0s for global r=239 +18:49:05 global r=239 load 4.253 -> 4.230 +18:50:26 step 89720 loss=3.0547 lr=9.39e-05 tok=11.760B dt=9871 wait=141 +18:51:41 step 89740 loss=3.0820 lr=9.39e-05 tok=11.762B dt=9946 wait=142 +18:52:56 step 89760 loss=3.0990 lr=9.38e-05 tok=11.765B dt=10022 wait=142 +18:54:11 step 89780 loss=3.1860 lr=9.38e-05 tok=11.768B dt=10097 wait=143 +18:55:26 step 89800 loss=3.1570 lr=9.38e-05 tok=11.770B dt=10172 wait=143 +18:56:42 step 89820 loss=3.0531 lr=9.38e-05 tok=11.773B dt=10247 wait=144 +18:57:57 step 89840 loss=3.1198 lr=9.38e-05 tok=11.776B dt=10323 wait=144 +18:58:50 pushed ckpt_w53.pt,log_w53.txt +19:00:20 waited 91s for global r=240 +19:00:32 global r=240 load 4.255 -> 4.227 +19:01:17 step 89860 loss=3.0825 lr=9.37e-05 tok=11.778B dt=10522 wait=145 +19:02:32 step 89880 loss=3.1184 lr=9.37e-05 tok=11.781B dt=10598 wait=145 +19:03:47 step 89900 loss=3.1539 lr=9.37e-05 tok=11.783B dt=10673 wait=146 +19:05:03 step 89920 loss=3.1711 lr=9.37e-05 tok=11.786B dt=10748 wait=146 +19:06:18 step 89940 loss=3.0820 lr=9.36e-05 tok=11.789B dt=10823 wait=146 +19:07:33 step 89960 loss=3.2212 lr=9.36e-05 tok=11.791B dt=10899 wait=147 +19:08:48 step 89980 loss=3.0855 lr=9.36e-05 tok=11.794B dt=10974 wait=147 +19:10:04 step 90000 loss=3.1243 lr=9.36e-05 tok=11.796B dt=11049 wait=148 +19:10:16 pushed ckpt_w53.pt,log_w53.txt +19:14:43 step 90020 loss=3.0927 lr=9.36e-05 tok=11.799B dt=11329 wait=148 +19:15:58 step 90040 loss=3.1011 lr=9.35e-05 tok=11.802B dt=11404 wait=149 +19:17:13 step 90060 loss=3.0943 lr=9.35e-05 tok=11.804B dt=11479 wait=149 +19:18:28 step 90080 loss=3.0849 lr=9.35e-05 tok=11.807B dt=11554 wait=150 +19:19:43 step 90100 loss=3.0553 lr=9.35e-05 tok=11.810B dt=11629 wait=150 +19:20:59 step 90120 loss=3.1988 lr=9.35e-05 tok=11.812B dt=11704 wait=151 +19:22:14 step 90140 loss=3.1541 lr=9.34e-05 tok=11.815B dt=11780 wait=151 +19:23:04 pushed ckpt_w53.pt,log_w53.txt +19:23:05 waited 0s for global r=241 +19:23:14 global r=241 load 4.240 -> 4.224 +19:23:58 step 90160 loss=3.0966 lr=9.34e-05 tok=11.817B dt=11883 wait=151 +19:25:13 step 90180 loss=3.0334 lr=9.34e-05 tok=11.820B dt=11958 wait=152 +19:26:28 step 90200 loss=3.1417 lr=9.34e-05 tok=11.823B dt=12034 wait=152 +19:27:43 step 90220 loss=3.1707 lr=9.34e-05 tok=11.825B dt=12109 wait=153 +19:28:59 step 90240 loss=3.0891 lr=9.33e-05 tok=11.828B dt=12184 wait=153 +19:30:14 step 90260 loss=3.1891 lr=9.33e-05 tok=11.831B dt=12259 wait=154 +19:31:29 step 90280 loss=3.0832 lr=9.33e-05 tok=11.833B dt=12334 wait=154 +19:32:44 step 90300 loss=3.1198 lr=9.33e-05 tok=11.836B dt=12410 wait=155 +19:32:56 pushed ckpt_w53.pt,log_w53.txt +19:34:42 waited 106s for global r=242 +19:34:50 global r=242 load 4.244 -> 4.224 +19:36:12 step 90320 loss=3.0919 lr=9.32e-05 tok=11.838B dt=12618 wait=155 +19:37:27 step 90340 loss=3.0980 lr=9.32e-05 tok=11.841B dt=12693 wait=156 +19:38:43 step 90360 loss=3.1076 lr=9.32e-05 tok=11.844B dt=12768 wait=156 +19:39:58 step 90380 loss=3.1360 lr=9.32e-05 tok=11.846B dt=12843 wait=156 +19:41:13 step 90400 loss=3.1677 lr=9.32e-05 tok=11.849B dt=12918 wait=157 +19:42:28 step 90420 loss=3.0864 lr=9.31e-05 tok=11.852B dt=12994 wait=157 +19:43:43 step 90440 loss=3.1455 lr=9.31e-05 tok=11.854B dt=13069 wait=158 +19:44:34 pushed ckpt_w53.pt,log_w53.txt +19:48:22 step 90460 loss=3.1279 lr=9.31e-05 tok=11.857B dt=13347 wait=158 +19:49:37 step 90480 loss=3.1356 lr=9.31e-05 tok=11.859B dt=13423 wait=159 +19:50:52 step 90500 loss=3.0710 lr=9.31e-05 tok=11.862B dt=13498 wait=159 +19:52:07 step 90520 loss=2.9999 lr=9.30e-05 tok=11.865B dt=13573 wait=160 +19:53:22 step 90540 loss=3.0776 lr=9.30e-05 tok=11.867B dt=13648 wait=160 +19:54:37 step 90560 loss=3.0896 lr=9.30e-05 tok=11.870B dt=13723 wait=161 +19:55:53 step 90580 loss=3.1527 lr=9.30e-05 tok=11.873B dt=13798 wait=161 +19:57:08 step 90600 loss=3.1358 lr=9.30e-05 tok=11.875B dt=13874 wait=161 +19:57:20 pushed ckpt_w53.pt,log_w53.txt +19:57:21 waited 0s for global r=243 +19:57:27 global r=243 load 4.256 -> 4.225 +19:58:49 step 90620 loss=3.1323 lr=9.29e-05 tok=11.878B dt=13974 wait=162 +20:00:04 step 90640 loss=3.1431 lr=9.29e-05 tok=11.880B dt=14050 wait=162 +20:01:20 step 90660 loss=3.1765 lr=9.29e-05 tok=11.883B dt=14125 wait=163 +launch +20:02:28 name=w53 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:02:33 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:02:43 params 1.08B +20:02:47 resumed step=90600 seen=11.88B round=243 +20:02:47 stream salt=90600 +20:04:45 start: accum=4, 131072 tokens per optimiser step +20:06:04 step 90601 loss=2.7732 lr=9.30e-05 tok=11.875B dt=79 wait=61 +20:06:09 step 90602 loss=3.0349 lr=9.30e-05 tok=11.875B dt=84 wait=61 +20:07:39 step 90620 loss=3.2267 lr=9.29e-05 tok=11.878B dt=173 wait=62 +20:09:04 step 90640 loss=3.3600 lr=9.29e-05 tok=11.880B dt=258 wait=62 +20:10:20 step 90660 loss=3.2923 lr=9.29e-05 tok=11.883B dt=334 wait=63 +20:11:35 step 90680 loss=3.3649 lr=9.29e-05 tok=11.886B dt=410 wait=63 +20:12:51 step 90700 loss=3.2304 lr=9.29e-05 tok=11.888B dt=485 wait=63 +20:14:06 step 90720 loss=3.2847 lr=9.28e-05 tok=11.891B dt=561 wait=64 +20:15:22 step 90740 loss=3.2966 lr=9.28e-05 tok=11.893B dt=637 wait=64 +20:16:15 pushed ckpt_w53.pt,log_w53.txt +20:17:00 waited 46s for global r=244 +20:17:14 global r=244 load 4.243 -> 4.228 +20:17:57 step 90760 loss=3.2723 lr=9.28e-05 tok=11.896B dt=792 wait=65 +20:19:13 step 90780 loss=3.2180 lr=9.28e-05 tok=11.899B dt=867 wait=65 +20:20:28 step 90800 loss=3.2632 lr=9.27e-05 tok=11.901B dt=943 wait=66 +20:21:44 step 90820 loss=3.2529 lr=9.27e-05 tok=11.904B dt=1018 wait=66 +20:22:59 step 90840 loss=3.4026 lr=9.27e-05 tok=11.907B dt=1094 wait=67 +20:24:15 step 90860 loss=3.3811 lr=9.27e-05 tok=11.909B dt=1170 wait=67 +20:25:31 step 90880 loss=3.3794 lr=9.27e-05 tok=11.912B dt=1246 wait=68 +20:26:47 step 90900 loss=3.2814 lr=9.26e-05 tok=11.914B dt=1321 wait=68 +20:27:06 pushed ckpt_w53.pt,log_w53.txt +20:28:52 waited 106s for global r=245 +20:29:01 global r=245 load 4.246 -> 4.225 +20:30:25 step 90920 loss=3.2523 lr=9.26e-05 tok=11.917B dt=1539 wait=68 +20:31:40 step 90940 loss=3.1965 lr=9.26e-05 tok=11.920B dt=1614 wait=69 +20:32:55 step 90960 loss=3.2249 lr=9.26e-05 tok=11.922B dt=1690 wait=69 +20:34:11 step 90980 loss=3.3016 lr=9.26e-05 tok=11.925B dt=1766 wait=70 +20:35:27 step 91000 loss=3.4092 lr=9.25e-05 tok=11.928B dt=1841 wait=70 +20:36:42 step 91020 loss=3.3596 lr=9.25e-05 tok=11.930B dt=1917 wait=71 +20:37:58 step 91040 loss=3.3084 lr=9.25e-05 tok=11.933B dt=1992 wait=71 +20:38:49 pushed ckpt_w53.pt,log_w53.txt +20:41:20 waited 151s for global r=246 +20:41:48 global r=246 load 4.240 -> 4.220 +20:42:30 step 91060 loss=3.3777 lr=9.25e-05 tok=11.935B dt=2264 wait=72 +20:43:45 step 91080 loss=3.2321 lr=9.25e-05 tok=11.938B dt=2340 wait=72 +20:45:00 step 91100 loss=3.2678 lr=9.24e-05 tok=11.941B dt=2415 wait=73 +20:46:16 step 91120 loss=3.2567 lr=9.24e-05 tok=11.943B dt=2491 wait=73 +20:47:32 step 91140 loss=3.3411 lr=9.24e-05 tok=11.946B dt=2566 wait=74 +20:48:47 step 91160 loss=3.3160 lr=9.24e-05 tok=11.949B dt=2642 wait=74 +20:50:03 step 91180 loss=3.3730 lr=9.23e-05 tok=11.951B dt=2717 wait=75 +20:51:18 step 91200 loss=3.3143 lr=9.23e-05 tok=11.954B dt=2793 wait=75 +20:51:30 pushed ckpt_w53.pt,log_w53.txt +20:54:02 waited 151s for global r=247 +20:54:16 global r=247 load 4.243 -> 4.224 +20:55:36 step 91220 loss=3.3124 lr=9.23e-05 tok=11.956B dt=3051 wait=75 +20:56:52 step 91240 loss=3.2800 lr=9.23e-05 tok=11.959B dt=3126 wait=76 +20:58:07 step 91260 loss=3.2850 lr=9.23e-05 tok=11.962B dt=3202 wait=76 +20:59:23 step 91280 loss=3.2951 lr=9.22e-05 tok=11.964B dt=3278 wait=77 +21:00:38 step 91300 loss=3.3544 lr=9.22e-05 tok=11.967B dt=3353 wait=77 +21:01:54 step 91320 loss=3.2984 lr=9.22e-05 tok=11.969B dt=3429 wait=78 +21:03:09 step 91340 loss=3.2735 lr=9.22e-05 tok=11.972B dt=3504 wait=78 +21:04:02 pushed ckpt_w53.pt,log_w53.txt +21:05:33 waited 91s for global r=248 +21:05:44 global r=248 load 4.242 -> 4.227 +21:06:30 step 91360 loss=3.3435 lr=9.22e-05 tok=11.975B dt=3705 wait=79 +21:07:46 step 91380 loss=3.2829 lr=9.21e-05 tok=11.977B dt=3781 wait=79 +21:09:01 step 91400 loss=3.2932 lr=9.21e-05 tok=11.980B dt=3856 wait=80 +21:10:17 step 91420 loss=3.2719 lr=9.21e-05 tok=11.983B dt=3932 wait=80 +21:11:33 step 91440 loss=3.3286 lr=9.21e-05 tok=11.985B dt=4007 wait=80 +21:12:48 step 91460 loss=3.3375 lr=9.21e-05 tok=11.988B dt=4083 wait=81 +21:14:04 step 91480 loss=3.3637 lr=9.20e-05 tok=11.990B dt=4158 wait=81 +21:15:19 step 91500 loss=3.3071 lr=9.20e-05 tok=11.993B dt=4234 wait=82 +21:15:31 pushed ckpt_w53.pt,log_w53.txt +21:17:32 waited 121s for global r=249 +21:17:40 global r=249 load 4.248 -> 4.226 +21:19:00 step 91520 loss=3.2920 lr=9.20e-05 tok=11.996B dt=4455 wait=82 +21:20:16 step 91540 loss=3.2246 lr=9.20e-05 tok=11.998B dt=4530 wait=83 +21:21:31 step 91560 loss=3.2903 lr=9.19e-05 tok=12.001B dt=4606 wait=83 +21:22:47 step 91580 loss=3.3056 lr=9.19e-05 tok=12.004B dt=4681 wait=84 +21:24:02 step 91600 loss=3.3988 lr=9.19e-05 tok=12.006B dt=4757 wait=84 +21:25:18 step 91620 loss=3.3031 lr=9.19e-05 tok=12.009B dt=4832 wait=85 +21:26:33 step 91640 loss=3.2749 lr=9.19e-05 tok=12.011B dt=4908 wait=85 diff --git a/log_w54.txt b/log_w54.txt new file mode 100644 index 0000000000000000000000000000000000000000..fc6d1c9a6715eb5bdc8995dcc8ee24fc8448598b --- /dev/null +++ b/log_w54.txt @@ -0,0 +1,461 @@ +launch +16:03:03 name=w54 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 3/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:03:09 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:03:20 params 1.08B +16:03:22 resumed step=87600 seen=11.48B round=229 +16:03:22 stream salt=87600 +16:06:04 start: accum=4, 131072 tokens per optimiser step +16:08:08 step 87601 loss=3.1281 lr=9.61e-05 tok=11.482B dt=124 wait=93 +16:08:13 step 87602 loss=3.0120 lr=9.61e-05 tok=11.482B dt=129 wait=93 +16:09:40 step 87620 loss=3.2123 lr=9.61e-05 tok=11.485B dt=216 wait=93 +16:11:01 step 87640 loss=3.1790 lr=9.60e-05 tok=11.487B dt=298 wait=94 +16:12:17 step 87660 loss=3.2198 lr=9.60e-05 tok=11.490B dt=373 wait=94 +16:13:32 step 87680 loss=3.0949 lr=9.60e-05 tok=11.492B dt=448 wait=95 +16:14:47 step 87700 loss=3.1617 lr=9.60e-05 tok=11.495B dt=523 wait=95 +16:16:01 step 87720 loss=3.0745 lr=9.60e-05 tok=11.498B dt=597 wait=95 +16:17:16 step 87740 loss=3.0724 lr=9.59e-05 tok=11.500B dt=672 wait=96 +16:18:12 pushed ckpt_w54.pt,log_w54.txt +16:19:43 waited 91s for global r=230 +16:20:03 global r=230 load 4.249 -> 4.231 +16:20:48 step 87760 loss=3.1696 lr=9.59e-05 tok=11.503B dt=884 wait=96 +16:22:03 step 87780 loss=3.0527 lr=9.59e-05 tok=11.506B dt=959 wait=97 +16:23:18 step 87800 loss=3.1400 lr=9.59e-05 tok=11.508B dt=1034 wait=97 +16:24:33 step 87820 loss=3.1616 lr=9.59e-05 tok=11.511B dt=1109 wait=97 +16:25:48 step 87840 loss=3.1135 lr=9.58e-05 tok=11.513B dt=1184 wait=98 +16:27:03 step 87860 loss=3.1764 lr=9.58e-05 tok=11.516B dt=1259 wait=98 +16:28:18 step 87880 loss=3.0959 lr=9.58e-05 tok=11.519B dt=1334 wait=99 +16:29:33 step 87900 loss=3.1308 lr=9.58e-05 tok=11.521B dt=1409 wait=99 +16:29:45 pushed ckpt_w54.pt,log_w54.txt +16:31:15 waited 91s for global r=231 +16:31:25 global r=231 load 4.254 -> 4.232 +16:32:47 step 87920 loss=3.0890 lr=9.58e-05 tok=11.524B dt=1604 wait=99 +16:34:02 step 87940 loss=3.1039 lr=9.57e-05 tok=11.526B dt=1678 wait=100 +16:35:18 step 87960 loss=3.1114 lr=9.57e-05 tok=11.529B dt=1754 wait=100 +16:36:39 step 87980 loss=3.1586 lr=9.57e-05 tok=11.532B dt=1835 wait=100 +16:38:26 step 88000 loss=3.1392 lr=9.57e-05 tok=11.534B dt=1942 wait=101 +16:40:13 step 88020 loss=3.1223 lr=9.56e-05 tok=11.537B dt=2049 wait=101 +16:42:00 step 88040 loss=3.1342 lr=9.56e-05 tok=11.540B dt=2156 wait=101 +16:43:11 pushed ckpt_w54.pt,log_w54.txt +16:44:12 waited 61s for global r=232 +16:44:22 global r=232 load 4.246 -> 4.227 +16:45:23 step 88060 loss=3.0943 lr=9.56e-05 tok=11.542B dt=2359 wait=102 +16:47:10 step 88080 loss=3.0775 lr=9.56e-05 tok=11.545B dt=2466 wait=102 +16:48:57 step 88100 loss=3.1314 lr=9.56e-05 tok=11.547B dt=2573 wait=103 +16:50:39 step 88120 loss=3.1978 lr=9.55e-05 tok=11.550B dt=2675 wait=103 +16:52:26 step 88140 loss=3.1140 lr=9.55e-05 tok=11.553B dt=2782 wait=103 +16:54:13 step 88160 loss=3.1810 lr=9.55e-05 tok=11.555B dt=2889 wait=104 +16:55:59 step 88180 loss=3.0648 lr=9.55e-05 tok=11.558B dt=2995 wait=104 +16:57:47 step 88200 loss=3.0960 lr=9.55e-05 tok=11.561B dt=3103 wait=104 +16:58:01 pushed ckpt_w54.pt,log_w54.txt +16:59:02 waited 61s for global r=233 +16:59:11 global r=233 load 4.245 -> 4.228 +17:00:32 step 88220 loss=3.0738 lr=9.54e-05 tok=11.563B dt=3268 wait=105 +17:01:47 step 88240 loss=3.1118 lr=9.54e-05 tok=11.566B dt=3343 wait=105 +17:03:02 step 88260 loss=3.1111 lr=9.54e-05 tok=11.568B dt=3418 wait=106 +17:04:17 step 88280 loss=3.1728 lr=9.54e-05 tok=11.571B dt=3493 wait=106 +17:05:32 step 88300 loss=3.1668 lr=9.54e-05 tok=11.574B dt=3568 wait=106 +17:07:09 step 88320 loss=3.1215 lr=9.53e-05 tok=11.576B dt=3665 wait=107 +17:08:56 step 88340 loss=3.1427 lr=9.53e-05 tok=11.579B dt=3772 wait=107 +17:10:04 pushed ckpt_w54.pt,log_w54.txt +17:13:50 step 88360 loss=3.1174 lr=9.53e-05 tok=11.582B dt=4066 wait=107 +17:15:04 step 88380 loss=3.1568 lr=9.53e-05 tok=11.584B dt=4140 wait=108 +17:16:19 step 88400 loss=3.0384 lr=9.53e-05 tok=11.587B dt=4215 wait=108 +17:17:34 step 88420 loss=2.9953 lr=9.52e-05 tok=11.589B dt=4290 wait=109 +17:18:49 step 88440 loss=3.0657 lr=9.52e-05 tok=11.592B dt=4365 wait=109 +17:20:04 step 88460 loss=3.1276 lr=9.52e-05 tok=11.595B dt=4440 wait=109 +17:21:19 step 88480 loss=3.1583 lr=9.52e-05 tok=11.597B dt=4515 wait=110 +17:22:34 step 88500 loss=3.1325 lr=9.51e-05 tok=11.600B dt=4590 wait=110 +17:22:50 pushed ckpt_w54.pt,log_w54.txt +17:22:50 waited 0s for global r=234 +17:23:02 global r=234 load 4.260 -> 4.230 +17:24:22 step 88520 loss=3.1090 lr=9.51e-05 tok=11.602B dt=4698 wait=111 +17:25:37 step 88540 loss=3.1273 lr=9.51e-05 tok=11.605B dt=4773 wait=111 +17:26:52 step 88560 loss=3.0827 lr=9.51e-05 tok=11.608B dt=4848 wait=111 +17:28:07 step 88580 loss=3.1310 lr=9.51e-05 tok=11.610B dt=4923 wait=112 +17:29:22 step 88600 loss=3.1569 lr=9.50e-05 tok=11.613B dt=4998 wait=112 +17:30:37 step 88620 loss=3.1303 lr=9.50e-05 tok=11.616B dt=5073 wait=112 +17:31:52 step 88640 loss=3.1782 lr=9.50e-05 tok=11.618B dt=5148 wait=113 +17:32:41 pushed ckpt_w54.pt,log_w54.txt +17:33:57 waited 76s for global r=235 +17:34:10 global r=235 load 4.251 -> 4.231 +17:35:10 step 88660 loss=3.1407 lr=9.50e-05 tok=11.621B dt=5346 wait=113 +17:36:57 step 88680 loss=3.1006 lr=9.50e-05 tok=11.623B dt=5453 wait=114 +17:38:45 step 88700 loss=3.1307 lr=9.49e-05 tok=11.626B dt=5561 wait=114 +17:40:32 step 88720 loss=3.1740 lr=9.49e-05 tok=11.629B dt=5668 wait=114 +17:41:48 step 88740 loss=3.0927 lr=9.49e-05 tok=11.631B dt=5744 wait=115 +17:43:26 step 88760 loss=3.1576 lr=9.49e-05 tok=11.634B dt=5842 wait=115 +17:45:13 step 88780 loss=3.0362 lr=9.49e-05 tok=11.637B dt=5949 wait=115 +17:47:01 step 88800 loss=3.1212 lr=9.48e-05 tok=11.639B dt=6057 wait=116 +17:47:13 pushed ckpt_w54.pt,log_w54.txt +17:51:34 step 88820 loss=3.1818 lr=9.48e-05 tok=11.642B dt=6330 wait=116 +17:52:48 step 88840 loss=3.0682 lr=9.48e-05 tok=11.644B dt=6404 wait=117 +17:54:03 step 88860 loss=3.1391 lr=9.48e-05 tok=11.647B dt=6479 wait=117 +17:55:18 step 88880 loss=3.1266 lr=9.48e-05 tok=11.650B dt=6554 wait=117 +17:56:33 step 88900 loss=3.0951 lr=9.47e-05 tok=11.652B dt=6629 wait=118 +17:57:48 step 88920 loss=3.1814 lr=9.47e-05 tok=11.655B dt=6704 wait=118 +17:59:16 step 88940 loss=3.0977 lr=9.47e-05 tok=11.658B dt=6792 wait=118 +18:00:25 pushed ckpt_w54.pt,log_w54.txt +18:00:25 waited 0s for global r=236 +18:00:36 global r=236 load 4.241 -> 4.225 +18:01:38 step 88960 loss=3.0871 lr=9.47e-05 tok=11.660B dt=6934 wait=119 +18:03:27 step 88980 loss=3.0826 lr=9.46e-05 tok=11.663B dt=7043 wait=119 +18:05:14 step 89000 loss=3.1742 lr=9.46e-05 tok=11.665B dt=7150 wait=120 +18:06:29 step 89020 loss=3.2058 lr=9.46e-05 tok=11.668B dt=7225 wait=120 +18:08:13 step 89040 loss=3.1303 lr=9.46e-05 tok=11.671B dt=7329 wait=120 +18:10:00 step 89060 loss=3.1996 lr=9.46e-05 tok=11.673B dt=7436 wait=121 +18:11:48 step 89080 loss=3.0643 lr=9.45e-05 tok=11.676B dt=7544 wait=121 +18:13:36 step 89100 loss=3.0989 lr=9.45e-05 tok=11.679B dt=7652 wait=121 +18:13:48 pushed ckpt_w54.pt,log_w54.txt +18:13:48 waited 0s for global r=237 +18:13:58 global r=237 load 4.241 -> 4.232 +18:15:20 step 89120 loss=3.0389 lr=9.45e-05 tok=11.681B dt=7756 wait=122 +18:16:35 step 89140 loss=3.1304 lr=9.45e-05 tok=11.684B dt=7831 wait=122 +18:17:50 step 89160 loss=3.1180 lr=9.45e-05 tok=11.686B dt=7906 wait=123 +18:19:05 step 89180 loss=3.1730 lr=9.44e-05 tok=11.689B dt=7981 wait=123 +18:20:20 step 89200 loss=3.1781 lr=9.44e-05 tok=11.692B dt=8056 wait=123 +18:21:35 step 89220 loss=3.0991 lr=9.44e-05 tok=11.694B dt=8131 wait=124 +18:22:50 step 89240 loss=3.1039 lr=9.44e-05 tok=11.697B dt=8206 wait=124 +18:23:42 pushed ckpt_w54.pt,log_w54.txt +18:26:28 waited 166s for global r=238 +18:26:38 global r=238 load 4.248 -> 4.231 +18:27:23 step 89260 loss=3.0431 lr=9.44e-05 tok=11.699B dt=8479 wait=124 +18:28:38 step 89280 loss=3.0526 lr=9.43e-05 tok=11.702B dt=8554 wait=125 +18:29:53 step 89300 loss=3.1292 lr=9.43e-05 tok=11.705B dt=8629 wait=125 +18:31:42 step 89320 loss=3.1905 lr=9.43e-05 tok=11.707B dt=8738 wait=126 +18:34:05 step 89340 loss=3.1186 lr=9.43e-05 tok=11.710B dt=8881 wait=126 +18:36:23 step 89360 loss=3.1901 lr=9.43e-05 tok=11.713B dt=9019 wait=126 +18:39:03 step 89380 loss=3.0589 lr=9.42e-05 tok=11.715B dt=9179 wait=127 +18:40:32 step 89400 loss=3.1486 lr=9.42e-05 tok=11.718B dt=9268 wait=127 +18:40:44 pushed ckpt_w54.pt,log_w54.txt +18:41:44 waited 61s for global r=239 +18:41:54 global r=239 load 4.246 -> 4.230 +18:43:21 step 89420 loss=3.1317 lr=9.42e-05 tok=11.720B dt=9437 wait=128 +18:44:36 step 89440 loss=3.1462 lr=9.42e-05 tok=11.723B dt=9512 wait=128 +18:45:51 step 89460 loss=3.0854 lr=9.41e-05 tok=11.726B dt=9587 wait=128 +18:47:06 step 89480 loss=3.1626 lr=9.41e-05 tok=11.728B dt=9662 wait=129 +18:48:21 step 89500 loss=3.1350 lr=9.41e-05 tok=11.731B dt=9737 wait=129 +18:49:36 step 89520 loss=3.0864 lr=9.41e-05 tok=11.734B dt=9812 wait=129 +18:50:51 step 89540 loss=3.1270 lr=9.41e-05 tok=11.736B dt=9887 wait=130 +18:51:43 pushed ckpt_w54.pt,log_w54.txt +18:55:29 step 89560 loss=3.1168 lr=9.40e-05 tok=11.739B dt=10165 wait=130 +18:56:44 step 89580 loss=3.1741 lr=9.40e-05 tok=11.741B dt=10240 wait=131 +18:57:59 step 89600 loss=3.0503 lr=9.40e-05 tok=11.744B dt=10315 wait=131 +18:59:14 step 89620 loss=3.0216 lr=9.40e-05 tok=11.747B dt=10390 wait=131 +19:00:29 step 89640 loss=3.0908 lr=9.40e-05 tok=11.749B dt=10465 wait=132 +19:01:44 step 89660 loss=3.1185 lr=9.39e-05 tok=11.752B dt=10540 wait=132 +19:02:59 step 89680 loss=3.1386 lr=9.39e-05 tok=11.755B dt=10615 wait=133 +19:04:14 step 89700 loss=3.1325 lr=9.39e-05 tok=11.757B dt=10690 wait=133 +19:04:26 pushed ckpt_w54.pt,log_w54.txt +19:04:26 waited 0s for global r=240 +19:04:33 global r=240 load 4.246 -> 4.227 +19:05:54 step 89720 loss=3.0948 lr=9.39e-05 tok=11.760B dt=10790 wait=133 +19:07:09 step 89740 loss=3.1506 lr=9.39e-05 tok=11.762B dt=10865 wait=134 +19:08:24 step 89760 loss=3.1507 lr=9.38e-05 tok=11.765B dt=10940 wait=134 +19:09:39 step 89780 loss=3.1642 lr=9.38e-05 tok=11.768B dt=11015 wait=135 +19:10:54 step 89800 loss=3.1266 lr=9.38e-05 tok=11.770B dt=11090 wait=135 +19:12:08 step 89820 loss=3.1022 lr=9.38e-05 tok=11.773B dt=11164 wait=135 +19:13:23 step 89840 loss=3.1738 lr=9.38e-05 tok=11.776B dt=11239 wait=136 +19:14:13 pushed ckpt_w54.pt,log_w54.txt +19:15:29 waited 76s for global r=241 +19:15:38 global r=241 load 4.242 -> 4.224 +19:16:22 step 89860 loss=3.0802 lr=9.37e-05 tok=11.778B dt=11418 wait=136 +19:17:37 step 89880 loss=3.0419 lr=9.37e-05 tok=11.781B dt=11493 wait=136 +19:18:51 step 89900 loss=3.1340 lr=9.37e-05 tok=11.783B dt=11567 wait=137 +19:20:07 step 89920 loss=3.1746 lr=9.37e-05 tok=11.786B dt=11643 wait=137 +19:21:22 step 89940 loss=3.1271 lr=9.36e-05 tok=11.789B dt=11718 wait=138 +19:22:37 step 89960 loss=3.1921 lr=9.36e-05 tok=11.791B dt=11793 wait=138 +19:23:52 step 89980 loss=3.0880 lr=9.36e-05 tok=11.794B dt=11868 wait=138 +19:25:07 step 90000 loss=3.1096 lr=9.36e-05 tok=11.796B dt=11943 wait=139 +19:25:19 pushed ckpt_w54.pt,log_w54.txt +19:29:40 step 90020 loss=3.1355 lr=9.36e-05 tok=11.799B dt=12216 wait=139 +19:30:55 step 90040 loss=3.0294 lr=9.35e-05 tok=11.802B dt=12291 wait=140 +19:32:10 step 90060 loss=3.0684 lr=9.35e-05 tok=11.804B dt=12366 wait=140 +19:33:24 step 90080 loss=3.0993 lr=9.35e-05 tok=11.807B dt=12440 wait=140 +19:34:40 step 90100 loss=3.0596 lr=9.35e-05 tok=11.810B dt=12516 wait=141 +19:35:54 step 90120 loss=3.1559 lr=9.35e-05 tok=11.812B dt=12590 wait=141 +19:37:10 step 90140 loss=3.1303 lr=9.34e-05 tok=11.815B dt=12666 wait=142 +19:37:59 pushed ckpt_w54.pt,log_w54.txt +19:37:59 waited 0s for global r=242 +19:38:08 global r=242 load 4.249 -> 4.224 +19:38:57 step 90160 loss=3.0901 lr=9.34e-05 tok=11.817B dt=12773 wait=142 +19:40:12 step 90180 loss=3.1123 lr=9.34e-05 tok=11.820B dt=12848 wait=142 +19:41:27 step 90200 loss=3.1550 lr=9.34e-05 tok=11.823B dt=12923 wait=143 +19:42:42 step 90220 loss=3.1821 lr=9.34e-05 tok=11.825B dt=12998 wait=143 +19:43:57 step 90240 loss=3.0743 lr=9.33e-05 tok=11.828B dt=13073 wait=143 +19:45:12 step 90260 loss=3.1530 lr=9.33e-05 tok=11.831B dt=13148 wait=144 +19:46:27 step 90280 loss=3.0325 lr=9.33e-05 tok=11.833B dt=13223 wait=144 +19:47:42 step 90300 loss=3.1149 lr=9.33e-05 tok=11.836B dt=13298 wait=145 +19:47:54 pushed ckpt_w54.pt,log_w54.txt +19:49:09 waited 76s for global r=243 +19:49:16 global r=243 load 4.246 -> 4.225 +19:50:37 step 90320 loss=3.1136 lr=9.32e-05 tok=11.838B dt=13473 wait=145 +19:51:52 step 90340 loss=3.1721 lr=9.32e-05 tok=11.841B dt=13548 wait=145 +19:53:07 step 90360 loss=3.1239 lr=9.32e-05 tok=11.844B dt=13623 wait=146 +19:54:22 step 90380 loss=3.1358 lr=9.32e-05 tok=11.846B dt=13698 wait=146 +19:55:37 step 90400 loss=3.1517 lr=9.32e-05 tok=11.849B dt=13773 wait=147 +19:56:52 step 90420 loss=3.1062 lr=9.31e-05 tok=11.852B dt=13848 wait=147 +19:58:07 step 90440 loss=3.1412 lr=9.31e-05 tok=11.854B dt=13923 wait=147 +19:59:01 pushed ckpt_w54.pt,log_w54.txt +launch +20:02:47 name=w54 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 3/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:02:52 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:03:02 params 1.08B +20:03:06 resumed step=90450 seen=11.86B round=243 +20:03:06 stream salt=90450 +20:05:14 start: accum=4, 131072 tokens per optimiser step +20:06:29 step 90451 loss=2.9604 lr=9.31e-05 tok=11.856B dt=75 wait=57 +20:06:34 step 90452 loss=3.1948 lr=9.31e-05 tok=11.856B dt=80 wait=57 +20:07:12 step 90460 loss=3.3267 lr=9.31e-05 tok=11.857B dt=118 wait=57 +20:08:47 step 90480 loss=3.2690 lr=9.31e-05 tok=11.859B dt=213 wait=58 +20:10:03 step 90500 loss=3.4179 lr=9.31e-05 tok=11.862B dt=289 wait=58 +20:11:19 step 90520 loss=3.3519 lr=9.30e-05 tok=11.865B dt=365 wait=58 +20:12:34 step 90540 loss=3.3054 lr=9.30e-05 tok=11.867B dt=440 wait=59 +20:13:50 step 90560 loss=3.2478 lr=9.30e-05 tok=11.870B dt=516 wait=59 +20:15:05 step 90580 loss=3.2621 lr=9.30e-05 tok=11.873B dt=591 wait=60 +20:16:20 step 90600 loss=3.3276 lr=9.30e-05 tok=11.875B dt=666 wait=60 +20:16:35 pushed ckpt_w54.pt,log_w54.txt +20:17:05 waited 30s for global r=244 +20:17:21 global r=244 load 4.242 -> 4.228 +20:18:43 step 90620 loss=3.2589 lr=9.29e-05 tok=11.878B dt=809 wait=60 +20:19:58 step 90640 loss=3.2380 lr=9.29e-05 tok=11.880B dt=884 wait=61 +20:21:14 step 90660 loss=3.2357 lr=9.29e-05 tok=11.883B dt=960 wait=61 +20:22:29 step 90680 loss=3.2771 lr=9.29e-05 tok=11.886B dt=1035 wait=62 +20:23:44 step 90700 loss=3.3968 lr=9.29e-05 tok=11.888B dt=1110 wait=62 +20:25:00 step 90720 loss=3.4081 lr=9.28e-05 tok=11.891B dt=1186 wait=62 +20:26:15 step 90740 loss=3.2676 lr=9.28e-05 tok=11.893B dt=1261 wait=63 +20:27:05 pushed ckpt_w54.pt,log_w54.txt +20:28:51 waited 106s for global r=245 +20:29:01 global r=245 load 4.244 -> 4.225 +20:29:47 step 90760 loss=3.3143 lr=9.28e-05 tok=11.896B dt=1473 wait=63 +20:31:02 step 90780 loss=3.2168 lr=9.28e-05 tok=11.899B dt=1548 wait=64 +20:32:17 step 90800 loss=3.2484 lr=9.27e-05 tok=11.901B dt=1623 wait=64 +20:33:32 step 90820 loss=3.2480 lr=9.27e-05 tok=11.904B dt=1698 wait=64 +20:34:48 step 90840 loss=3.3721 lr=9.27e-05 tok=11.907B dt=1774 wait=65 +20:36:03 step 90860 loss=3.3655 lr=9.27e-05 tok=11.909B dt=1849 wait=65 +20:37:19 step 90880 loss=3.3997 lr=9.27e-05 tok=11.912B dt=1925 wait=66 +20:38:34 step 90900 loss=3.3315 lr=9.26e-05 tok=11.914B dt=2000 wait=66 +20:38:46 pushed ckpt_w54.pt,log_w54.txt +20:41:17 waited 151s for global r=246 +20:41:39 global r=246 load 4.239 -> 4.220 +20:42:59 step 90920 loss=3.2913 lr=9.26e-05 tok=11.917B dt=2265 wait=66 +20:44:14 step 90940 loss=3.2215 lr=9.26e-05 tok=11.920B dt=2340 wait=67 +20:45:29 step 90960 loss=3.2618 lr=9.26e-05 tok=11.922B dt=2415 wait=67 +20:46:44 step 90980 loss=3.2852 lr=9.26e-05 tok=11.925B dt=2490 wait=67 +20:48:00 step 91000 loss=3.3662 lr=9.25e-05 tok=11.928B dt=2566 wait=68 +20:49:15 step 91020 loss=3.3574 lr=9.25e-05 tok=11.930B dt=2641 wait=68 +20:50:30 step 91040 loss=3.3079 lr=9.25e-05 tok=11.933B dt=2716 wait=69 +20:51:22 pushed ckpt_w54.pt,log_w54.txt +20:53:53 waited 151s for global r=247 +20:54:05 global r=247 load 4.242 -> 4.224 +20:54:50 step 91060 loss=3.3824 lr=9.25e-05 tok=11.935B dt=2976 wait=69 +20:56:06 step 91080 loss=3.2412 lr=9.25e-05 tok=11.938B dt=3052 wait=69 +20:57:21 step 91100 loss=3.2839 lr=9.24e-05 tok=11.941B dt=3127 wait=70 +20:58:36 step 91120 loss=3.2605 lr=9.24e-05 tok=11.943B dt=3202 wait=70 +20:59:51 step 91140 loss=3.3419 lr=9.24e-05 tok=11.946B dt=3277 wait=71 +21:01:07 step 91160 loss=3.3331 lr=9.24e-05 tok=11.949B dt=3353 wait=71 +21:02:22 step 91180 loss=3.3495 lr=9.23e-05 tok=11.951B dt=3428 wait=71 +21:03:37 step 91200 loss=3.3052 lr=9.23e-05 tok=11.954B dt=3503 wait=72 +21:03:51 pushed ckpt_w54.pt,log_w54.txt +21:05:37 waited 106s for global r=248 +21:05:48 global r=248 load 4.257 -> 4.227 +21:07:07 step 91220 loss=3.3087 lr=9.23e-05 tok=11.956B dt=3713 wait=72 +21:08:23 step 91240 loss=3.3109 lr=9.23e-05 tok=11.959B dt=3789 wait=73 +21:09:38 step 91260 loss=3.3043 lr=9.23e-05 tok=11.962B dt=3864 wait=73 +21:10:54 step 91280 loss=3.3363 lr=9.22e-05 tok=11.964B dt=3940 wait=73 +21:12:09 step 91300 loss=3.3728 lr=9.22e-05 tok=11.967B dt=4015 wait=74 +21:13:24 step 91320 loss=3.3121 lr=9.22e-05 tok=11.969B dt=4090 wait=74 +21:14:39 step 91340 loss=3.2499 lr=9.22e-05 tok=11.972B dt=4165 wait=74 +21:15:29 pushed ckpt_w54.pt,log_w54.txt +21:17:29 waited 121s for global r=249 +21:17:37 global r=249 load 4.247 -> 4.226 +21:18:22 step 91360 loss=3.3272 lr=9.22e-05 tok=11.975B dt=4389 wait=75 +21:19:38 step 91380 loss=3.2428 lr=9.21e-05 tok=11.977B dt=4464 wait=75 +21:20:53 step 91400 loss=3.2661 lr=9.21e-05 tok=11.980B dt=4539 wait=76 +21:22:09 step 91420 loss=3.2874 lr=9.21e-05 tok=11.983B dt=4615 wait=76 +21:23:24 step 91440 loss=3.3714 lr=9.21e-05 tok=11.985B dt=4690 wait=76 +21:24:39 step 91460 loss=3.3384 lr=9.21e-05 tok=11.988B dt=4765 wait=77 +21:25:55 step 91480 loss=3.3724 lr=9.20e-05 tok=11.990B dt=4841 wait=77 +21:27:10 step 91500 loss=3.2902 lr=9.20e-05 tok=11.993B dt=4916 wait=77 +21:27:22 pushed ckpt_w54.pt,log_w54.txt +21:31:46 step 91520 loss=3.3737 lr=9.20e-05 tok=11.996B dt=5192 wait=78 +21:33:01 step 91540 loss=3.4150 lr=9.20e-05 tok=11.998B dt=5267 wait=78 +21:34:17 step 91560 loss=3.4530 lr=9.19e-05 tok=12.001B dt=5343 wait=79 +21:35:32 step 91580 loss=3.3746 lr=9.19e-05 tok=12.004B dt=5418 wait=79 +21:36:48 step 91600 loss=3.3374 lr=9.19e-05 tok=12.006B dt=5494 wait=79 +21:38:03 step 91620 loss=3.3325 lr=9.19e-05 tok=12.009B dt=5569 wait=80 +21:39:18 step 91640 loss=3.2997 lr=9.19e-05 tok=12.011B dt=5644 wait=80 +21:40:10 pushed ckpt_w54.pt,log_w54.txt +21:43:54 step 91660 loss=3.2430 lr=9.18e-05 tok=12.014B dt=5920 wait=81 +21:45:09 step 91680 loss=3.3171 lr=9.18e-05 tok=12.017B dt=5995 wait=81 +21:46:25 step 91700 loss=3.3050 lr=9.18e-05 tok=12.019B dt=6071 wait=81 +21:47:40 step 91720 loss=3.3238 lr=9.18e-05 tok=12.022B dt=6146 wait=82 +21:48:55 step 91740 loss=3.2779 lr=9.18e-05 tok=12.025B dt=6221 wait=82 +21:50:11 step 91760 loss=3.3109 lr=9.17e-05 tok=12.027B dt=6297 wait=83 +21:51:26 step 91780 loss=3.3339 lr=9.17e-05 tok=12.030B dt=6372 wait=83 +21:52:41 step 91800 loss=3.3705 lr=9.17e-05 tok=12.032B dt=6447 wait=83 +21:52:53 pushed ckpt_w54.pt,log_w54.txt +21:57:14 step 91820 loss=3.3587 lr=9.17e-05 tok=12.035B dt=6720 wait=84 +21:58:29 step 91840 loss=3.3432 lr=9.17e-05 tok=12.038B dt=6795 wait=84 +21:59:44 step 91860 loss=3.2257 lr=9.16e-05 tok=12.040B dt=6870 wait=84 +22:01:00 step 91880 loss=3.3240 lr=9.16e-05 tok=12.043B dt=6946 wait=85 +22:02:15 step 91900 loss=3.3379 lr=9.16e-05 tok=12.046B dt=7021 wait=85 +22:03:31 step 91920 loss=3.2368 lr=9.16e-05 tok=12.048B dt=7097 wait=86 +22:04:46 step 91940 loss=3.2479 lr=9.15e-05 tok=12.051B dt=7172 wait=86 +22:06:00 pushed ckpt_w54.pt,log_w54.txt +22:06:15 waited 15s for global r=250 +22:06:23 global r=250 load 4.253 -> 4.238 +22:07:07 step 91960 loss=3.3114 lr=9.15e-05 tok=12.053B dt=7313 wait=86 +22:08:23 step 91980 loss=3.2612 lr=9.15e-05 tok=12.056B dt=7389 wait=87 +22:09:39 step 92000 loss=3.2992 lr=9.15e-05 tok=12.059B dt=7465 wait=87 +22:10:55 step 92020 loss=3.3090 lr=9.15e-05 tok=12.061B dt=7541 wait=88 +22:12:10 step 92040 loss=3.3872 lr=9.14e-05 tok=12.064B dt=7616 wait=88 +22:13:26 step 92060 loss=3.3286 lr=9.14e-05 tok=12.066B dt=7692 wait=89 +22:14:42 step 92080 loss=3.3225 lr=9.14e-05 tok=12.069B dt=7768 wait=89 +22:15:57 step 92100 loss=3.2545 lr=9.14e-05 tok=12.072B dt=7844 wait=90 +22:16:16 pushed ckpt_w54.pt,log_w54.txt +22:20:38 step 92120 loss=3.3534 lr=9.14e-05 tok=12.074B dt=8124 wait=90 +22:21:54 step 92140 loss=3.4212 lr=9.13e-05 tok=12.077B dt=8200 wait=91 +22:23:10 step 92160 loss=3.4913 lr=9.13e-05 tok=12.080B dt=8276 wait=91 +22:24:25 step 92180 loss=3.4191 lr=9.13e-05 tok=12.082B dt=8351 wait=91 +22:25:41 step 92200 loss=3.3358 lr=9.13e-05 tok=12.085B dt=8427 wait=92 +22:26:56 step 92220 loss=3.3305 lr=9.13e-05 tok=12.087B dt=8502 wait=92 +22:28:12 step 92240 loss=3.2715 lr=9.12e-05 tok=12.090B dt=8578 wait=93 +22:29:12 pushed ckpt_w54.pt,log_w54.txt +22:29:12 waited 0s for global r=251 +22:29:19 global r=251 load 4.247 -> 4.237 +22:30:05 step 92260 loss=3.3097 lr=9.12e-05 tok=12.093B dt=8691 wait=93 +22:31:20 step 92280 loss=3.2431 lr=9.12e-05 tok=12.095B dt=8766 wait=94 +22:32:36 step 92300 loss=3.2813 lr=9.12e-05 tok=12.098B dt=8842 wait=94 +22:33:52 step 92320 loss=3.2845 lr=9.12e-05 tok=12.101B dt=8918 wait=95 +22:35:07 step 92340 loss=3.3744 lr=9.11e-05 tok=12.103B dt=8993 wait=95 +22:36:23 step 92360 loss=3.2952 lr=9.11e-05 tok=12.106B dt=9069 wait=95 +22:37:39 step 92380 loss=3.3742 lr=9.11e-05 tok=12.108B dt=9145 wait=96 +22:38:54 step 92400 loss=3.2717 lr=9.11e-05 tok=12.111B dt=9221 wait=96 +22:39:09 pushed ckpt_w54.pt,log_w54.txt +22:40:24 waited 76s for global r=252 +22:40:33 global r=252 load 4.243 -> 4.228 +22:41:56 step 92420 loss=3.2946 lr=9.10e-05 tok=12.114B dt=9402 wait=97 +22:43:11 step 92440 loss=3.2407 lr=9.10e-05 tok=12.116B dt=9477 wait=97 +22:44:27 step 92460 loss=3.2471 lr=9.10e-05 tok=12.119B dt=9553 wait=98 +22:45:42 step 92480 loss=3.3215 lr=9.10e-05 tok=12.122B dt=9628 wait=98 +22:46:58 step 92500 loss=3.3894 lr=9.10e-05 tok=12.124B dt=9704 wait=99 +22:48:14 step 92520 loss=3.3262 lr=9.09e-05 tok=12.127B dt=9780 wait=99 +22:49:30 step 92540 loss=3.2938 lr=9.09e-05 tok=12.129B dt=9856 wait=100 +22:50:25 pushed ckpt_w54.pt,log_w54.txt +22:54:12 step 92560 loss=3.4199 lr=9.09e-05 tok=12.132B dt=10138 wait=100 +22:55:28 step 92580 loss=3.4903 lr=9.09e-05 tok=12.135B dt=10214 wait=101 +22:56:44 step 92600 loss=3.4306 lr=9.09e-05 tok=12.137B dt=10290 wait=101 +22:58:00 step 92620 loss=3.3747 lr=9.08e-05 tok=12.140B dt=10366 wait=102 +22:59:16 step 92640 loss=3.4114 lr=9.08e-05 tok=12.143B dt=10442 wait=102 +23:00:32 step 92660 loss=3.2937 lr=9.08e-05 tok=12.145B dt=10518 wait=102 +23:01:48 step 92680 loss=3.3238 lr=9.08e-05 tok=12.148B dt=10594 wait=103 +23:03:03 step 92700 loss=3.3103 lr=9.08e-05 tok=12.150B dt=10669 wait=103 +23:03:17 pushed ckpt_w54.pt,log_w54.txt +23:03:17 waited 0s for global r=253 +23:03:28 global r=253 load 4.243 -> 4.223 +23:04:49 step 92720 loss=3.2910 lr=9.07e-05 tok=12.153B dt=10776 wait=104 +23:06:05 step 92740 loss=3.2081 lr=9.07e-05 tok=12.156B dt=10851 wait=104 +23:07:21 step 92760 loss=3.2476 lr=9.07e-05 tok=12.158B dt=10927 wait=105 +23:08:37 step 92780 loss=3.3025 lr=9.07e-05 tok=12.161B dt=11003 wait=105 +23:09:53 step 92800 loss=3.3763 lr=9.06e-05 tok=12.163B dt=11079 wait=106 +23:11:09 step 92820 loss=3.3136 lr=9.06e-05 tok=12.166B dt=11155 wait=106 +23:12:25 step 92840 loss=3.2810 lr=9.06e-05 tok=12.169B dt=11231 wait=107 +23:13:19 pushed ckpt_w54.pt,log_w54.txt +23:14:34 waited 76s for global r=254 +23:14:42 global r=254 load 4.237 -> 4.223 +23:15:25 step 92860 loss=3.3380 lr=9.06e-05 tok=12.171B dt=11411 wait=107 +23:16:40 step 92880 loss=3.2306 lr=9.06e-05 tok=12.174B dt=11486 wait=108 +23:17:56 step 92900 loss=3.2890 lr=9.05e-05 tok=12.177B dt=11562 wait=108 +23:19:12 step 92920 loss=3.2913 lr=9.05e-05 tok=12.179B dt=11638 wait=109 +23:20:28 step 92940 loss=3.3761 lr=9.05e-05 tok=12.182B dt=11714 wait=109 +23:21:44 step 92960 loss=3.3466 lr=9.05e-05 tok=12.184B dt=11790 wait=110 +23:23:00 step 92980 loss=3.3583 lr=9.05e-05 tok=12.187B dt=11866 wait=110 +23:24:16 step 93000 loss=3.2813 lr=9.04e-05 tok=12.190B dt=11942 wait=110 +23:24:29 pushed ckpt_w54.pt,log_w54.txt +23:28:52 step 93020 loss=3.3790 lr=9.04e-05 tok=12.192B dt=12219 wait=111 +23:30:08 step 93040 loss=3.4546 lr=9.04e-05 tok=12.195B dt=12294 wait=111 +23:31:25 step 93060 loss=3.4924 lr=9.04e-05 tok=12.198B dt=12371 wait=112 +23:32:41 step 93080 loss=3.3567 lr=9.04e-05 tok=12.200B dt=12447 wait=112 +23:33:57 step 93100 loss=3.3146 lr=9.03e-05 tok=12.203B dt=12523 wait=113 +23:35:12 step 93120 loss=3.3327 lr=9.03e-05 tok=12.205B dt=12599 wait=113 +23:36:28 step 93140 loss=3.3008 lr=9.03e-05 tok=12.208B dt=12674 wait=114 +23:37:25 pushed ckpt_w54.pt,log_w54.txt +23:37:25 waited 0s for global r=255 +23:37:35 global r=255 load 4.247 -> 4.222 +23:38:20 step 93160 loss=3.3249 lr=9.03e-05 tok=12.211B dt=12786 wait=114 +23:39:36 step 93180 loss=3.2119 lr=9.02e-05 tok=12.213B dt=12862 wait=115 +23:40:52 step 93200 loss=3.2684 lr=9.02e-05 tok=12.216B dt=12938 wait=115 +23:42:07 step 93220 loss=3.2398 lr=9.02e-05 tok=12.219B dt=13013 wait=116 +23:43:23 step 93240 loss=3.3375 lr=9.02e-05 tok=12.221B dt=13089 wait=116 +23:44:39 step 93260 loss=3.3412 lr=9.02e-05 tok=12.224B dt=13165 wait=117 +23:45:55 step 93280 loss=3.3699 lr=9.01e-05 tok=12.226B dt=13241 wait=117 +23:47:11 step 93300 loss=3.3268 lr=9.01e-05 tok=12.229B dt=13317 wait=118 +23:47:32 pushed ckpt_w54.pt,log_w54.txt +23:48:48 waited 76s for global r=256 +23:48:56 global r=256 load 4.240 -> 4.221 +23:50:18 step 93320 loss=3.2954 lr=9.01e-05 tok=12.232B dt=13504 wait=118 +23:51:33 step 93340 loss=3.2375 lr=9.01e-05 tok=12.234B dt=13579 wait=118 +23:52:48 step 93360 loss=3.2495 lr=9.01e-05 tok=12.237B dt=13654 wait=119 +23:54:04 step 93380 loss=3.2893 lr=9.00e-05 tok=12.240B dt=13730 wait=119 +23:55:20 step 93400 loss=3.3638 lr=9.00e-05 tok=12.242B dt=13806 wait=120 +23:56:36 step 93420 loss=3.3085 lr=9.00e-05 tok=12.245B dt=13882 wait=120 +23:57:52 step 93440 loss=3.2687 lr=9.00e-05 tok=12.247B dt=13958 wait=121 +23:58:44 pushed ckpt_w54.pt,log_w54.txt +00:02:27 step 93460 loss=3.3632 lr=9.00e-05 tok=12.250B dt=14233 wait=121 +00:03:43 step 93480 loss=3.4429 lr=8.99e-05 tok=12.253B dt=14309 wait=122 +00:04:59 step 93500 loss=3.4317 lr=8.99e-05 tok=12.255B dt=14385 wait=122 +00:06:15 step 93520 loss=3.4114 lr=8.99e-05 tok=12.258B dt=14461 wait=123 +00:07:31 step 93540 loss=3.3903 lr=8.99e-05 tok=12.260B dt=14537 wait=123 +00:08:47 step 93560 loss=3.2604 lr=8.98e-05 tok=12.263B dt=14613 wait=124 +00:10:03 step 93580 loss=3.2764 lr=8.98e-05 tok=12.266B dt=14689 wait=124 +00:11:18 step 93600 loss=3.3090 lr=8.98e-05 tok=12.268B dt=14764 wait=125 +00:11:37 pushed ckpt_w54.pt,log_w54.txt +00:11:37 waited 0s for global r=257 +00:11:44 global r=257 load 4.251 -> 4.220 +00:13:06 step 93620 loss=3.2609 lr=8.98e-05 tok=12.271B dt=14872 wait=125 +00:14:22 step 93640 loss=3.2380 lr=8.98e-05 tok=12.274B dt=14948 wait=126 +00:15:38 step 93660 loss=3.2669 lr=8.97e-05 tok=12.276B dt=15024 wait=126 +00:16:54 step 93680 loss=3.3429 lr=8.97e-05 tok=12.279B dt=15100 wait=127 +00:18:10 step 93700 loss=3.4548 lr=8.97e-05 tok=12.281B dt=15176 wait=127 +00:19:26 step 93720 loss=3.3539 lr=8.97e-05 tok=12.284B dt=15252 wait=128 +00:20:42 step 93740 loss=3.2662 lr=8.97e-05 tok=12.287B dt=15328 wait=128 +00:21:35 pushed ckpt_w54.pt,log_w54.txt +00:22:51 waited 76s for global r=258 +00:22:59 global r=258 load 4.242 -> 4.222 +00:23:45 step 93760 loss=3.3373 lr=8.96e-05 tok=12.289B dt=15511 wait=129 +00:25:01 step 93780 loss=3.1930 lr=8.96e-05 tok=12.292B dt=15587 wait=129 +00:26:16 step 93800 loss=3.2525 lr=8.96e-05 tok=12.295B dt=15662 wait=129 +00:27:32 step 93820 loss=3.2557 lr=8.96e-05 tok=12.297B dt=15738 wait=130 +00:28:47 step 93840 loss=3.3632 lr=8.95e-05 tok=12.300B dt=15813 wait=130 +00:30:03 step 93860 loss=3.3731 lr=8.95e-05 tok=12.302B dt=15890 wait=131 +00:31:19 step 93880 loss=3.4251 lr=8.95e-05 tok=12.305B dt=15966 wait=131 +00:32:35 step 93900 loss=3.2861 lr=8.95e-05 tok=12.308B dt=16041 wait=132 +00:32:50 pushed ckpt_w54.pt,log_w54.txt +00:37:15 step 93920 loss=3.3499 lr=8.95e-05 tok=12.310B dt=16322 wait=132 +00:38:31 step 93940 loss=3.3975 lr=8.94e-05 tok=12.313B dt=16397 wait=133 +00:39:47 step 93960 loss=3.4795 lr=8.94e-05 tok=12.316B dt=16473 wait=133 +00:41:03 step 93980 loss=3.3343 lr=8.94e-05 tok=12.318B dt=16549 wait=134 +00:42:19 step 94000 loss=3.3320 lr=8.94e-05 tok=12.321B dt=16625 wait=134 +00:43:35 step 94020 loss=3.3579 lr=8.94e-05 tok=12.323B dt=16701 wait=135 +00:44:50 step 94040 loss=3.3018 lr=8.93e-05 tok=12.326B dt=16776 wait=135 +00:45:43 pushed ckpt_w54.pt,log_w54.txt +00:45:43 waited 0s for global r=259 +00:45:50 global r=259 load 4.240 -> 4.221 +00:46:35 step 94060 loss=3.3317 lr=8.93e-05 tok=12.329B dt=16881 wait=136 +00:47:51 step 94080 loss=3.2418 lr=8.93e-05 tok=12.331B dt=16957 wait=136 +00:49:06 step 94100 loss=3.2647 lr=8.93e-05 tok=12.334B dt=17032 wait=136 +00:50:22 step 94120 loss=3.2353 lr=8.93e-05 tok=12.336B dt=17108 wait=137 +00:51:38 step 94140 loss=3.3374 lr=8.92e-05 tok=12.339B dt=17184 wait=137 +00:52:54 step 94160 loss=3.3485 lr=8.92e-05 tok=12.342B dt=17260 wait=138 +00:54:09 step 94180 loss=3.3659 lr=8.92e-05 tok=12.344B dt=17335 wait=138 +00:55:25 step 94200 loss=3.3017 lr=8.92e-05 tok=12.347B dt=17411 wait=139 +00:55:40 pushed ckpt_w54.pt,log_w54.txt +00:56:56 waited 76s for global r=260 +00:57:04 global r=260 load 4.234 -> 4.218 +00:58:27 step 94220 loss=3.2843 lr=8.91e-05 tok=12.350B dt=17593 wait=139 +00:59:43 step 94240 loss=3.2417 lr=8.91e-05 tok=12.352B dt=17669 wait=140 +01:00:58 step 94260 loss=3.2805 lr=8.91e-05 tok=12.355B dt=17744 wait=140 +01:02:14 step 94280 loss=3.3021 lr=8.91e-05 tok=12.357B dt=17820 wait=141 +01:03:30 step 94300 loss=3.3591 lr=8.91e-05 tok=12.360B dt=17896 wait=141 +01:04:45 step 94320 loss=3.3161 lr=8.90e-05 tok=12.363B dt=17971 wait=142 +01:06:01 step 94340 loss=3.2417 lr=8.90e-05 tok=12.365B dt=18047 wait=142 diff --git a/log_w55.txt b/log_w55.txt new file mode 100644 index 0000000000000000000000000000000000000000..6968e03c5f4385388b9a05c71f39a545d847242c --- /dev/null +++ b/log_w55.txt @@ -0,0 +1,164 @@ +launch +22:13:26 name=w55 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:13:33 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:13:42 params 1.08B +22:13:44 resumed step=91950 seen=12.05B round=250 +22:13:44 stream salt=91950 +22:15:33 start: accum=4, 131072 tokens per optimiser step +22:16:55 step 91951 loss=3.0339 lr=9.15e-05 tok=12.052B dt=82 wait=47 +22:17:00 step 91952 loss=3.2260 lr=9.15e-05 tok=12.052B dt=87 wait=47 +22:17:40 step 91960 loss=3.3239 lr=9.15e-05 tok=12.053B dt=127 wait=47 +22:19:03 step 91980 loss=3.2594 lr=9.15e-05 tok=12.056B dt=210 wait=47 +22:20:19 step 92000 loss=3.4209 lr=9.15e-05 tok=12.059B dt=286 wait=48 +22:21:34 step 92020 loss=3.3668 lr=9.15e-05 tok=12.061B dt=362 wait=48 +22:22:50 step 92040 loss=3.3624 lr=9.14e-05 tok=12.064B dt=437 wait=48 +22:24:05 step 92060 loss=3.2268 lr=9.14e-05 tok=12.066B dt=512 wait=49 +22:25:20 step 92080 loss=3.2749 lr=9.14e-05 tok=12.069B dt=587 wait=49 +22:26:36 step 92100 loss=3.2900 lr=9.14e-05 tok=12.072B dt=663 wait=49 +22:26:52 pushed ckpt_w55.pt,log_w55.txt +22:27:07 waited 15s for global r=251 +22:27:20 global r=251 load 4.253 -> 4.237 +22:28:42 step 92120 loss=3.2376 lr=9.14e-05 tok=12.074B dt=789 wait=50 +22:29:57 step 92140 loss=3.2141 lr=9.13e-05 tok=12.077B dt=864 wait=50 +22:31:12 step 92160 loss=3.2074 lr=9.13e-05 tok=12.080B dt=939 wait=51 +22:32:28 step 92180 loss=3.2989 lr=9.13e-05 tok=12.082B dt=1015 wait=51 +22:33:43 step 92200 loss=3.3968 lr=9.13e-05 tok=12.085B dt=1090 wait=51 +22:34:59 step 92220 loss=3.3477 lr=9.13e-05 tok=12.087B dt=1166 wait=52 +22:36:14 step 92240 loss=3.3459 lr=9.12e-05 tok=12.090B dt=1242 wait=52 +22:37:05 pushed ckpt_w55.pt,log_w55.txt +22:40:55 step 92260 loss=3.3837 lr=9.12e-05 tok=12.093B dt=1522 wait=53 +22:42:10 step 92280 loss=3.4371 lr=9.12e-05 tok=12.095B dt=1597 wait=53 +22:43:26 step 92300 loss=3.4031 lr=9.12e-05 tok=12.098B dt=1673 wait=54 +22:44:42 step 92320 loss=3.3835 lr=9.12e-05 tok=12.101B dt=1749 wait=54 +22:45:57 step 92340 loss=3.3927 lr=9.11e-05 tok=12.103B dt=1824 wait=54 +22:47:13 step 92360 loss=3.3033 lr=9.11e-05 tok=12.106B dt=1900 wait=55 +22:48:28 step 92380 loss=3.3492 lr=9.11e-05 tok=12.108B dt=1975 wait=55 +22:49:44 step 92400 loss=3.3349 lr=9.11e-05 tok=12.111B dt=2051 wait=56 +22:49:58 pushed ckpt_w55.pt,log_w55.txt +22:49:58 waited 0s for global r=252 +22:50:04 global r=252 load 4.251 -> 4.228 +22:51:29 step 92420 loss=3.3005 lr=9.10e-05 tok=12.114B dt=2156 wait=56 +22:52:44 step 92440 loss=3.2373 lr=9.10e-05 tok=12.116B dt=2232 wait=56 +22:54:00 step 92460 loss=3.2584 lr=9.10e-05 tok=12.119B dt=2307 wait=57 +22:55:15 step 92480 loss=3.2665 lr=9.10e-05 tok=12.122B dt=2382 wait=57 +22:56:31 step 92500 loss=3.3777 lr=9.10e-05 tok=12.124B dt=2458 wait=58 +22:57:46 step 92520 loss=3.3589 lr=9.09e-05 tok=12.127B dt=2533 wait=58 +22:59:02 step 92540 loss=3.2885 lr=9.09e-05 tok=12.129B dt=2609 wait=59 +22:59:52 pushed ckpt_w55.pt,log_w55.txt +23:00:52 waited 61s for global r=253 +23:01:03 global r=253 load 4.235 -> 4.223 +23:01:46 step 92560 loss=3.3547 lr=9.09e-05 tok=12.132B dt=2773 wait=59 +23:03:01 step 92580 loss=3.2087 lr=9.09e-05 tok=12.135B dt=2848 wait=59 +23:04:16 step 92600 loss=3.2409 lr=9.09e-05 tok=12.137B dt=2924 wait=60 +23:05:32 step 92620 loss=3.2260 lr=9.08e-05 tok=12.140B dt=2999 wait=60 +23:06:47 step 92640 loss=3.3604 lr=9.08e-05 tok=12.143B dt=3075 wait=61 +23:08:03 step 92660 loss=3.3567 lr=9.08e-05 tok=12.145B dt=3150 wait=61 +23:09:19 step 92680 loss=3.3917 lr=9.08e-05 tok=12.148B dt=3226 wait=61 +23:10:34 step 92700 loss=3.3252 lr=9.08e-05 tok=12.150B dt=3301 wait=62 +23:10:46 pushed ckpt_w55.pt,log_w55.txt +23:15:10 step 92720 loss=3.3785 lr=9.07e-05 tok=12.153B dt=3577 wait=62 +23:16:25 step 92740 loss=3.4247 lr=9.07e-05 tok=12.156B dt=3652 wait=63 +23:17:41 step 92760 loss=3.4669 lr=9.07e-05 tok=12.158B dt=3728 wait=63 +23:18:56 step 92780 loss=3.3582 lr=9.07e-05 tok=12.161B dt=3804 wait=64 +23:20:12 step 92800 loss=3.3135 lr=9.06e-05 tok=12.163B dt=3879 wait=64 +23:21:28 step 92820 loss=3.3403 lr=9.06e-05 tok=12.166B dt=3955 wait=64 +23:22:43 step 92840 loss=3.2928 lr=9.06e-05 tok=12.169B dt=4030 wait=65 +23:23:33 pushed ckpt_w55.pt,log_w55.txt +23:23:33 waited 0s for global r=254 +23:23:39 global r=254 load 4.244 -> 4.223 +23:24:24 step 92860 loss=3.3380 lr=9.06e-05 tok=12.171B dt=4131 wait=65 +23:25:39 step 92880 loss=3.2504 lr=9.06e-05 tok=12.174B dt=4206 wait=66 +23:26:54 step 92900 loss=3.3006 lr=9.05e-05 tok=12.177B dt=4282 wait=66 +23:28:10 step 92920 loss=3.2581 lr=9.05e-05 tok=12.179B dt=4357 wait=66 +23:29:25 step 92940 loss=3.3485 lr=9.05e-05 tok=12.182B dt=4432 wait=67 +23:30:41 step 92960 loss=3.3605 lr=9.05e-05 tok=12.184B dt=4508 wait=67 +23:31:56 step 92980 loss=3.3930 lr=9.05e-05 tok=12.187B dt=4583 wait=68 +23:33:12 step 93000 loss=3.3138 lr=9.04e-05 tok=12.190B dt=4659 wait=68 +23:33:24 pushed ckpt_w55.pt,log_w55.txt +23:34:25 waited 61s for global r=255 +23:34:37 global r=255 load 4.237 -> 4.222 +23:36:00 step 93020 loss=3.2768 lr=9.04e-05 tok=12.192B dt=4827 wait=68 +23:37:15 step 93040 loss=3.2294 lr=9.04e-05 tok=12.195B dt=4902 wait=69 +23:38:30 step 93060 loss=3.2639 lr=9.04e-05 tok=12.198B dt=4977 wait=69 +23:39:46 step 93080 loss=3.3089 lr=9.04e-05 tok=12.200B dt=5053 wait=70 +23:41:01 step 93100 loss=3.4057 lr=9.03e-05 tok=12.203B dt=5129 wait=70 +23:42:17 step 93120 loss=3.3774 lr=9.03e-05 tok=12.205B dt=5204 wait=71 +23:43:32 step 93140 loss=3.3173 lr=9.03e-05 tok=12.208B dt=5279 wait=71 +23:44:24 pushed ckpt_w55.pt,log_w55.txt +23:48:06 step 93160 loss=3.3871 lr=9.03e-05 tok=12.211B dt=5554 wait=71 +23:49:22 step 93180 loss=3.4362 lr=9.02e-05 tok=12.213B dt=5629 wait=72 +23:50:37 step 93200 loss=3.3969 lr=9.02e-05 tok=12.216B dt=5704 wait=72 +23:51:53 step 93220 loss=3.3606 lr=9.02e-05 tok=12.219B dt=5780 wait=73 +23:53:08 step 93240 loss=3.3694 lr=9.02e-05 tok=12.221B dt=5855 wait=73 +23:54:23 step 93260 loss=3.2665 lr=9.02e-05 tok=12.224B dt=5930 wait=73 +23:55:39 step 93280 loss=3.3732 lr=9.01e-05 tok=12.226B dt=6006 wait=74 +23:56:54 step 93300 loss=3.3504 lr=9.01e-05 tok=12.229B dt=6081 wait=74 +23:57:08 pushed ckpt_w55.pt,log_w55.txt +23:57:08 waited 0s for global r=256 +23:57:16 global r=256 load 4.253 -> 4.221 +23:58:38 step 93320 loss=3.2784 lr=9.01e-05 tok=12.232B dt=6185 wait=75 +23:59:53 step 93340 loss=3.2259 lr=9.01e-05 tok=12.234B dt=6260 wait=75 +00:01:08 step 93360 loss=3.2364 lr=9.01e-05 tok=12.237B dt=6335 wait=75 +00:02:24 step 93380 loss=3.3067 lr=9.00e-05 tok=12.240B dt=6411 wait=76 +00:03:39 step 93400 loss=3.3659 lr=9.00e-05 tok=12.242B dt=6486 wait=76 +00:04:55 step 93420 loss=3.3452 lr=9.00e-05 tok=12.245B dt=6562 wait=77 +00:06:10 step 93440 loss=3.2881 lr=9.00e-05 tok=12.247B dt=6637 wait=77 +00:07:02 pushed ckpt_w55.pt,log_w55.txt +00:07:48 waited 46s for global r=257 +00:07:54 global r=257 load 4.240 -> 4.220 +00:08:38 step 93460 loss=3.3328 lr=9.00e-05 tok=12.250B dt=6785 wait=78 +00:09:53 step 93480 loss=3.2083 lr=8.99e-05 tok=12.253B dt=6860 wait=78 +00:11:08 step 93500 loss=3.2852 lr=8.99e-05 tok=12.255B dt=6935 wait=78 +00:12:24 step 93520 loss=3.2717 lr=8.99e-05 tok=12.258B dt=7011 wait=79 +00:13:39 step 93540 loss=3.3933 lr=8.99e-05 tok=12.260B dt=7086 wait=79 +00:14:55 step 93560 loss=3.3626 lr=8.98e-05 tok=12.263B dt=7162 wait=80 +00:16:11 step 93580 loss=3.3584 lr=8.98e-05 tok=12.266B dt=7238 wait=80 +00:17:26 step 93600 loss=3.2915 lr=8.98e-05 tok=12.268B dt=7313 wait=80 +00:17:40 pushed ckpt_w55.pt,log_w55.txt +00:22:00 step 93620 loss=3.3526 lr=8.98e-05 tok=12.271B dt=7587 wait=81 +00:23:16 step 93640 loss=3.4177 lr=8.98e-05 tok=12.274B dt=7663 wait=81 +00:24:31 step 93660 loss=3.4835 lr=8.97e-05 tok=12.276B dt=7739 wait=82 +00:25:47 step 93680 loss=3.3910 lr=8.97e-05 tok=12.279B dt=7814 wait=82 +00:27:03 step 93700 loss=3.3553 lr=8.97e-05 tok=12.281B dt=7890 wait=82 +00:28:18 step 93720 loss=3.3699 lr=8.97e-05 tok=12.284B dt=7965 wait=83 +00:29:34 step 93740 loss=3.2448 lr=8.97e-05 tok=12.287B dt=8041 wait=83 +00:30:25 pushed ckpt_w55.pt,log_w55.txt +00:30:25 waited 0s for global r=258 +00:30:32 global r=258 load 4.238 -> 4.222 +00:31:17 step 93760 loss=3.3098 lr=8.96e-05 tok=12.289B dt=8144 wait=84 +00:32:32 step 93780 loss=3.2014 lr=8.96e-05 tok=12.292B dt=8219 wait=84 +00:33:47 step 93800 loss=3.2971 lr=8.96e-05 tok=12.295B dt=8294 wait=85 +00:35:03 step 93820 loss=3.2717 lr=8.96e-05 tok=12.297B dt=8370 wait=85 +00:36:19 step 93840 loss=3.3937 lr=8.95e-05 tok=12.300B dt=8446 wait=85 +00:37:34 step 93860 loss=3.3656 lr=8.95e-05 tok=12.302B dt=8521 wait=86 +00:38:50 step 93880 loss=3.3826 lr=8.95e-05 tok=12.305B dt=8597 wait=86 +00:40:05 step 93900 loss=3.2829 lr=8.95e-05 tok=12.308B dt=8672 wait=87 +00:40:19 pushed ckpt_w55.pt,log_w55.txt +00:41:19 waited 61s for global r=259 +00:41:25 global r=259 load 4.235 -> 4.221 +00:42:44 step 93920 loss=3.2366 lr=8.95e-05 tok=12.310B dt=8832 wait=87 +00:44:00 step 93940 loss=3.2411 lr=8.94e-05 tok=12.313B dt=8907 wait=88 +00:45:15 step 93960 loss=3.2536 lr=8.94e-05 tok=12.316B dt=8982 wait=88 +00:46:31 step 93980 loss=3.2799 lr=8.94e-05 tok=12.318B dt=9058 wait=88 +00:47:46 step 94000 loss=3.4119 lr=8.94e-05 tok=12.321B dt=9134 wait=89 +00:49:02 step 94020 loss=3.3682 lr=8.94e-05 tok=12.323B dt=9209 wait=89 +00:50:18 step 94040 loss=3.2850 lr=8.93e-05 tok=12.326B dt=9285 wait=90 +00:51:08 pushed ckpt_w55.pt,log_w55.txt +00:54:51 step 94060 loss=3.3354 lr=8.93e-05 tok=12.329B dt=9558 wait=90 +00:56:07 step 94080 loss=3.4501 lr=8.93e-05 tok=12.331B dt=9634 wait=90 +00:57:22 step 94100 loss=3.4076 lr=8.93e-05 tok=12.334B dt=9709 wait=91 +00:58:38 step 94120 loss=3.3678 lr=8.93e-05 tok=12.336B dt=9785 wait=91 +00:59:53 step 94140 loss=3.3634 lr=8.92e-05 tok=12.339B dt=9860 wait=92 +01:01:09 step 94160 loss=3.2524 lr=8.92e-05 tok=12.342B dt=9936 wait=92 +01:02:24 step 94180 loss=3.3196 lr=8.92e-05 tok=12.344B dt=10011 wait=93 +01:03:39 step 94200 loss=3.3273 lr=8.92e-05 tok=12.347B dt=10086 wait=93 +01:03:52 pushed ckpt_w55.pt,log_w55.txt +01:03:53 waited 0s for global r=260 +01:03:58 global r=260 load 4.246 -> 4.218 +01:05:21 step 94220 loss=3.3179 lr=8.91e-05 tok=12.350B dt=10188 wait=93 +01:06:37 step 94240 loss=3.2860 lr=8.91e-05 tok=12.352B dt=10264 wait=94 +01:07:52 step 94260 loss=3.2699 lr=8.91e-05 tok=12.355B dt=10339 wait=94 +01:09:08 step 94280 loss=3.3008 lr=8.91e-05 tok=12.357B dt=10415 wait=95 +01:10:23 step 94300 loss=3.3853 lr=8.91e-05 tok=12.360B dt=10490 wait=95 +01:11:39 step 94320 loss=3.3310 lr=8.90e-05 tok=12.363B dt=10566 wait=95 +01:12:54 step 94340 loss=3.2577 lr=8.90e-05 tok=12.365B dt=10641 wait=96 diff --git a/log_w56.txt b/log_w56.txt new file mode 100644 index 0000000000000000000000000000000000000000..747028cb58a23c89cb4724b9b8dd5dbec80c00d8 --- /dev/null +++ b/log_w56.txt @@ -0,0 +1,156 @@ +launch +22:14:11 name=w56 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:14:17 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:14:27 params 1.08B +22:14:28 resumed step=91950 seen=12.05B round=250 +22:14:28 stream salt=91950 +22:16:23 start: accum=4, 131072 tokens per optimiser step +22:17:47 step 91951 loss=2.8182 lr=9.15e-05 tok=12.052B dt=85 wait=50 +22:17:53 step 91952 loss=3.1393 lr=9.15e-05 tok=12.052B dt=90 wait=50 +22:18:33 step 91960 loss=3.2714 lr=9.15e-05 tok=12.053B dt=131 wait=51 +22:19:54 step 91980 loss=3.2693 lr=9.15e-05 tok=12.056B dt=212 wait=51 +22:21:10 step 92000 loss=3.3945 lr=9.15e-05 tok=12.059B dt=287 wait=52 +22:22:26 step 92020 loss=3.3498 lr=9.15e-05 tok=12.061B dt=363 wait=52 +22:23:41 step 92040 loss=3.3001 lr=9.14e-05 tok=12.064B dt=438 wait=53 +22:24:56 step 92060 loss=3.2448 lr=9.14e-05 tok=12.066B dt=514 wait=53 +22:26:12 step 92080 loss=3.2413 lr=9.14e-05 tok=12.069B dt=589 wait=53 +22:27:27 step 92100 loss=3.3416 lr=9.14e-05 tok=12.072B dt=664 wait=54 +22:27:40 pushed ckpt_w56.pt,log_w56.txt +22:27:40 waited 0s for global r=251 +22:27:49 global r=251 load 4.248 -> 4.237 +22:29:09 step 92120 loss=3.2660 lr=9.14e-05 tok=12.074B dt=766 wait=54 +22:30:24 step 92140 loss=3.2287 lr=9.13e-05 tok=12.077B dt=841 wait=55 +22:31:39 step 92160 loss=3.2592 lr=9.13e-05 tok=12.080B dt=916 wait=55 +22:32:55 step 92180 loss=3.3134 lr=9.13e-05 tok=12.082B dt=992 wait=55 +22:34:10 step 92200 loss=3.4145 lr=9.13e-05 tok=12.085B dt=1068 wait=56 +22:35:26 step 92220 loss=3.4026 lr=9.13e-05 tok=12.087B dt=1143 wait=56 +22:36:41 step 92240 loss=3.2905 lr=9.12e-05 tok=12.090B dt=1219 wait=56 +22:37:32 pushed ckpt_w56.pt,log_w56.txt +22:40:18 waited 166s for global r=252 +22:40:24 global r=252 load 4.252 -> 4.228 +22:41:06 step 92260 loss=3.3230 lr=9.12e-05 tok=12.093B dt=1484 wait=57 +22:42:21 step 92280 loss=3.2111 lr=9.12e-05 tok=12.095B dt=1559 wait=57 +22:43:37 step 92300 loss=3.2507 lr=9.12e-05 tok=12.098B dt=1634 wait=58 +22:44:52 step 92320 loss=3.2369 lr=9.12e-05 tok=12.101B dt=1709 wait=58 +22:46:07 step 92340 loss=3.3851 lr=9.11e-05 tok=12.103B dt=1785 wait=58 +22:47:23 step 92360 loss=3.3329 lr=9.11e-05 tok=12.106B dt=1860 wait=59 +22:48:38 step 92380 loss=3.3553 lr=9.11e-05 tok=12.108B dt=1935 wait=59 +22:49:53 step 92400 loss=3.2565 lr=9.11e-05 tok=12.111B dt=2011 wait=60 +22:50:15 pushed ckpt_w56.pt,log_w56.txt +22:54:35 step 92420 loss=3.3570 lr=9.10e-05 tok=12.114B dt=2292 wait=60 +22:55:50 step 92440 loss=3.4360 lr=9.10e-05 tok=12.116B dt=2367 wait=60 +22:57:06 step 92460 loss=3.4801 lr=9.10e-05 tok=12.119B dt=2443 wait=61 +22:58:21 step 92480 loss=3.3397 lr=9.10e-05 tok=12.122B dt=2518 wait=61 +22:59:37 step 92500 loss=3.3141 lr=9.10e-05 tok=12.124B dt=2594 wait=62 +23:00:52 step 92520 loss=3.3572 lr=9.09e-05 tok=12.127B dt=2670 wait=62 +23:02:08 step 92540 loss=3.2718 lr=9.09e-05 tok=12.129B dt=2745 wait=62 +23:02:59 pushed ckpt_w56.pt,log_w56.txt +23:02:59 waited 0s for global r=253 +23:03:07 global r=253 load 4.249 -> 4.223 +23:03:48 step 92560 loss=3.2915 lr=9.09e-05 tok=12.132B dt=2845 wait=63 +23:05:03 step 92580 loss=3.2062 lr=9.09e-05 tok=12.135B dt=2920 wait=63 +23:06:18 step 92600 loss=3.2555 lr=9.09e-05 tok=12.137B dt=2995 wait=64 +23:07:33 step 92620 loss=3.2380 lr=9.08e-05 tok=12.140B dt=3071 wait=64 +23:08:49 step 92640 loss=3.3860 lr=9.08e-05 tok=12.143B dt=3146 wait=64 +23:10:04 step 92660 loss=3.3566 lr=9.08e-05 tok=12.145B dt=3222 wait=65 +23:11:20 step 92680 loss=3.3821 lr=9.08e-05 tok=12.148B dt=3297 wait=65 +23:12:35 step 92700 loss=3.2815 lr=9.08e-05 tok=12.150B dt=3372 wait=65 +23:12:47 pushed ckpt_w56.pt,log_w56.txt +23:14:33 waited 106s for global r=254 +23:14:39 global r=254 load 4.247 -> 4.223 +23:15:57 step 92720 loss=3.2464 lr=9.07e-05 tok=12.153B dt=3575 wait=66 +23:17:13 step 92740 loss=3.2091 lr=9.07e-05 tok=12.156B dt=3650 wait=66 +23:18:28 step 92760 loss=3.2312 lr=9.07e-05 tok=12.158B dt=3725 wait=67 +23:19:43 step 92780 loss=3.3223 lr=9.07e-05 tok=12.161B dt=3800 wait=67 +23:20:59 step 92800 loss=3.4241 lr=9.06e-05 tok=12.163B dt=3876 wait=67 +23:22:14 step 92820 loss=3.3647 lr=9.06e-05 tok=12.166B dt=3952 wait=68 +23:23:30 step 92840 loss=3.3164 lr=9.06e-05 tok=12.169B dt=4027 wait=68 +23:24:19 pushed ckpt_w56.pt,log_w56.txt +23:28:03 step 92860 loss=3.3657 lr=9.06e-05 tok=12.171B dt=4301 wait=68 +23:29:19 step 92880 loss=3.4539 lr=9.06e-05 tok=12.174B dt=4376 wait=69 +23:30:34 step 92900 loss=3.4243 lr=9.05e-05 tok=12.177B dt=4451 wait=69 +23:31:49 step 92920 loss=3.3636 lr=9.05e-05 tok=12.179B dt=4527 wait=70 +23:33:05 step 92940 loss=3.3824 lr=9.05e-05 tok=12.182B dt=4602 wait=70 +23:34:20 step 92960 loss=3.3026 lr=9.05e-05 tok=12.184B dt=4677 wait=70 +23:35:36 step 92980 loss=3.3330 lr=9.05e-05 tok=12.187B dt=4753 wait=71 +23:36:51 step 93000 loss=3.3433 lr=9.04e-05 tok=12.190B dt=4828 wait=71 +23:37:04 pushed ckpt_w56.pt,log_w56.txt +23:37:04 waited 0s for global r=255 +23:37:11 global r=255 load 4.241 -> 4.222 +23:38:34 step 93020 loss=3.3204 lr=9.04e-05 tok=12.192B dt=4931 wait=72 +23:39:49 step 93040 loss=3.2389 lr=9.04e-05 tok=12.195B dt=5006 wait=72 +23:41:04 step 93060 loss=3.2527 lr=9.04e-05 tok=12.198B dt=5081 wait=72 +23:42:19 step 93080 loss=3.2693 lr=9.04e-05 tok=12.200B dt=5157 wait=73 +23:43:35 step 93100 loss=3.3515 lr=9.03e-05 tok=12.203B dt=5232 wait=73 +23:44:50 step 93120 loss=3.3454 lr=9.03e-05 tok=12.205B dt=5307 wait=73 +23:46:05 step 93140 loss=3.3012 lr=9.03e-05 tok=12.208B dt=5383 wait=74 +23:46:55 pushed ckpt_w56.pt,log_w56.txt +23:48:40 waited 106s for global r=256 +23:48:47 global r=256 load 4.236 -> 4.221 +23:49:31 step 93160 loss=3.3558 lr=9.03e-05 tok=12.211B dt=5589 wait=74 +23:50:47 step 93180 loss=3.2592 lr=9.02e-05 tok=12.213B dt=5664 wait=75 +23:52:02 step 93200 loss=3.2984 lr=9.02e-05 tok=12.216B dt=5739 wait=75 +23:53:17 step 93220 loss=3.2564 lr=9.02e-05 tok=12.219B dt=5814 wait=75 +23:54:32 step 93240 loss=3.3253 lr=9.02e-05 tok=12.221B dt=5890 wait=76 +23:55:48 step 93260 loss=3.3286 lr=9.02e-05 tok=12.224B dt=5965 wait=76 +23:57:03 step 93280 loss=3.3730 lr=9.01e-05 tok=12.226B dt=6040 wait=77 +23:58:18 step 93300 loss=3.3031 lr=9.01e-05 tok=12.229B dt=6116 wait=77 +23:58:33 pushed ckpt_w56.pt,log_w56.txt +00:02:57 step 93320 loss=3.3784 lr=9.01e-05 tok=12.232B dt=6394 wait=77 +00:04:12 step 93340 loss=3.4340 lr=9.01e-05 tok=12.234B dt=6470 wait=78 +00:05:28 step 93360 loss=3.4472 lr=9.01e-05 tok=12.237B dt=6545 wait=78 +00:06:43 step 93380 loss=3.3211 lr=9.00e-05 tok=12.240B dt=6620 wait=78 +00:07:59 step 93400 loss=3.3105 lr=9.00e-05 tok=12.242B dt=6696 wait=79 +00:09:14 step 93420 loss=3.3113 lr=9.00e-05 tok=12.245B dt=6771 wait=79 +00:10:29 step 93440 loss=3.2956 lr=9.00e-05 tok=12.247B dt=6847 wait=80 +00:11:21 pushed ckpt_w56.pt,log_w56.txt +00:11:21 waited 0s for global r=257 +00:11:29 global r=257 load 4.251 -> 4.220 +00:12:13 step 93460 loss=3.3122 lr=9.00e-05 tok=12.250B dt=6950 wait=80 +00:13:28 step 93480 loss=3.2431 lr=8.99e-05 tok=12.253B dt=7025 wait=80 +00:14:43 step 93500 loss=3.2533 lr=8.99e-05 tok=12.255B dt=7101 wait=81 +00:15:59 step 93520 loss=3.2417 lr=8.99e-05 tok=12.258B dt=7176 wait=81 +00:17:14 step 93540 loss=3.3222 lr=8.99e-05 tok=12.260B dt=7251 wait=82 +00:18:29 step 93560 loss=3.3155 lr=8.98e-05 tok=12.263B dt=7326 wait=82 +00:19:44 step 93580 loss=3.3747 lr=8.98e-05 tok=12.266B dt=7402 wait=82 +00:21:00 step 93600 loss=3.2861 lr=8.98e-05 tok=12.268B dt=7477 wait=83 +00:21:12 pushed ckpt_w56.pt,log_w56.txt +00:22:59 waited 107s for global r=258 +00:23:07 global r=258 load 4.240 -> 4.222 +00:24:28 step 93620 loss=3.2726 lr=8.98e-05 tok=12.271B dt=7685 wait=83 +00:25:43 step 93640 loss=3.2497 lr=8.98e-05 tok=12.274B dt=7761 wait=84 +00:26:59 step 93660 loss=3.2383 lr=8.97e-05 tok=12.276B dt=7836 wait=84 +00:28:14 step 93680 loss=3.3148 lr=8.97e-05 tok=12.279B dt=7911 wait=84 +00:29:29 step 93700 loss=3.3840 lr=8.97e-05 tok=12.281B dt=7987 wait=85 +00:30:45 step 93720 loss=3.3251 lr=8.97e-05 tok=12.284B dt=8062 wait=85 +00:32:00 step 93740 loss=3.2831 lr=8.97e-05 tok=12.287B dt=8138 wait=85 +00:32:51 pushed ckpt_w56.pt,log_w56.txt +00:36:40 step 93760 loss=3.3910 lr=8.96e-05 tok=12.289B dt=8417 wait=86 +00:37:55 step 93780 loss=3.4416 lr=8.96e-05 tok=12.292B dt=8493 wait=86 +00:39:11 step 93800 loss=3.4020 lr=8.96e-05 tok=12.295B dt=8568 wait=87 +00:40:26 step 93820 loss=3.3722 lr=8.96e-05 tok=12.297B dt=8643 wait=87 +00:41:41 step 93840 loss=3.3870 lr=8.95e-05 tok=12.300B dt=8719 wait=87 +00:42:57 step 93860 loss=3.2507 lr=8.95e-05 tok=12.302B dt=8794 wait=88 +00:44:12 step 93880 loss=3.3307 lr=8.95e-05 tok=12.305B dt=8869 wait=88 +00:45:27 step 93900 loss=3.3080 lr=8.95e-05 tok=12.308B dt=8945 wait=88 +00:45:41 pushed ckpt_w56.pt,log_w56.txt +00:45:41 waited 0s for global r=259 +00:45:49 global r=259 load 4.248 -> 4.221 +00:47:10 step 93920 loss=3.2840 lr=8.95e-05 tok=12.310B dt=9047 wait=89 +00:48:25 step 93940 loss=3.2158 lr=8.94e-05 tok=12.313B dt=9122 wait=89 +00:49:40 step 93960 loss=3.2350 lr=8.94e-05 tok=12.316B dt=9198 wait=90 +00:50:56 step 93980 loss=3.3186 lr=8.94e-05 tok=12.318B dt=9273 wait=90 +00:52:11 step 94000 loss=3.4277 lr=8.94e-05 tok=12.321B dt=9349 wait=90 +00:53:27 step 94020 loss=3.3534 lr=8.94e-05 tok=12.323B dt=9424 wait=91 +00:54:42 step 94040 loss=3.2830 lr=8.93e-05 tok=12.326B dt=9499 wait=91 +00:55:31 pushed ckpt_w56.pt,log_w56.txt +00:57:02 waited 91s for global r=260 +00:57:09 global r=260 load 4.243 -> 4.218 +00:57:54 step 94060 loss=3.3450 lr=8.93e-05 tok=12.329B dt=9691 wait=92 +00:59:09 step 94080 loss=3.2218 lr=8.93e-05 tok=12.331B dt=9767 wait=92 +01:00:24 step 94100 loss=3.2339 lr=8.93e-05 tok=12.334B dt=9842 wait=92 +01:01:40 step 94120 loss=3.2317 lr=8.93e-05 tok=12.336B dt=9917 wait=93 +01:02:55 step 94140 loss=3.3639 lr=8.92e-05 tok=12.339B dt=9992 wait=93 +01:04:11 step 94160 loss=3.3952 lr=8.92e-05 tok=12.342B dt=10068 wait=94 +01:05:26 step 94180 loss=3.4052 lr=8.92e-05 tok=12.344B dt=10143 wait=94 +01:06:42 step 94200 loss=3.3231 lr=8.92e-05 tok=12.347B dt=10219 wait=94 diff --git a/log_w57.txt b/log_w57.txt new file mode 100644 index 0000000000000000000000000000000000000000..67220492f5548882b37fad117744ffb7d4dd3802 --- /dev/null +++ b/log_w57.txt @@ -0,0 +1,260 @@ +launch +12:41:04 name=w57 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 4/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:41:10 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:41:20 params 1.08B +12:41:22 resumed step=94350 seen=12.37B round=260 +12:41:22 stream salt=94350 +12:43:21 start: accum=4, 131072 tokens per optimiser step +12:45:01 step 94351 loss=2.9174 lr=8.90e-05 tok=12.367B dt=100 wait=58 +12:45:06 step 94352 loss=3.1430 lr=8.90e-05 tok=12.367B dt=105 wait=58 +12:45:48 step 94360 loss=3.3237 lr=8.90e-05 tok=12.368B dt=147 wait=58 +12:47:22 step 94380 loss=3.2568 lr=8.90e-05 tok=12.371B dt=241 wait=59 +12:48:38 step 94400 loss=3.4413 lr=8.90e-05 tok=12.373B dt=317 wait=59 +12:49:54 step 94420 loss=3.3380 lr=8.89e-05 tok=12.376B dt=393 wait=60 +12:51:10 step 94440 loss=3.2923 lr=8.89e-05 tok=12.378B dt=469 wait=60 +12:52:25 step 94460 loss=3.2523 lr=8.89e-05 tok=12.381B dt=544 wait=61 +12:53:41 step 94480 loss=3.2441 lr=8.89e-05 tok=12.384B dt=620 wait=61 +12:54:56 step 94500 loss=3.3076 lr=8.89e-05 tok=12.386B dt=696 wait=62 +12:55:16 pushed ckpt_w57.pt,log_w57.txt +12:56:47 waited 91s for global r=261 +12:57:00 global r=261 load 4.242 -> 4.218 +12:58:24 step 94520 loss=3.2587 lr=8.88e-05 tok=12.389B dt=903 wait=62 +12:59:39 step 94540 loss=3.2351 lr=8.88e-05 tok=12.392B dt=978 wait=63 +13:00:55 step 94560 loss=3.2431 lr=8.88e-05 tok=12.394B dt=1054 wait=63 +13:02:10 step 94580 loss=3.3121 lr=8.88e-05 tok=12.397B dt=1130 wait=64 +13:03:26 step 94600 loss=3.4056 lr=8.87e-05 tok=12.399B dt=1205 wait=64 +13:04:42 step 94620 loss=3.3368 lr=8.87e-05 tok=12.402B dt=1281 wait=64 +13:05:57 step 94640 loss=3.2826 lr=8.87e-05 tok=12.405B dt=1356 wait=65 +13:06:53 pushed ckpt_w57.pt,log_w57.txt +13:08:24 waited 91s for global r=262 +13:08:35 global r=262 load 4.236 -> 4.222 +13:09:21 step 94660 loss=3.3388 lr=8.87e-05 tok=12.407B dt=1560 wait=65 +13:10:37 step 94680 loss=3.2197 lr=8.87e-05 tok=12.410B dt=1636 wait=66 +13:11:52 step 94700 loss=3.2327 lr=8.86e-05 tok=12.413B dt=1711 wait=66 +13:13:08 step 94720 loss=3.2421 lr=8.86e-05 tok=12.415B dt=1787 wait=67 +13:14:23 step 94740 loss=3.3603 lr=8.86e-05 tok=12.418B dt=1862 wait=67 +13:15:39 step 94760 loss=3.3350 lr=8.86e-05 tok=12.420B dt=1938 wait=68 +13:16:55 step 94780 loss=3.3826 lr=8.86e-05 tok=12.423B dt=2014 wait=68 +13:18:11 step 94800 loss=3.3276 lr=8.85e-05 tok=12.426B dt=2090 wait=69 +13:18:27 pushed ckpt_w57.pt,log_w57.txt +13:19:59 waited 91s for global r=263 +13:20:06 global r=263 load 4.235 -> 4.216 +13:21:31 step 94820 loss=3.2726 lr=8.85e-05 tok=12.428B dt=2291 wait=69 +13:22:47 step 94840 loss=3.2330 lr=8.85e-05 tok=12.431B dt=2366 wait=69 +13:24:02 step 94860 loss=3.2419 lr=8.85e-05 tok=12.433B dt=2441 wait=70 +13:25:18 step 94880 loss=3.2934 lr=8.85e-05 tok=12.436B dt=2517 wait=70 +13:26:34 step 94900 loss=3.3700 lr=8.84e-05 tok=12.439B dt=2593 wait=71 +13:27:50 step 94920 loss=3.3553 lr=8.84e-05 tok=12.441B dt=2669 wait=71 +13:29:05 step 94940 loss=3.2929 lr=8.84e-05 tok=12.444B dt=2745 wait=72 +13:29:59 pushed ckpt_w57.pt,log_w57.txt +13:31:30 waited 91s for global r=264 +13:31:38 global r=264 load 4.238 -> 4.220 +13:32:24 step 94960 loss=3.3649 lr=8.84e-05 tok=12.447B dt=2943 wait=72 +13:33:40 step 94980 loss=3.2176 lr=8.83e-05 tok=12.449B dt=3019 wait=73 +13:34:55 step 95000 loss=3.2947 lr=8.83e-05 tok=12.452B dt=3094 wait=73 +13:36:11 step 95020 loss=3.2658 lr=8.83e-05 tok=12.454B dt=3170 wait=74 +13:37:26 step 95040 loss=3.3877 lr=8.83e-05 tok=12.457B dt=3245 wait=74 +13:38:42 step 95060 loss=3.3578 lr=8.83e-05 tok=12.460B dt=3321 wait=74 +13:39:58 step 95080 loss=3.3873 lr=8.82e-05 tok=12.462B dt=3397 wait=75 +13:41:13 step 95100 loss=3.3188 lr=8.82e-05 tok=12.465B dt=3473 wait=75 +13:41:28 pushed ckpt_w57.pt,log_w57.txt +13:43:00 waited 92s for global r=265 +13:43:11 global r=265 load 4.234 -> 4.216 +13:44:33 step 95120 loss=3.3068 lr=8.82e-05 tok=12.468B dt=3672 wait=76 +13:45:49 step 95140 loss=3.2499 lr=8.82e-05 tok=12.470B dt=3748 wait=76 +13:47:04 step 95160 loss=3.2442 lr=8.82e-05 tok=12.473B dt=3824 wait=77 +13:48:20 step 95180 loss=3.2725 lr=8.81e-05 tok=12.475B dt=3899 wait=77 +13:49:36 step 95200 loss=3.3874 lr=8.81e-05 tok=12.478B dt=3975 wait=78 +13:50:52 step 95220 loss=3.3020 lr=8.81e-05 tok=12.481B dt=4051 wait=78 +13:52:07 step 95240 loss=3.2646 lr=8.81e-05 tok=12.483B dt=4126 wait=79 +13:52:59 pushed ckpt_w57.pt,log_w57.txt +13:54:31 waited 92s for global r=266 +13:54:41 global r=266 load 4.227 -> 4.215 +13:55:27 step 95260 loss=3.3371 lr=8.81e-05 tok=12.486B dt=4326 wait=79 +13:56:42 step 95280 loss=3.2505 lr=8.80e-05 tok=12.489B dt=4402 wait=80 +13:57:58 step 95300 loss=3.3021 lr=8.80e-05 tok=12.491B dt=4477 wait=80 +13:59:14 step 95320 loss=3.2697 lr=8.80e-05 tok=12.494B dt=4553 wait=81 +14:00:29 step 95340 loss=3.3612 lr=8.80e-05 tok=12.496B dt=4629 wait=81 +14:01:45 step 95360 loss=3.2901 lr=8.79e-05 tok=12.499B dt=4704 wait=82 +14:03:01 step 95380 loss=3.3208 lr=8.79e-05 tok=12.502B dt=4780 wait=82 +14:04:16 step 95400 loss=3.2734 lr=8.79e-05 tok=12.504B dt=4855 wait=83 +14:04:33 pushed ckpt_w57.pt,log_w57.txt +14:06:05 waited 92s for global r=267 +14:06:14 global r=267 load 4.245 -> 4.219 +14:07:40 step 95420 loss=3.2851 lr=8.79e-05 tok=12.507B dt=5059 wait=83 +14:08:55 step 95440 loss=3.2504 lr=8.79e-05 tok=12.510B dt=5134 wait=83 +14:10:11 step 95460 loss=3.2752 lr=8.78e-05 tok=12.512B dt=5210 wait=84 +14:11:26 step 95480 loss=3.3175 lr=8.78e-05 tok=12.515B dt=5286 wait=84 +14:12:42 step 95500 loss=3.3926 lr=8.78e-05 tok=12.517B dt=5361 wait=85 +14:13:58 step 95520 loss=3.3492 lr=8.78e-05 tok=12.520B dt=5437 wait=85 +14:15:13 step 95540 loss=3.2462 lr=8.78e-05 tok=12.523B dt=5513 wait=86 +14:16:06 pushed ckpt_w57.pt,log_w57.txt +14:17:37 waited 91s for global r=268 +14:17:47 global r=268 load 4.232 -> 4.216 +14:18:32 step 95560 loss=3.3260 lr=8.77e-05 tok=12.525B dt=5711 wait=86 +14:19:47 step 95580 loss=3.2189 lr=8.77e-05 tok=12.528B dt=5786 wait=87 +14:21:02 step 95600 loss=3.2188 lr=8.77e-05 tok=12.530B dt=5862 wait=87 +14:22:18 step 95620 loss=3.1953 lr=8.77e-05 tok=12.533B dt=5937 wait=88 +14:23:34 step 95640 loss=3.3823 lr=8.76e-05 tok=12.536B dt=6013 wait=88 +14:24:50 step 95660 loss=3.3472 lr=8.76e-05 tok=12.538B dt=6089 wait=89 +14:26:06 step 95680 loss=3.3833 lr=8.76e-05 tok=12.541B dt=6165 wait=89 +14:27:21 step 95700 loss=3.2591 lr=8.76e-05 tok=12.544B dt=6240 wait=89 +14:27:41 pushed ckpt_w57.pt,log_w57.txt +14:28:57 waited 77s for global r=269 +14:29:13 global r=269 load 4.235 -> 4.216 +14:30:41 step 95720 loss=3.2674 lr=8.76e-05 tok=12.546B dt=6440 wait=90 +14:31:56 step 95740 loss=3.2147 lr=8.75e-05 tok=12.549B dt=6516 wait=90 +14:33:12 step 95760 loss=3.2569 lr=8.75e-05 tok=12.551B dt=6591 wait=91 +14:34:27 step 95780 loss=3.2965 lr=8.75e-05 tok=12.554B dt=6667 wait=91 +14:35:43 step 95800 loss=3.3840 lr=8.75e-05 tok=12.557B dt=6743 wait=92 +14:36:59 step 95820 loss=3.3549 lr=8.75e-05 tok=12.559B dt=6818 wait=92 +14:38:15 step 95840 loss=3.2922 lr=8.74e-05 tok=12.562B dt=6894 wait=93 +14:39:07 pushed ckpt_w57.pt,log_w57.txt +14:40:53 waited 106s for global r=270 +14:41:05 global r=270 load 4.229 -> 4.212 +14:41:52 step 95860 loss=3.3358 lr=8.74e-05 tok=12.565B dt=7111 wait=93 +14:43:07 step 95880 loss=3.1978 lr=8.74e-05 tok=12.567B dt=7187 wait=94 +14:44:23 step 95900 loss=3.2336 lr=8.74e-05 tok=12.570B dt=7262 wait=94 +14:45:38 step 95920 loss=3.2151 lr=8.74e-05 tok=12.572B dt=7337 wait=94 +14:46:54 step 95940 loss=3.3449 lr=8.73e-05 tok=12.575B dt=7413 wait=95 +14:48:10 step 95960 loss=3.3502 lr=8.73e-05 tok=12.578B dt=7489 wait=95 +14:49:25 step 95980 loss=3.4111 lr=8.73e-05 tok=12.580B dt=7564 wait=96 +14:50:41 step 96000 loss=3.3040 lr=8.73e-05 tok=12.583B dt=7640 wait=96 +14:50:58 pushed ckpt_w57.pt,log_w57.txt +14:52:44 waited 106s for global r=271 +14:52:59 global r=271 load 4.238 -> 4.216 +14:54:20 step 96020 loss=3.2663 lr=8.72e-05 tok=12.586B dt=7860 wait=97 +14:55:36 step 96040 loss=3.2085 lr=8.72e-05 tok=12.588B dt=7935 wait=97 +14:56:51 step 96060 loss=3.2262 lr=8.72e-05 tok=12.591B dt=8010 wait=98 +14:58:07 step 96080 loss=3.2668 lr=8.72e-05 tok=12.593B dt=8086 wait=98 +14:59:23 step 96100 loss=3.3860 lr=8.72e-05 tok=12.596B dt=8162 wait=99 +15:00:38 step 96120 loss=3.3079 lr=8.71e-05 tok=12.599B dt=8237 wait=99 +15:01:54 step 96140 loss=3.2782 lr=8.71e-05 tok=12.601B dt=8313 wait=100 +15:02:47 pushed ckpt_w57.pt,log_w57.txt +15:04:33 waited 106s for global r=272 +15:04:49 global r=272 load 4.241 -> 4.219 +15:05:35 step 96160 loss=3.3631 lr=8.71e-05 tok=12.604B dt=8534 wait=100 +15:06:51 step 96180 loss=3.2327 lr=8.71e-05 tok=12.607B dt=8610 wait=101 +15:08:06 step 96200 loss=3.3071 lr=8.71e-05 tok=12.609B dt=8686 wait=101 +15:09:23 step 96220 loss=3.2639 lr=8.70e-05 tok=12.612B dt=8762 wait=101 +15:10:38 step 96240 loss=3.3610 lr=8.70e-05 tok=12.614B dt=8837 wait=102 +15:11:54 step 96260 loss=3.3204 lr=8.70e-05 tok=12.617B dt=8913 wait=102 +15:13:09 step 96280 loss=3.3214 lr=8.70e-05 tok=12.620B dt=8988 wait=103 +15:14:25 step 96300 loss=3.2780 lr=8.70e-05 tok=12.622B dt=9064 wait=103 +15:14:39 pushed ckpt_w57.pt,log_w57.txt +15:16:25 waited 106s for global r=273 +15:16:36 global r=273 load 4.231 -> 4.214 +15:17:59 step 96320 loss=3.2847 lr=8.69e-05 tok=12.625B dt=9278 wait=104 +15:19:14 step 96340 loss=3.2312 lr=8.69e-05 tok=12.627B dt=9353 wait=104 +15:20:30 step 96360 loss=3.2867 lr=8.69e-05 tok=12.630B dt=9429 wait=105 +15:21:47 step 96380 loss=3.2866 lr=8.69e-05 tok=12.633B dt=9506 wait=105 +15:23:03 step 96400 loss=3.3924 lr=8.68e-05 tok=12.635B dt=9582 wait=106 +15:24:18 step 96420 loss=3.3154 lr=8.68e-05 tok=12.638B dt=9657 wait=106 +15:25:33 step 96440 loss=3.2324 lr=8.68e-05 tok=12.641B dt=9732 wait=107 +15:26:27 pushed ckpt_w57.pt,log_w57.txt +15:28:14 waited 107s for global r=274 +15:28:36 global r=274 load 4.240 -> 4.219 +15:29:20 step 96460 loss=3.3395 lr=8.68e-05 tok=12.643B dt=9960 wait=107 +15:30:36 step 96480 loss=3.2377 lr=8.68e-05 tok=12.646B dt=10035 wait=108 +15:31:52 step 96500 loss=3.2956 lr=8.67e-05 tok=12.648B dt=10111 wait=108 +15:33:08 step 96520 loss=3.2579 lr=8.67e-05 tok=12.651B dt=10187 wait=109 +15:34:24 step 96540 loss=3.3514 lr=8.67e-05 tok=12.654B dt=10263 wait=109 +15:35:40 step 96560 loss=3.3552 lr=8.67e-05 tok=12.656B dt=10339 wait=110 +15:36:55 step 96580 loss=3.3642 lr=8.67e-05 tok=12.659B dt=10415 wait=110 +15:38:11 step 96600 loss=3.2903 lr=8.66e-05 tok=12.662B dt=10491 wait=111 +15:38:28 pushed ckpt_w57.pt,log_w57.txt +15:40:14 waited 106s for global r=275 +15:40:27 global r=275 load 4.228 -> 4.214 +15:41:49 step 96620 loss=3.2830 lr=8.66e-05 tok=12.664B dt=10708 wait=111 +15:43:05 step 96640 loss=3.2140 lr=8.66e-05 tok=12.667B dt=10784 wait=112 +15:44:20 step 96660 loss=3.2598 lr=8.66e-05 tok=12.669B dt=10859 wait=112 +15:45:36 step 96680 loss=3.2801 lr=8.65e-05 tok=12.672B dt=10935 wait=113 +15:46:52 step 96700 loss=3.3836 lr=8.65e-05 tok=12.675B dt=11011 wait=113 +15:48:08 step 96720 loss=3.3254 lr=8.65e-05 tok=12.677B dt=11087 wait=113 +15:49:23 step 96740 loss=3.2701 lr=8.65e-05 tok=12.680B dt=11162 wait=114 +15:50:14 pushed ckpt_w57.pt,log_w57.txt +15:51:45 waited 91s for global r=276 +15:51:57 global r=276 load 4.232 -> 4.217 +15:52:43 step 96760 loss=3.3362 lr=8.65e-05 tok=12.683B dt=11362 wait=114 +15:53:59 step 96780 loss=3.2326 lr=8.64e-05 tok=12.685B dt=11438 wait=115 +15:55:14 step 96800 loss=3.2406 lr=8.64e-05 tok=12.688B dt=11513 wait=115 +15:56:30 step 96820 loss=3.2227 lr=8.64e-05 tok=12.690B dt=11589 wait=116 +15:57:46 step 96840 loss=3.3544 lr=8.64e-05 tok=12.693B dt=11665 wait=116 +15:59:02 step 96860 loss=3.3639 lr=8.64e-05 tok=12.696B dt=11741 wait=117 +16:00:18 step 96880 loss=3.4340 lr=8.63e-05 tok=12.698B dt=11817 wait=117 +16:01:34 step 96900 loss=3.3452 lr=8.63e-05 tok=12.701B dt=11893 wait=118 +16:01:53 pushed ckpt_w57.pt,log_w57.txt +16:06:16 step 96920 loss=3.3788 lr=8.63e-05 tok=12.703B dt=12175 wait=118 +16:07:32 step 96940 loss=3.4052 lr=8.63e-05 tok=12.706B dt=12251 wait=119 +16:08:47 step 96960 loss=3.4610 lr=8.63e-05 tok=12.709B dt=12326 wait=119 +16:10:03 step 96980 loss=3.3442 lr=8.62e-05 tok=12.711B dt=12402 wait=120 +16:11:19 step 97000 loss=3.3296 lr=8.62e-05 tok=12.714B dt=12478 wait=120 +16:12:36 step 97020 loss=3.3500 lr=8.62e-05 tok=12.717B dt=12555 wait=121 +16:13:52 step 97040 loss=3.2917 lr=8.62e-05 tok=12.719B dt=12631 wait=121 +16:14:44 pushed ckpt_w57.pt,log_w57.txt +16:14:45 waited 0s for global r=277 +16:15:00 global r=277 load 4.241 -> 4.219 +16:15:46 step 97060 loss=3.3167 lr=8.61e-05 tok=12.722B dt=12745 wait=122 +16:17:02 step 97080 loss=3.2400 lr=8.61e-05 tok=12.724B dt=12821 wait=122 +16:18:18 step 97100 loss=3.2601 lr=8.61e-05 tok=12.727B dt=12897 wait=123 +launch +16:19:37 name=w57 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 4/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:19:43 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:19:43 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:19:56 params 1.08B +16:20:03 resumed step=97050 seen=12.72B round=277 +16:20:03 stream salt=97050 +16:22:41 start: accum=4, 131072 tokens per optimiser step +16:24:25 step 97051 loss=3.5000 lr=8.62e-05 tok=12.721B dt=105 wait=82 +16:24:31 step 97052 loss=3.3987 lr=8.62e-05 tok=12.721B dt=110 wait=82 +16:25:13 step 97060 loss=3.6581 lr=8.61e-05 tok=12.722B dt=152 wait=83 +16:26:54 step 97080 loss=3.5223 lr=8.61e-05 tok=12.724B dt=253 wait=84 +16:28:12 step 97100 loss=3.5814 lr=8.61e-05 tok=12.727B dt=331 wait=84 +16:29:28 step 97120 loss=3.5578 lr=8.61e-05 tok=12.730B dt=407 wait=84 +16:30:44 step 97140 loss=3.5992 lr=8.61e-05 tok=12.732B dt=483 wait=85 +16:32:00 step 97160 loss=3.7254 lr=8.60e-05 tok=12.735B dt=560 wait=85 +16:33:17 step 97180 loss=3.5259 lr=8.60e-05 tok=12.738B dt=636 wait=86 +16:34:33 step 97200 loss=3.6002 lr=8.60e-05 tok=12.740B dt=712 wait=86 +16:34:51 pushed ckpt_w57.pt,log_w57.txt +16:36:36 waited 106s for global r=278 +16:36:56 global r=278 load 4.252 -> 4.230 +16:38:19 step 97220 loss=3.4875 lr=8.60e-05 tok=12.743B dt=938 wait=87 +16:39:35 step 97240 loss=3.5358 lr=8.60e-05 tok=12.745B dt=1014 wait=87 +16:40:51 step 97260 loss=3.5294 lr=8.59e-05 tok=12.748B dt=1091 wait=88 +16:42:07 step 97280 loss=3.4673 lr=8.59e-05 tok=12.751B dt=1167 wait=88 +launch +16:42:35 name=w57 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 4/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:42:41 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:42:41 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:42:54 params 1.08B +16:43:01 resumed step=97200 seen=12.74B round=278 +16:43:01 stream salt=97200 +16:45:24 start: accum=4, 131072 tokens per optimiser step +16:48:53 step 97201 loss=3.3956 lr=8.60e-05 tok=12.740B dt=209 wait=187 +16:48:58 step 97202 loss=3.4025 lr=8.60e-05 tok=12.740B dt=214 wait=187 +16:50:30 step 97220 loss=3.6355 lr=8.60e-05 tok=12.743B dt=306 wait=187 +16:51:54 step 97240 loss=3.6093 lr=8.60e-05 tok=12.745B dt=390 wait=188 +16:53:11 step 97260 loss=3.6236 lr=8.59e-05 tok=12.748B dt=467 wait=188 +16:54:27 step 97280 loss=3.6070 lr=8.59e-05 tok=12.751B dt=543 wait=188 +16:55:45 step 97300 loss=3.7296 lr=8.59e-05 tok=12.753B dt=621 wait=189 +16:57:02 step 97320 loss=3.6424 lr=8.59e-05 tok=12.756B dt=698 wait=189 +16:58:19 step 97340 loss=3.6200 lr=8.59e-05 tok=12.759B dt=774 wait=190 +16:59:24 pushed ckpt_w57.pt,log_w57.txt +17:03:13 step 97360 loss=3.6067 lr=8.58e-05 tok=12.761B dt=1069 wait=190 +17:04:30 step 97380 loss=3.6399 lr=8.58e-05 tok=12.764B dt=1146 wait=190 +17:05:47 step 97400 loss=3.5819 lr=8.58e-05 tok=12.766B dt=1223 wait=191 +17:07:03 step 97420 loss=3.6565 lr=8.58e-05 tok=12.769B dt=1299 wait=191 +17:08:20 step 97440 loss=3.4634 lr=8.57e-05 tok=12.772B dt=1376 wait=191 +17:09:37 step 97460 loss=3.4949 lr=8.57e-05 tok=12.774B dt=1453 wait=192 +17:10:54 step 97480 loss=3.5967 lr=8.57e-05 tok=12.777B dt=1530 wait=192 +17:12:11 step 97500 loss=3.4795 lr=8.57e-05 tok=12.780B dt=1607 wait=193 +17:12:34 pushed ckpt_w57.pt,log_w57.txt +17:12:34 waited 0s for global r=279 +17:12:52 global r=279 load 4.264 -> 4.251 +17:14:14 step 97520 loss=3.5413 lr=8.57e-05 tok=12.782B dt=1730 wait=193 +17:15:32 step 97540 loss=3.6765 lr=8.56e-05 tok=12.785B dt=1808 wait=193 +17:16:49 step 97560 loss=3.5605 lr=8.56e-05 tok=12.787B dt=1885 wait=194 +17:18:06 step 97580 loss=3.5346 lr=8.56e-05 tok=12.790B dt=1962 wait=194 +17:19:23 step 97600 loss=3.5048 lr=8.56e-05 tok=12.793B dt=2039 wait=195 +17:20:40 step 97620 loss=3.6104 lr=8.56e-05 tok=12.795B dt=2116 wait=195 +17:21:57 step 97640 loss=3.5150 lr=8.55e-05 tok=12.798B dt=2193 wait=195 diff --git a/log_w58.txt b/log_w58.txt new file mode 100644 index 0000000000000000000000000000000000000000..e1fecf6941fe39ec42496864174a4dc26679aba8 --- /dev/null +++ b/log_w58.txt @@ -0,0 +1,270 @@ +launch +12:41:04 name=w58 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 5/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:41:10 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:41:20 params 1.08B +12:41:21 resumed step=94350 seen=12.37B round=260 +12:41:21 stream salt=94350 +12:43:17 start: accum=4, 131072 tokens per optimiser step +12:44:56 step 94351 loss=3.0195 lr=8.90e-05 tok=12.367B dt=99 wait=68 +12:45:01 step 94352 loss=3.1584 lr=8.90e-05 tok=12.367B dt=104 wait=68 +12:45:39 step 94360 loss=3.3123 lr=8.90e-05 tok=12.368B dt=142 wait=68 +12:47:11 step 94380 loss=3.2770 lr=8.90e-05 tok=12.371B dt=233 wait=69 +12:48:26 step 94400 loss=3.3852 lr=8.90e-05 tok=12.373B dt=309 wait=69 +12:49:41 step 94420 loss=3.3198 lr=8.89e-05 tok=12.376B dt=384 wait=70 +12:50:56 step 94440 loss=3.2636 lr=8.89e-05 tok=12.378B dt=459 wait=70 +12:52:11 step 94460 loss=3.2072 lr=8.89e-05 tok=12.381B dt=534 wait=70 +12:53:26 step 94480 loss=3.2354 lr=8.89e-05 tok=12.384B dt=609 wait=71 +12:54:42 step 94500 loss=3.3283 lr=8.89e-05 tok=12.386B dt=685 wait=71 +12:54:56 pushed ckpt_w58.pt,log_w58.txt +12:56:42 waited 106s for global r=261 +12:56:55 global r=261 load 4.240 -> 4.218 +12:58:18 step 94520 loss=3.2743 lr=8.88e-05 tok=12.389B dt=901 wait=72 +12:59:33 step 94540 loss=3.2123 lr=8.88e-05 tok=12.392B dt=976 wait=72 +13:00:48 step 94560 loss=3.2546 lr=8.88e-05 tok=12.394B dt=1051 wait=72 +13:02:03 step 94580 loss=3.3336 lr=8.88e-05 tok=12.397B dt=1126 wait=73 +13:03:19 step 94600 loss=3.4229 lr=8.87e-05 tok=12.399B dt=1202 wait=73 +13:04:34 step 94620 loss=3.3940 lr=8.87e-05 tok=12.402B dt=1277 wait=74 +13:05:50 step 94640 loss=3.3090 lr=8.87e-05 tok=12.405B dt=1353 wait=74 +13:06:43 pushed ckpt_w58.pt,log_w58.txt +13:08:29 waited 106s for global r=262 +13:08:37 global r=262 load 4.238 -> 4.222 +13:09:18 step 94660 loss=3.3560 lr=8.87e-05 tok=12.407B dt=1561 wait=74 +13:10:33 step 94680 loss=3.2254 lr=8.87e-05 tok=12.410B dt=1636 wait=75 +13:11:48 step 94700 loss=3.2708 lr=8.86e-05 tok=12.413B dt=1711 wait=75 +13:13:03 step 94720 loss=3.2354 lr=8.86e-05 tok=12.415B dt=1786 wait=75 +13:14:18 step 94740 loss=3.3217 lr=8.86e-05 tok=12.418B dt=1861 wait=76 +13:15:33 step 94760 loss=3.3291 lr=8.86e-05 tok=12.420B dt=1936 wait=76 +13:16:48 step 94780 loss=3.3672 lr=8.86e-05 tok=12.423B dt=2011 wait=77 +13:18:04 step 94800 loss=3.3175 lr=8.85e-05 tok=12.426B dt=2086 wait=77 +13:18:15 pushed ckpt_w58.pt,log_w58.txt +13:19:46 waited 91s for global r=263 +13:19:56 global r=263 load 4.237 -> 4.216 +13:21:18 step 94820 loss=3.2803 lr=8.85e-05 tok=12.428B dt=2281 wait=77 +13:22:33 step 94840 loss=3.2181 lr=8.85e-05 tok=12.431B dt=2356 wait=78 +13:23:48 step 94860 loss=3.2442 lr=8.85e-05 tok=12.433B dt=2430 wait=78 +13:25:03 step 94880 loss=3.3268 lr=8.85e-05 tok=12.436B dt=2506 wait=79 +13:26:18 step 94900 loss=3.4051 lr=8.84e-05 tok=12.439B dt=2581 wait=79 +13:27:33 step 94920 loss=3.3356 lr=8.84e-05 tok=12.441B dt=2656 wait=79 +13:28:48 step 94940 loss=3.2631 lr=8.84e-05 tok=12.444B dt=2731 wait=80 +13:29:38 pushed ckpt_w58.pt,log_w58.txt +13:31:39 waited 121s for global r=264 +13:31:46 global r=264 load 4.240 -> 4.220 +13:32:30 step 94960 loss=3.3368 lr=8.84e-05 tok=12.447B dt=2952 wait=80 +13:33:45 step 94980 loss=3.2155 lr=8.83e-05 tok=12.449B dt=3028 wait=81 +13:35:00 step 95000 loss=3.3075 lr=8.83e-05 tok=12.452B dt=3103 wait=81 +13:36:15 step 95020 loss=3.2814 lr=8.83e-05 tok=12.454B dt=3178 wait=81 +13:37:30 step 95040 loss=3.3619 lr=8.83e-05 tok=12.457B dt=3253 wait=82 +13:38:46 step 95060 loss=3.3802 lr=8.83e-05 tok=12.460B dt=3328 wait=82 +13:40:01 step 95080 loss=3.3905 lr=8.82e-05 tok=12.462B dt=3404 wait=82 +13:41:16 step 95100 loss=3.3053 lr=8.82e-05 tok=12.465B dt=3479 wait=83 +13:41:29 pushed ckpt_w58.pt,log_w58.txt +13:42:59 waited 91s for global r=265 +13:43:13 global r=265 load 4.238 -> 4.216 +13:44:32 step 95120 loss=3.2372 lr=8.82e-05 tok=12.468B dt=3675 wait=83 +13:45:47 step 95140 loss=3.2142 lr=8.82e-05 tok=12.470B dt=3750 wait=84 +13:47:02 step 95160 loss=3.2527 lr=8.82e-05 tok=12.473B dt=3825 wait=84 +13:48:17 step 95180 loss=3.2859 lr=8.81e-05 tok=12.475B dt=3900 wait=84 +13:49:33 step 95200 loss=3.4087 lr=8.81e-05 tok=12.478B dt=3975 wait=85 +13:50:48 step 95220 loss=3.3589 lr=8.81e-05 tok=12.481B dt=4051 wait=85 +13:52:03 step 95240 loss=3.2992 lr=8.81e-05 tok=12.483B dt=4126 wait=86 +13:52:54 pushed ckpt_w58.pt,log_w58.txt +13:54:40 waited 106s for global r=266 +13:54:54 global r=266 load 4.234 -> 4.215 +13:55:37 step 95260 loss=3.3709 lr=8.81e-05 tok=12.486B dt=4340 wait=86 +13:56:52 step 95280 loss=3.2243 lr=8.80e-05 tok=12.489B dt=4415 wait=86 +13:58:07 step 95300 loss=3.2510 lr=8.80e-05 tok=12.491B dt=4490 wait=87 +13:59:22 step 95320 loss=3.2471 lr=8.80e-05 tok=12.494B dt=4565 wait=87 +14:00:37 step 95340 loss=3.3665 lr=8.80e-05 tok=12.496B dt=4640 wait=87 +14:01:53 step 95360 loss=3.3434 lr=8.79e-05 tok=12.499B dt=4716 wait=88 +14:03:08 step 95380 loss=3.3710 lr=8.79e-05 tok=12.502B dt=4791 wait=88 +14:04:23 step 95400 loss=3.3155 lr=8.79e-05 tok=12.504B dt=4866 wait=89 +14:04:35 pushed ckpt_w58.pt,log_w58.txt +14:06:05 waited 91s for global r=267 +14:06:13 global r=267 load 4.234 -> 4.219 +14:07:36 step 95420 loss=3.3153 lr=8.79e-05 tok=12.507B dt=5059 wait=89 +14:08:51 step 95440 loss=3.2432 lr=8.79e-05 tok=12.510B dt=5134 wait=89 +14:10:06 step 95460 loss=3.2437 lr=8.78e-05 tok=12.512B dt=5209 wait=90 +14:11:21 step 95480 loss=3.2874 lr=8.78e-05 tok=12.515B dt=5284 wait=90 +14:12:37 step 95500 loss=3.3788 lr=8.78e-05 tok=12.517B dt=5359 wait=90 +14:13:52 step 95520 loss=3.3141 lr=8.78e-05 tok=12.520B dt=5435 wait=91 +14:15:07 step 95540 loss=3.2749 lr=8.78e-05 tok=12.523B dt=5510 wait=91 +14:15:56 pushed ckpt_w58.pt,log_w58.txt +14:17:27 waited 91s for global r=268 +14:17:40 global r=268 load 4.236 -> 4.216 +14:18:21 step 95560 loss=3.3472 lr=8.77e-05 tok=12.525B dt=5704 wait=92 +14:19:36 step 95580 loss=3.2393 lr=8.77e-05 tok=12.528B dt=5779 wait=92 +14:20:51 step 95600 loss=3.2996 lr=8.77e-05 tok=12.530B dt=5854 wait=92 +14:22:07 step 95620 loss=3.2524 lr=8.77e-05 tok=12.533B dt=5929 wait=93 +14:23:22 step 95640 loss=3.3643 lr=8.76e-05 tok=12.536B dt=6005 wait=93 +14:24:37 step 95660 loss=3.3570 lr=8.76e-05 tok=12.538B dt=6080 wait=94 +14:25:52 step 95680 loss=3.3904 lr=8.76e-05 tok=12.541B dt=6155 wait=94 +14:27:08 step 95700 loss=3.3162 lr=8.76e-05 tok=12.544B dt=6231 wait=94 +14:27:22 pushed ckpt_w58.pt,log_w58.txt +14:29:07 waited 106s for global r=269 +14:29:16 global r=269 load 4.234 -> 4.216 +14:30:35 step 95720 loss=3.3117 lr=8.76e-05 tok=12.546B dt=6438 wait=95 +14:31:50 step 95740 loss=3.2713 lr=8.75e-05 tok=12.549B dt=6513 wait=95 +14:33:05 step 95760 loss=3.2879 lr=8.75e-05 tok=12.551B dt=6588 wait=96 +14:34:20 step 95780 loss=3.2517 lr=8.75e-05 tok=12.554B dt=6663 wait=96 +14:35:35 step 95800 loss=3.3796 lr=8.75e-05 tok=12.557B dt=6738 wait=96 +14:36:50 step 95820 loss=3.2973 lr=8.75e-05 tok=12.559B dt=6813 wait=97 +14:38:05 step 95840 loss=3.2634 lr=8.74e-05 tok=12.562B dt=6888 wait=97 +14:38:58 pushed ckpt_w58.pt,log_w58.txt +14:40:44 waited 106s for global r=270 +14:40:51 global r=270 load 4.236 -> 4.212 +14:41:34 step 95860 loss=3.3515 lr=8.74e-05 tok=12.565B dt=7097 wait=97 +14:42:50 step 95880 loss=3.2524 lr=8.74e-05 tok=12.567B dt=7173 wait=98 +14:44:05 step 95900 loss=3.2753 lr=8.74e-05 tok=12.570B dt=7248 wait=98 +14:45:20 step 95920 loss=3.2238 lr=8.74e-05 tok=12.572B dt=7323 wait=99 +14:46:35 step 95940 loss=3.3324 lr=8.73e-05 tok=12.575B dt=7398 wait=99 +14:47:50 step 95960 loss=3.3185 lr=8.73e-05 tok=12.578B dt=7473 wait=99 +14:49:05 step 95980 loss=3.3244 lr=8.73e-05 tok=12.580B dt=7548 wait=100 +14:50:20 step 96000 loss=3.2804 lr=8.73e-05 tok=12.583B dt=7623 wait=100 +14:50:32 pushed ckpt_w58.pt,log_w58.txt +14:52:48 waited 136s for global r=271 +14:53:00 global r=271 load 4.230 -> 4.216 +14:54:23 step 96020 loss=3.2560 lr=8.72e-05 tok=12.586B dt=7866 wait=101 +14:55:38 step 96040 loss=3.2081 lr=8.72e-05 tok=12.588B dt=7941 wait=101 +14:56:53 step 96060 loss=3.2710 lr=8.72e-05 tok=12.591B dt=8016 wait=101 +14:58:08 step 96080 loss=3.3000 lr=8.72e-05 tok=12.593B dt=8091 wait=102 +14:59:23 step 96100 loss=3.3818 lr=8.72e-05 tok=12.596B dt=8166 wait=102 +15:00:39 step 96120 loss=3.3134 lr=8.71e-05 tok=12.599B dt=8242 wait=103 +15:01:54 step 96140 loss=3.3012 lr=8.71e-05 tok=12.601B dt=8317 wait=103 +15:02:44 pushed ckpt_w58.pt,log_w58.txt +15:04:30 waited 106s for global r=272 +15:04:41 global r=272 load 4.230 -> 4.219 +15:05:25 step 96160 loss=3.3638 lr=8.71e-05 tok=12.604B dt=8528 wait=103 +15:06:40 step 96180 loss=3.2385 lr=8.71e-05 tok=12.607B dt=8603 wait=104 +15:07:56 step 96200 loss=3.2993 lr=8.71e-05 tok=12.609B dt=8678 wait=104 +15:09:11 step 96220 loss=3.2323 lr=8.70e-05 tok=12.612B dt=8754 wait=104 +15:10:26 step 96240 loss=3.3183 lr=8.70e-05 tok=12.614B dt=8829 wait=105 +15:11:41 step 96260 loss=3.3354 lr=8.70e-05 tok=12.617B dt=8904 wait=105 +15:12:56 step 96280 loss=3.4005 lr=8.70e-05 tok=12.620B dt=8979 wait=106 +15:14:12 step 96300 loss=3.3199 lr=8.70e-05 tok=12.622B dt=9054 wait=106 +15:14:27 pushed ckpt_w58.pt,log_w58.txt +15:16:28 waited 121s for global r=273 +15:16:41 global r=273 load 4.233 -> 4.214 +15:18:06 step 96320 loss=3.3085 lr=8.69e-05 tok=12.625B dt=9289 wait=106 +15:19:21 step 96340 loss=3.2254 lr=8.69e-05 tok=12.627B dt=9364 wait=107 +15:20:36 step 96360 loss=3.2855 lr=8.69e-05 tok=12.630B dt=9439 wait=107 +15:21:51 step 96380 loss=3.2707 lr=8.69e-05 tok=12.633B dt=9514 wait=108 +15:23:06 step 96400 loss=3.3680 lr=8.68e-05 tok=12.635B dt=9589 wait=108 +15:24:22 step 96420 loss=3.3004 lr=8.68e-05 tok=12.638B dt=9665 wait=108 +15:25:37 step 96440 loss=3.2757 lr=8.68e-05 tok=12.641B dt=9740 wait=109 +15:26:28 pushed ckpt_w58.pt,log_w58.txt +15:28:18 waited 111s for global r=274 +15:28:26 global r=274 load 4.229 -> 4.219 +15:29:09 step 96460 loss=3.3373 lr=8.68e-05 tok=12.643B dt=9952 wait=109 +15:30:24 step 96480 loss=3.2454 lr=8.68e-05 tok=12.646B dt=10027 wait=110 +15:31:39 step 96500 loss=3.2912 lr=8.67e-05 tok=12.648B dt=10102 wait=110 +15:32:55 step 96520 loss=3.2576 lr=8.67e-05 tok=12.651B dt=10178 wait=110 +15:34:10 step 96540 loss=3.3748 lr=8.67e-05 tok=12.654B dt=10253 wait=111 +15:35:25 step 96560 loss=3.3476 lr=8.67e-05 tok=12.656B dt=10328 wait=111 +15:36:40 step 96580 loss=3.3506 lr=8.67e-05 tok=12.659B dt=10403 wait=111 +15:37:55 step 96600 loss=3.2795 lr=8.66e-05 tok=12.662B dt=10478 wait=112 +15:38:11 pushed ckpt_w58.pt,log_w58.txt +15:40:12 waited 121s for global r=275 +15:40:24 global r=275 load 4.229 -> 4.214 +15:41:48 step 96620 loss=3.2484 lr=8.66e-05 tok=12.664B dt=10711 wait=112 +15:43:03 step 96640 loss=3.2402 lr=8.66e-05 tok=12.667B dt=10786 wait=113 +15:44:18 step 96660 loss=3.2408 lr=8.66e-05 tok=12.669B dt=10861 wait=113 +15:45:33 step 96680 loss=3.2931 lr=8.65e-05 tok=12.672B dt=10936 wait=113 +15:46:49 step 96700 loss=3.4317 lr=8.65e-05 tok=12.675B dt=11012 wait=114 +15:48:04 step 96720 loss=3.3721 lr=8.65e-05 tok=12.677B dt=11087 wait=114 +15:49:20 step 96740 loss=3.2900 lr=8.65e-05 tok=12.680B dt=11162 wait=115 +15:50:11 pushed ckpt_w58.pt,log_w58.txt +15:51:42 waited 91s for global r=276 +15:51:51 global r=276 load 4.235 -> 4.217 +15:52:31 step 96760 loss=3.3494 lr=8.65e-05 tok=12.683B dt=11354 wait=115 +15:53:46 step 96780 loss=3.2104 lr=8.64e-05 tok=12.685B dt=11429 wait=115 +15:55:01 step 96800 loss=3.2160 lr=8.64e-05 tok=12.688B dt=11504 wait=116 +15:56:17 step 96820 loss=3.2568 lr=8.64e-05 tok=12.690B dt=11580 wait=116 +15:57:32 step 96840 loss=3.3745 lr=8.64e-05 tok=12.693B dt=11655 wait=117 +15:58:47 step 96860 loss=3.3468 lr=8.64e-05 tok=12.696B dt=11730 wait=117 +16:00:02 step 96880 loss=3.3782 lr=8.63e-05 tok=12.698B dt=11805 wait=117 +16:01:18 step 96900 loss=3.3180 lr=8.63e-05 tok=12.701B dt=11880 wait=118 +16:01:29 pushed ckpt_w58.pt,log_w58.txt +16:05:53 step 96920 loss=3.3637 lr=8.63e-05 tok=12.703B dt=12155 wait=118 +16:07:08 step 96940 loss=3.3932 lr=8.63e-05 tok=12.706B dt=12231 wait=119 +16:08:23 step 96960 loss=3.4470 lr=8.63e-05 tok=12.709B dt=12306 wait=119 +16:09:38 step 96980 loss=3.3513 lr=8.62e-05 tok=12.711B dt=12381 wait=119 +16:10:54 step 97000 loss=3.3508 lr=8.62e-05 tok=12.714B dt=12457 wait=120 +16:12:09 step 97020 loss=3.3359 lr=8.62e-05 tok=12.717B dt=12532 wait=120 +16:13:24 step 97040 loss=3.2987 lr=8.62e-05 tok=12.719B dt=12607 wait=120 +16:14:18 pushed ckpt_w58.pt,log_w58.txt +16:14:18 waited 0s for global r=277 +16:14:34 global r=277 load 4.228 -> 4.219 +16:15:18 step 97060 loss=3.3322 lr=8.61e-05 tok=12.722B dt=12721 wait=121 +16:16:33 step 97080 loss=3.2344 lr=8.61e-05 tok=12.724B dt=12796 wait=121 +16:17:48 step 97100 loss=3.2838 lr=8.61e-05 tok=12.727B dt=12871 wait=122 +16:19:03 step 97120 loss=3.2340 lr=8.61e-05 tok=12.730B dt=12946 wait=122 +launch +16:20:20 name=w58 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 5/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:20:24 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:20:24 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:20:34 params 1.08B +16:20:38 resumed step=97050 seen=12.72B round=277 +16:20:38 stream salt=97050 +16:23:13 start: accum=4, 131072 tokens per optimiser step +16:25:12 step 97051 loss=3.2314 lr=8.62e-05 tok=12.721B dt=119 wait=102 +16:25:17 step 97052 loss=3.2366 lr=8.62e-05 tok=12.721B dt=124 wait=102 +16:25:54 step 97060 loss=3.6086 lr=8.61e-05 tok=12.722B dt=161 wait=102 +16:27:22 step 97080 loss=3.5509 lr=8.61e-05 tok=12.724B dt=249 wait=103 +16:28:38 step 97100 loss=3.6011 lr=8.61e-05 tok=12.727B dt=325 wait=103 +16:29:54 step 97120 loss=3.5359 lr=8.61e-05 tok=12.730B dt=401 wait=103 +16:31:10 step 97140 loss=3.6186 lr=8.61e-05 tok=12.732B dt=477 wait=104 +16:32:26 step 97160 loss=3.6958 lr=8.60e-05 tok=12.735B dt=553 wait=104 +16:33:42 step 97180 loss=3.5889 lr=8.60e-05 tok=12.738B dt=629 wait=105 +16:34:58 step 97200 loss=3.5535 lr=8.60e-05 tok=12.740B dt=705 wait=105 +16:35:23 pushed ckpt_w58.pt,log_w58.txt +16:36:39 waited 76s for global r=278 +16:36:54 global r=278 load 4.249 -> 4.230 +16:38:17 step 97220 loss=3.4751 lr=8.60e-05 tok=12.743B dt=904 wait=105 +16:39:32 step 97240 loss=3.5438 lr=8.60e-05 tok=12.745B dt=979 wait=106 +16:40:48 step 97260 loss=3.5675 lr=8.59e-05 tok=12.748B dt=1055 wait=106 +16:42:04 step 97280 loss=3.4807 lr=8.59e-05 tok=12.751B dt=1131 wait=106 +launch +16:42:53 name=w58 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 5/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:42:58 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:42:58 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:43:07 params 1.08B +16:43:11 resumed step=97200 seen=12.74B round=278 +16:43:11 stream salt=97200 +16:45:29 start: accum=4, 131072 tokens per optimiser step +16:48:45 step 97201 loss=3.3547 lr=8.60e-05 tok=12.740B dt=197 wait=183 +16:48:50 step 97202 loss=3.4657 lr=8.60e-05 tok=12.740B dt=201 wait=183 +16:50:10 step 97220 loss=3.6476 lr=8.60e-05 tok=12.743B dt=281 wait=183 +16:51:35 step 97240 loss=3.5956 lr=8.60e-05 tok=12.745B dt=366 wait=184 +16:52:52 step 97260 loss=3.6129 lr=8.59e-05 tok=12.748B dt=443 wait=184 +16:54:08 step 97280 loss=3.5879 lr=8.59e-05 tok=12.751B dt=519 wait=184 +16:55:25 step 97300 loss=3.7240 lr=8.59e-05 tok=12.753B dt=596 wait=185 +16:56:42 step 97320 loss=3.6709 lr=8.59e-05 tok=12.756B dt=673 wait=185 +16:57:58 step 97340 loss=3.5638 lr=8.59e-05 tok=12.759B dt=750 wait=185 +16:58:51 pushed ckpt_w58.pt,log_w58.txt +17:02:36 step 97360 loss=3.6539 lr=8.58e-05 tok=12.761B dt=1028 wait=186 +17:03:53 step 97380 loss=3.6012 lr=8.58e-05 tok=12.764B dt=1104 wait=186 +17:05:09 step 97400 loss=3.4931 lr=8.58e-05 tok=12.766B dt=1181 wait=186 +17:06:26 step 97420 loss=3.6719 lr=8.58e-05 tok=12.769B dt=1257 wait=187 +17:07:42 step 97440 loss=3.5405 lr=8.57e-05 tok=12.772B dt=1333 wait=187 +17:08:58 step 97460 loss=3.5550 lr=8.57e-05 tok=12.774B dt=1409 wait=187 +17:10:15 step 97480 loss=3.5541 lr=8.57e-05 tok=12.777B dt=1486 wait=187 +17:11:31 step 97500 loss=3.4600 lr=8.57e-05 tok=12.780B dt=1562 wait=188 +17:11:47 pushed ckpt_w58.pt,log_w58.txt +17:11:47 waited 0s for global r=279 +17:12:01 global r=279 load 4.254 -> 4.251 +17:13:29 step 97520 loss=3.5268 lr=8.57e-05 tok=12.782B dt=1680 wait=188 +17:14:46 step 97540 loss=3.5874 lr=8.56e-05 tok=12.785B dt=1757 wait=188 +17:16:02 step 97560 loss=3.5670 lr=8.56e-05 tok=12.787B dt=1834 wait=189 +17:17:19 step 97580 loss=3.5490 lr=8.56e-05 tok=12.790B dt=1910 wait=189 +17:18:35 step 97600 loss=3.5230 lr=8.56e-05 tok=12.793B dt=1987 wait=189 +17:19:52 step 97620 loss=3.6354 lr=8.56e-05 tok=12.795B dt=2063 wait=190 +17:21:09 step 97640 loss=3.4606 lr=8.55e-05 tok=12.798B dt=2140 wait=190 +17:22:05 pushed ckpt_w58.pt,log_w58.txt +17:25:51 step 97660 loss=3.5880 lr=8.55e-05 tok=12.800B dt=2422 wait=190 +17:27:07 step 97680 loss=3.4791 lr=8.55e-05 tok=12.803B dt=2498 wait=191 +17:28:23 step 97700 loss=3.5003 lr=8.55e-05 tok=12.806B dt=2574 wait=191 +17:29:39 step 97720 loss=3.4991 lr=8.54e-05 tok=12.808B dt=2651 wait=191 +17:30:56 step 97740 loss=3.4671 lr=8.54e-05 tok=12.811B dt=2727 wait=192 +17:32:12 step 97760 loss=3.4874 lr=8.54e-05 tok=12.814B dt=2804 wait=192 +17:33:29 step 97780 loss=3.4878 lr=8.54e-05 tok=12.816B dt=2880 wait=192 +17:34:46 step 97800 loss=3.6853 lr=8.54e-05 tok=12.819B dt=2957 wait=192 diff --git a/log_w59.txt b/log_w59.txt new file mode 100644 index 0000000000000000000000000000000000000000..9f191745701ca03be733140fdc07fd43c3dd9666 --- /dev/null +++ b/log_w59.txt @@ -0,0 +1,266 @@ +launch +12:41:03 name=w59 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 6/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:41:09 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:41:19 params 1.08B +12:41:20 resumed step=94350 seen=12.37B round=260 +12:41:20 stream salt=94350 +12:43:18 start: accum=4, 131072 tokens per optimiser step +12:44:53 step 94351 loss=2.7831 lr=8.90e-05 tok=12.367B dt=95 wait=61 +12:44:58 step 94352 loss=3.0966 lr=8.90e-05 tok=12.367B dt=100 wait=61 +12:45:36 step 94360 loss=3.2868 lr=8.90e-05 tok=12.368B dt=138 wait=61 +12:47:07 step 94380 loss=3.2592 lr=8.90e-05 tok=12.371B dt=229 wait=62 +12:48:23 step 94400 loss=3.3824 lr=8.90e-05 tok=12.373B dt=305 wait=62 +12:49:39 step 94420 loss=3.3396 lr=8.89e-05 tok=12.376B dt=381 wait=63 +12:50:55 step 94440 loss=3.2980 lr=8.89e-05 tok=12.378B dt=457 wait=63 +12:52:11 step 94460 loss=3.2136 lr=8.89e-05 tok=12.381B dt=533 wait=64 +12:53:27 step 94480 loss=3.2780 lr=8.89e-05 tok=12.384B dt=609 wait=64 +12:54:43 step 94500 loss=3.3253 lr=8.89e-05 tok=12.386B dt=685 wait=65 +12:55:01 pushed ckpt_w59.pt,log_w59.txt +12:56:47 waited 106s for global r=261 +12:56:59 global r=261 load 4.236 -> 4.218 +12:58:19 step 94520 loss=3.2641 lr=8.88e-05 tok=12.389B dt=901 wait=65 +12:59:35 step 94540 loss=3.1888 lr=8.88e-05 tok=12.392B dt=977 wait=66 +13:00:51 step 94560 loss=3.2372 lr=8.88e-05 tok=12.394B dt=1053 wait=66 +13:02:07 step 94580 loss=3.3320 lr=8.88e-05 tok=12.397B dt=1129 wait=67 +13:03:23 step 94600 loss=3.4084 lr=8.87e-05 tok=12.399B dt=1205 wait=67 +13:04:40 step 94620 loss=3.3753 lr=8.87e-05 tok=12.402B dt=1281 wait=67 +13:05:55 step 94640 loss=3.2635 lr=8.87e-05 tok=12.405B dt=1357 wait=68 +13:06:48 pushed ckpt_w59.pt,log_w59.txt +13:08:19 waited 91s for global r=262 +13:08:27 global r=262 load 4.235 -> 4.222 +13:09:10 step 94660 loss=3.3725 lr=8.87e-05 tok=12.407B dt=1552 wait=68 +13:10:26 step 94680 loss=3.2201 lr=8.87e-05 tok=12.410B dt=1628 wait=69 +13:11:42 step 94700 loss=3.3164 lr=8.86e-05 tok=12.413B dt=1703 wait=69 +13:12:58 step 94720 loss=3.2228 lr=8.86e-05 tok=12.415B dt=1779 wait=70 +13:14:13 step 94740 loss=3.3382 lr=8.86e-05 tok=12.418B dt=1855 wait=70 +13:15:29 step 94760 loss=3.3237 lr=8.86e-05 tok=12.420B dt=1931 wait=71 +13:16:45 step 94780 loss=3.3581 lr=8.86e-05 tok=12.423B dt=2007 wait=71 +13:18:01 step 94800 loss=3.2957 lr=8.85e-05 tok=12.426B dt=2083 wait=71 +13:18:14 pushed ckpt_w59.pt,log_w59.txt +13:20:00 waited 106s for global r=263 +13:20:07 global r=263 load 4.232 -> 4.216 +13:21:29 step 94820 loss=3.2956 lr=8.85e-05 tok=12.428B dt=2290 wait=72 +13:22:44 step 94840 loss=3.2420 lr=8.85e-05 tok=12.431B dt=2366 wait=72 +13:24:00 step 94860 loss=3.2655 lr=8.85e-05 tok=12.433B dt=2442 wait=73 +13:25:16 step 94880 loss=3.2964 lr=8.85e-05 tok=12.436B dt=2518 wait=73 +13:26:32 step 94900 loss=3.3903 lr=8.84e-05 tok=12.439B dt=2594 wait=74 +13:27:48 step 94920 loss=3.3399 lr=8.84e-05 tok=12.441B dt=2670 wait=74 +13:29:04 step 94940 loss=3.2521 lr=8.84e-05 tok=12.444B dt=2746 wait=75 +13:29:54 pushed ckpt_w59.pt,log_w59.txt +13:31:40 waited 106s for global r=264 +13:31:46 global r=264 load 4.241 -> 4.220 +13:32:31 step 94960 loss=3.3144 lr=8.84e-05 tok=12.447B dt=2953 wait=75 +13:33:47 step 94980 loss=3.2364 lr=8.83e-05 tok=12.449B dt=3028 wait=76 +13:35:02 step 95000 loss=3.2888 lr=8.83e-05 tok=12.452B dt=3104 wait=76 +13:36:18 step 95020 loss=3.2456 lr=8.83e-05 tok=12.454B dt=3180 wait=76 +13:37:34 step 95040 loss=3.3476 lr=8.83e-05 tok=12.457B dt=3256 wait=77 +13:38:50 step 95060 loss=3.3090 lr=8.83e-05 tok=12.460B dt=3332 wait=77 +13:40:06 step 95080 loss=3.3559 lr=8.82e-05 tok=12.462B dt=3408 wait=78 +13:41:22 step 95100 loss=3.3067 lr=8.82e-05 tok=12.465B dt=3484 wait=78 +13:41:37 pushed ckpt_w59.pt,log_w59.txt +13:43:07 waited 91s for global r=265 +13:43:15 global r=265 load 4.232 -> 4.216 +13:44:35 step 95120 loss=3.2838 lr=8.82e-05 tok=12.468B dt=3676 wait=79 +13:45:50 step 95140 loss=3.2231 lr=8.82e-05 tok=12.470B dt=3752 wait=79 +13:47:06 step 95160 loss=3.2555 lr=8.82e-05 tok=12.473B dt=3828 wait=80 +13:48:22 step 95180 loss=3.3035 lr=8.81e-05 tok=12.475B dt=3904 wait=80 +13:49:38 step 95200 loss=3.4145 lr=8.81e-05 tok=12.478B dt=3980 wait=81 +13:50:54 step 95220 loss=3.3560 lr=8.81e-05 tok=12.481B dt=4056 wait=81 +13:52:10 step 95240 loss=3.2672 lr=8.81e-05 tok=12.483B dt=4132 wait=81 +13:53:00 pushed ckpt_w59.pt,log_w59.txt +13:54:31 waited 91s for global r=266 +13:54:39 global r=266 load 4.234 -> 4.215 +13:55:24 step 95260 loss=3.3466 lr=8.81e-05 tok=12.486B dt=4326 wait=82 +13:56:40 step 95280 loss=3.2545 lr=8.80e-05 tok=12.489B dt=4402 wait=82 +13:57:56 step 95300 loss=3.2864 lr=8.80e-05 tok=12.491B dt=4477 wait=83 +13:59:12 step 95320 loss=3.2661 lr=8.80e-05 tok=12.494B dt=4553 wait=83 +14:00:28 step 95340 loss=3.3506 lr=8.80e-05 tok=12.496B dt=4629 wait=84 +14:01:43 step 95360 loss=3.3360 lr=8.79e-05 tok=12.499B dt=4705 wait=84 +14:02:59 step 95380 loss=3.3421 lr=8.79e-05 tok=12.502B dt=4781 wait=85 +14:04:15 step 95400 loss=3.3187 lr=8.79e-05 tok=12.504B dt=4857 wait=85 +14:04:29 pushed ckpt_w59.pt,log_w59.txt +14:06:15 waited 106s for global r=267 +14:06:28 global r=267 load 4.238 -> 4.219 +14:07:50 step 95420 loss=3.2587 lr=8.79e-05 tok=12.507B dt=5072 wait=85 +14:09:06 step 95440 loss=3.2275 lr=8.79e-05 tok=12.510B dt=5148 wait=86 +14:10:21 step 95460 loss=3.2657 lr=8.78e-05 tok=12.512B dt=5223 wait=86 +14:11:37 step 95480 loss=3.3079 lr=8.78e-05 tok=12.515B dt=5299 wait=87 +14:12:53 step 95500 loss=3.4070 lr=8.78e-05 tok=12.517B dt=5375 wait=87 +14:14:09 step 95520 loss=3.3461 lr=8.78e-05 tok=12.520B dt=5451 wait=87 +14:15:24 step 95540 loss=3.2365 lr=8.78e-05 tok=12.523B dt=5526 wait=88 +14:16:15 pushed ckpt_w59.pt,log_w59.txt +14:17:30 waited 76s for global r=268 +14:17:38 global r=268 load 4.238 -> 4.216 +14:18:22 step 95560 loss=3.3107 lr=8.77e-05 tok=12.525B dt=5704 wait=88 +14:19:37 step 95580 loss=3.2095 lr=8.77e-05 tok=12.528B dt=5779 wait=88 +14:20:53 step 95600 loss=3.2629 lr=8.77e-05 tok=12.530B dt=5854 wait=89 +14:22:08 step 95620 loss=3.2598 lr=8.77e-05 tok=12.533B dt=5930 wait=89 +14:23:24 step 95640 loss=3.3675 lr=8.76e-05 tok=12.536B dt=6006 wait=90 +14:24:40 step 95660 loss=3.3892 lr=8.76e-05 tok=12.538B dt=6082 wait=90 +14:25:56 step 95680 loss=3.3668 lr=8.76e-05 tok=12.541B dt=6158 wait=90 +14:27:12 step 95700 loss=3.2709 lr=8.76e-05 tok=12.544B dt=6233 wait=91 +14:27:25 pushed ckpt_w59.pt,log_w59.txt +14:28:56 waited 91s for global r=269 +14:29:03 global r=269 load 4.242 -> 4.216 +14:30:25 step 95720 loss=3.2869 lr=8.76e-05 tok=12.546B dt=6426 wait=91 +14:31:40 step 95740 loss=3.2164 lr=8.75e-05 tok=12.549B dt=6502 wait=92 +14:32:55 step 95760 loss=3.2364 lr=8.75e-05 tok=12.551B dt=6577 wait=92 +14:34:11 step 95780 loss=3.2789 lr=8.75e-05 tok=12.554B dt=6653 wait=92 +14:35:27 step 95800 loss=3.3736 lr=8.75e-05 tok=12.557B dt=6729 wait=93 +14:36:43 step 95820 loss=3.3616 lr=8.75e-05 tok=12.559B dt=6804 wait=93 +14:37:58 step 95840 loss=3.2733 lr=8.74e-05 tok=12.562B dt=6880 wait=93 +14:38:49 pushed ckpt_w59.pt,log_w59.txt +14:40:50 waited 121s for global r=270 +14:41:03 global r=270 load 4.228 -> 4.212 +14:41:47 step 95860 loss=3.3893 lr=8.74e-05 tok=12.565B dt=7109 wait=94 +14:43:02 step 95880 loss=3.2457 lr=8.74e-05 tok=12.567B dt=7184 wait=94 +14:44:18 step 95900 loss=3.3036 lr=8.74e-05 tok=12.570B dt=7260 wait=95 +14:45:34 step 95920 loss=3.2581 lr=8.74e-05 tok=12.572B dt=7336 wait=95 +14:46:49 step 95940 loss=3.3570 lr=8.73e-05 tok=12.575B dt=7411 wait=95 +14:48:05 step 95960 loss=3.3193 lr=8.73e-05 tok=12.578B dt=7487 wait=96 +14:49:21 step 95980 loss=3.3912 lr=8.73e-05 tok=12.580B dt=7562 wait=96 +14:50:36 step 96000 loss=3.2765 lr=8.73e-05 tok=12.583B dt=7638 wait=97 +14:50:51 pushed ckpt_w59.pt,log_w59.txt +14:52:37 waited 106s for global r=271 +14:52:46 global r=271 load 4.231 -> 4.216 +14:54:09 step 96020 loss=3.2582 lr=8.72e-05 tok=12.586B dt=7850 wait=97 +14:55:24 step 96040 loss=3.2353 lr=8.72e-05 tok=12.588B dt=7926 wait=97 +14:56:40 step 96060 loss=3.2676 lr=8.72e-05 tok=12.591B dt=8002 wait=98 +14:57:56 step 96080 loss=3.2883 lr=8.72e-05 tok=12.593B dt=8078 wait=98 +14:59:12 step 96100 loss=3.3745 lr=8.72e-05 tok=12.596B dt=8154 wait=99 +15:00:28 step 96120 loss=3.3560 lr=8.71e-05 tok=12.599B dt=8229 wait=99 +15:01:43 step 96140 loss=3.2977 lr=8.71e-05 tok=12.601B dt=8305 wait=100 +15:02:36 pushed ckpt_w59.pt,log_w59.txt +15:04:37 waited 121s for global r=272 +15:04:48 global r=272 load 4.238 -> 4.219 +15:05:30 step 96160 loss=3.3239 lr=8.71e-05 tok=12.604B dt=8532 wait=100 +15:06:46 step 96180 loss=3.2345 lr=8.71e-05 tok=12.607B dt=8608 wait=100 +15:08:02 step 96200 loss=3.3055 lr=8.71e-05 tok=12.609B dt=8683 wait=101 +15:09:17 step 96220 loss=3.2546 lr=8.70e-05 tok=12.612B dt=8759 wait=101 +15:10:33 step 96240 loss=3.3475 lr=8.70e-05 tok=12.614B dt=8835 wait=102 +15:11:49 step 96260 loss=3.3308 lr=8.70e-05 tok=12.617B dt=8911 wait=102 +15:13:05 step 96280 loss=3.3754 lr=8.70e-05 tok=12.620B dt=8986 wait=103 +15:14:20 step 96300 loss=3.2917 lr=8.70e-05 tok=12.622B dt=9062 wait=103 +15:14:35 pushed ckpt_w59.pt,log_w59.txt +15:16:36 waited 121s for global r=273 +15:16:47 global r=273 load 4.231 -> 4.214 +15:18:10 step 96320 loss=3.2578 lr=8.69e-05 tok=12.625B dt=9292 wait=104 +15:19:26 step 96340 loss=3.2337 lr=8.69e-05 tok=12.627B dt=9368 wait=104 +15:20:42 step 96360 loss=3.2398 lr=8.69e-05 tok=12.630B dt=9444 wait=105 +15:21:59 step 96380 loss=3.2926 lr=8.69e-05 tok=12.633B dt=9520 wait=105 +15:23:15 step 96400 loss=3.4186 lr=8.68e-05 tok=12.635B dt=9597 wait=106 +15:24:31 step 96420 loss=3.3246 lr=8.68e-05 tok=12.638B dt=9673 wait=106 +15:25:48 step 96440 loss=3.2702 lr=8.68e-05 tok=12.641B dt=9749 wait=107 +15:26:42 pushed ckpt_w59.pt,log_w59.txt +15:28:13 waited 91s for global r=274 +15:28:23 global r=274 load 4.240 -> 4.219 +15:29:08 step 96460 loss=3.3705 lr=8.68e-05 tok=12.643B dt=9950 wait=107 +15:30:24 step 96480 loss=3.2398 lr=8.68e-05 tok=12.646B dt=10026 wait=108 +15:31:41 step 96500 loss=3.2586 lr=8.67e-05 tok=12.648B dt=10102 wait=108 +15:32:57 step 96520 loss=3.2108 lr=8.67e-05 tok=12.651B dt=10179 wait=109 +15:34:13 step 96540 loss=3.3442 lr=8.67e-05 tok=12.654B dt=10255 wait=109 +15:35:29 step 96560 loss=3.3383 lr=8.67e-05 tok=12.656B dt=10331 wait=110 +15:36:46 step 96580 loss=3.3701 lr=8.67e-05 tok=12.659B dt=10408 wait=110 +15:38:02 step 96600 loss=3.3051 lr=8.66e-05 tok=12.662B dt=10484 wait=111 +15:38:17 pushed ckpt_w59.pt,log_w59.txt +15:40:03 waited 106s for global r=275 +15:40:13 global r=275 load 4.233 -> 4.214 +15:41:39 step 96620 loss=3.2691 lr=8.66e-05 tok=12.664B dt=10701 wait=111 +15:42:55 step 96640 loss=3.2414 lr=8.66e-05 tok=12.667B dt=10777 wait=112 +15:44:11 step 96660 loss=3.2263 lr=8.66e-05 tok=12.669B dt=10853 wait=112 +15:45:28 step 96680 loss=3.2658 lr=8.65e-05 tok=12.672B dt=10929 wait=113 +15:46:44 step 96700 loss=3.3545 lr=8.65e-05 tok=12.675B dt=11006 wait=113 +15:48:00 step 96720 loss=3.3292 lr=8.65e-05 tok=12.677B dt=11082 wait=114 +15:49:17 step 96740 loss=3.2910 lr=8.65e-05 tok=12.680B dt=11159 wait=114 +15:50:13 pushed ckpt_w59.pt,log_w59.txt +15:51:44 waited 91s for global r=276 +15:51:55 global r=276 load 4.235 -> 4.217 +15:52:39 step 96760 loss=3.3701 lr=8.65e-05 tok=12.683B dt=11361 wait=115 +15:53:56 step 96780 loss=3.2633 lr=8.64e-05 tok=12.685B dt=11437 wait=115 +15:55:12 step 96800 loss=3.2323 lr=8.64e-05 tok=12.688B dt=11513 wait=116 +15:56:28 step 96820 loss=3.2108 lr=8.64e-05 tok=12.690B dt=11590 wait=116 +15:57:44 step 96840 loss=3.3105 lr=8.64e-05 tok=12.693B dt=11666 wait=117 +launch +16:04:56 name=w59 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 6/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:05:02 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:05:02 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:05:13 params 1.08B +16:05:18 resumed step=96750 seen=12.68B round=276 +16:05:18 stream salt=96750 +16:07:57 start: accum=4, 131072 tokens per optimiser step +16:10:46 step 96751 loss=3.2254 lr=8.65e-05 tok=12.681B dt=169 wait=154 +16:10:50 step 96752 loss=3.2447 lr=8.65e-05 tok=12.681B dt=173 wait=154 +16:11:29 step 96760 loss=3.6457 lr=8.65e-05 tok=12.683B dt=212 wait=155 +16:13:03 step 96780 loss=3.5191 lr=8.64e-05 tok=12.685B dt=306 wait=155 +16:14:30 step 96800 loss=3.5521 lr=8.64e-05 tok=12.688B dt=394 wait=156 +16:15:47 step 96820 loss=3.4838 lr=8.64e-05 tok=12.690B dt=470 wait=156 +16:17:04 step 96840 loss=3.5734 lr=8.64e-05 tok=12.693B dt=547 wait=157 +16:18:21 step 96860 loss=3.6957 lr=8.64e-05 tok=12.696B dt=624 wait=157 +16:19:37 step 96880 loss=3.5079 lr=8.63e-05 tok=12.698B dt=700 wait=158 +16:20:54 step 96900 loss=3.6068 lr=8.63e-05 tok=12.701B dt=777 wait=159 +16:21:10 pushed ckpt_w59.pt,log_w59.txt +16:21:10 waited 0s for global r=277 +16:21:25 global r=277 load 4.239 -> 4.219 +16:22:48 step 96920 loss=3.5393 lr=8.63e-05 tok=12.703B dt=892 wait=159 +16:24:05 step 96940 loss=3.5392 lr=8.63e-05 tok=12.706B dt=968 wait=160 +16:25:22 step 96960 loss=3.5113 lr=8.63e-05 tok=12.709B dt=1045 wait=160 +16:26:38 step 96980 loss=3.4927 lr=8.62e-05 tok=12.711B dt=1122 wait=160 +16:27:55 step 97000 loss=3.5042 lr=8.62e-05 tok=12.714B dt=1198 wait=161 +16:29:12 step 97020 loss=3.5956 lr=8.62e-05 tok=12.717B dt=1275 wait=161 +16:30:28 step 97040 loss=3.5052 lr=8.62e-05 tok=12.719B dt=1352 wait=162 +16:31:20 pushed ckpt_w59.pt,log_w59.txt +16:35:04 step 97060 loss=3.5955 lr=8.61e-05 tok=12.722B dt=1627 wait=162 +16:36:20 step 97080 loss=3.5325 lr=8.61e-05 tok=12.724B dt=1704 wait=163 +16:37:37 step 97100 loss=3.4742 lr=8.61e-05 tok=12.727B dt=1780 wait=163 +16:38:53 step 97120 loss=3.3850 lr=8.61e-05 tok=12.730B dt=1857 wait=164 +16:40:10 step 97140 loss=3.4645 lr=8.61e-05 tok=12.732B dt=1933 wait=164 +16:41:27 step 97160 loss=3.5002 lr=8.60e-05 tok=12.735B dt=2010 wait=165 +16:41:47 mix source 1 lap 1 +16:42:42 step 97180 loss=3.4761 lr=8.60e-05 tok=12.738B dt=2086 wait=165 +launch +16:43:12 name=w59 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 6/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +16:43:17 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +16:43:17 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +16:43:26 params 1.08B +16:43:30 resumed step=97050 seen=12.72B round=277 +16:43:30 stream salt=97050 +16:45:38 start: accum=4, 131072 tokens per optimiser step +16:49:01 step 97051 loss=3.3400 lr=8.62e-05 tok=12.721B dt=202 wait=183 +16:49:05 step 97052 loss=3.1993 lr=8.62e-05 tok=12.721B dt=207 wait=183 +16:49:41 step 97060 loss=3.6641 lr=8.61e-05 tok=12.722B dt=243 wait=183 +16:51:13 step 97080 loss=3.5924 lr=8.61e-05 tok=12.724B dt=334 wait=184 +16:52:30 step 97100 loss=3.6433 lr=8.61e-05 tok=12.727B dt=411 wait=184 +16:53:47 step 97120 loss=3.6007 lr=8.61e-05 tok=12.730B dt=489 wait=185 +16:55:04 step 97140 loss=3.6446 lr=8.61e-05 tok=12.732B dt=566 wait=185 +16:56:21 step 97160 loss=3.7566 lr=8.60e-05 tok=12.735B dt=643 wait=185 +16:57:38 step 97180 loss=3.5938 lr=8.60e-05 tok=12.738B dt=720 wait=186 +16:58:55 step 97200 loss=3.6260 lr=8.60e-05 tok=12.740B dt=797 wait=186 +16:59:30 pushed ckpt_w59.pt,log_w59.txt +16:59:31 waited 0s for global r=278 +16:59:49 global r=278 load 4.268 -> 4.230 +17:01:15 step 97220 loss=3.6347 lr=8.60e-05 tok=12.743B dt=937 wait=186 +17:02:33 step 97240 loss=3.6726 lr=8.60e-05 tok=12.745B dt=1014 wait=187 +17:03:50 step 97260 loss=3.5283 lr=8.59e-05 tok=12.748B dt=1091 wait=187 +17:05:08 step 97280 loss=3.5121 lr=8.59e-05 tok=12.751B dt=1169 wait=187 +17:06:25 step 97300 loss=3.5138 lr=8.59e-05 tok=12.753B dt=1247 wait=188 +17:07:43 step 97320 loss=3.7241 lr=8.59e-05 tok=12.756B dt=1325 wait=188 +17:09:00 step 97340 loss=3.5799 lr=8.59e-05 tok=12.759B dt=1402 wait=188 +17:09:53 pushed ckpt_w59.pt,log_w59.txt +17:09:53 waited 0s for global r=279 +17:10:04 global r=279 load 4.258 -> 4.251 +17:10:47 step 97360 loss=3.6350 lr=8.58e-05 tok=12.761B dt=1509 wait=189 +17:12:04 step 97380 loss=3.5605 lr=8.58e-05 tok=12.764B dt=1586 wait=189 +17:13:22 step 97400 loss=3.5421 lr=8.58e-05 tok=12.766B dt=1663 wait=189 +17:14:39 step 97420 loss=3.6228 lr=8.58e-05 tok=12.769B dt=1741 wait=190 +17:15:56 step 97440 loss=3.5381 lr=8.57e-05 tok=12.772B dt=1818 wait=190 +17:17:13 step 97460 loss=3.6032 lr=8.57e-05 tok=12.774B dt=1895 wait=190 +17:18:31 step 97480 loss=3.4871 lr=8.57e-05 tok=12.777B dt=1972 wait=191 +17:19:48 step 97500 loss=3.5943 lr=8.57e-05 tok=12.780B dt=2050 wait=191 +17:20:01 pushed ckpt_w59.pt,log_w59.txt +17:24:23 step 97520 loss=3.5573 lr=8.57e-05 tok=12.782B dt=2325 wait=191 +17:25:40 step 97540 loss=3.5241 lr=8.56e-05 tok=12.785B dt=2402 wait=192 +17:26:57 step 97560 loss=3.4316 lr=8.56e-05 tok=12.787B dt=2479 wait=192 +17:28:15 step 97580 loss=3.4675 lr=8.56e-05 tok=12.790B dt=2556 wait=193 +17:29:32 step 97600 loss=3.4151 lr=8.56e-05 tok=12.793B dt=2633 wait=193 +17:30:49 step 97620 loss=3.6203 lr=8.56e-05 tok=12.795B dt=2710 wait=193 +17:32:06 step 97640 loss=3.4917 lr=8.55e-05 tok=12.798B dt=2788 wait=194 diff --git a/log_w6.txt b/log_w6.txt new file mode 100644 index 0000000000000000000000000000000000000000..fc893fddffe85e10f62f576f086b14f901f83c91 --- /dev/null +++ b/log_w6.txt @@ -0,0 +1,134 @@ +launch +03:58:11 name=w6 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +03:58:16 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +03:58:25 params 1.08B +03:58:29 resumed step=36600 seen=4.80B round=17 +03:58:29 stream salt=36600 +04:00:00 start: accum=4, 131072 tokens per optimiser step +04:02:20 step 36620 loss=3.6693 lr=2.79e-04 tok=4.800B dt=140 +04:03:37 step 36640 loss=3.8152 lr=2.79e-04 tok=4.802B dt=217 +04:04:52 step 36660 loss=3.7260 lr=2.79e-04 tok=4.805B dt=292 +04:06:08 step 36680 loss=3.8051 lr=2.79e-04 tok=4.808B dt=368 +04:07:24 step 36700 loss=3.6499 lr=2.79e-04 tok=4.810B dt=444 +04:08:40 step 36720 loss=3.6909 lr=2.79e-04 tok=4.813B dt=520 +04:09:56 step 36740 loss=3.7297 lr=2.79e-04 tok=4.816B dt=596 +04:11:12 step 36760 loss=3.7656 lr=2.79e-04 tok=4.818B dt=672 +04:12:27 step 36780 loss=3.6897 lr=2.79e-04 tok=4.821B dt=747 +04:13:43 step 36800 loss=3.7751 lr=2.79e-04 tok=4.823B dt=823 +04:14:59 step 36820 loss=3.7094 lr=2.79e-04 tok=4.826B dt=899 +04:16:15 step 36840 loss=3.6570 lr=2.79e-04 tok=4.829B dt=975 +04:17:30 step 36860 loss=3.7163 lr=2.79e-04 tok=4.831B dt=1050 +04:18:46 step 36880 loss=3.6706 lr=2.79e-04 tok=4.834B dt=1126 +04:20:02 step 36900 loss=3.7805 lr=2.79e-04 tok=4.837B dt=1202 +04:20:15 pushed ckpt_w6.pt,log_w6.txt +04:21:34 step 36920 loss=3.8375 lr=2.79e-04 tok=4.839B dt=1294 +04:22:50 step 36940 loss=3.7724 lr=2.78e-04 tok=4.842B dt=1370 +04:24:06 step 36960 loss=3.7560 lr=2.78e-04 tok=4.844B dt=1446 +04:25:22 step 36980 loss=3.8343 lr=2.78e-04 tok=4.847B dt=1522 +04:26:38 step 37000 loss=3.7640 lr=2.78e-04 tok=4.850B dt=1598 +04:27:54 step 37020 loss=3.7325 lr=2.78e-04 tok=4.852B dt=1674 +04:29:09 step 37040 loss=3.6761 lr=2.78e-04 tok=4.855B dt=1749 +04:30:25 step 37060 loss=3.7249 lr=2.78e-04 tok=4.858B dt=1825 +04:31:41 step 37080 loss=3.6710 lr=2.78e-04 tok=4.860B dt=1901 +04:32:56 step 37100 loss=3.7571 lr=2.78e-04 tok=4.863B dt=1976 +04:34:12 step 37120 loss=3.7065 lr=2.78e-04 tok=4.865B dt=2052 +04:35:28 step 37140 loss=3.6725 lr=2.78e-04 tok=4.868B dt=2128 +04:36:43 step 37160 loss=3.7629 lr=2.78e-04 tok=4.871B dt=2203 +04:37:59 step 37180 loss=3.6179 lr=2.78e-04 tok=4.873B dt=2279 +04:39:15 step 37200 loss=3.7180 lr=2.78e-04 tok=4.876B dt=2355 +04:39:29 pushed ckpt_w6.pt,log_w6.txt +04:40:52 step 37220 loss=3.7501 lr=2.78e-04 tok=4.878B dt=2452 +04:42:07 step 37240 loss=3.7297 lr=2.78e-04 tok=4.881B dt=2527 +04:43:23 step 37260 loss=3.6476 lr=2.78e-04 tok=4.884B dt=2603 +04:44:39 step 37280 loss=3.6331 lr=2.78e-04 tok=4.886B dt=2679 +04:45:55 step 37300 loss=3.6960 lr=2.78e-04 tok=4.889B dt=2754 +04:47:10 step 37320 loss=3.7149 lr=2.78e-04 tok=4.892B dt=2830 +04:48:26 step 37340 loss=3.8510 lr=2.78e-04 tok=4.894B dt=2906 +04:49:42 step 37360 loss=3.8238 lr=2.78e-04 tok=4.897B dt=2982 +04:50:58 step 37380 loss=3.7235 lr=2.78e-04 tok=4.899B dt=3058 +04:52:14 step 37400 loss=3.7033 lr=2.78e-04 tok=4.902B dt=3134 +04:53:30 step 37420 loss=3.7623 lr=2.78e-04 tok=4.905B dt=3210 +04:54:45 step 37440 loss=3.7536 lr=2.78e-04 tok=4.907B dt=3285 +04:56:01 step 37460 loss=3.6709 lr=2.78e-04 tok=4.910B dt=3361 +04:57:17 step 37480 loss=3.7784 lr=2.78e-04 tok=4.913B dt=3437 +04:58:33 step 37500 loss=3.6757 lr=2.78e-04 tok=4.915B dt=3513 +04:58:45 pushed ckpt_w6.pt,log_w6.txt +05:00:06 step 37520 loss=3.7635 lr=2.78e-04 tok=4.918B dt=3606 +05:01:22 step 37540 loss=3.7436 lr=2.78e-04 tok=4.920B dt=3682 +05:02:37 step 37560 loss=3.7507 lr=2.78e-04 tok=4.923B dt=3757 +05:03:53 step 37580 loss=3.8321 lr=2.78e-04 tok=4.926B dt=3833 +05:05:09 step 37600 loss=3.6914 lr=2.78e-04 tok=4.928B dt=3909 +05:06:25 step 37620 loss=3.7086 lr=2.78e-04 tok=4.931B dt=3985 +05:07:41 step 37640 loss=3.8367 lr=2.78e-04 tok=4.934B dt=4061 +05:08:57 step 37660 loss=3.7461 lr=2.78e-04 tok=4.936B dt=4137 +05:10:12 step 37680 loss=3.7175 lr=2.78e-04 tok=4.939B dt=4212 +05:11:28 step 37700 loss=3.7478 lr=2.78e-04 tok=4.941B dt=4288 +05:12:44 step 37720 loss=3.7118 lr=2.78e-04 tok=4.944B dt=4364 +05:14:00 step 37740 loss=3.7454 lr=2.78e-04 tok=4.947B dt=4440 +05:15:15 step 37760 loss=3.6907 lr=2.78e-04 tok=4.949B dt=4515 +05:16:31 step 37780 loss=3.6850 lr=2.78e-04 tok=4.952B dt=4591 +05:17:47 step 37800 loss=3.8341 lr=2.77e-04 tok=4.955B dt=4667 +05:17:59 pushed ckpt_w6.pt,log_w6.txt +05:19:21 step 37820 loss=3.7208 lr=2.77e-04 tok=4.957B dt=4761 +05:20:36 step 37840 loss=3.7350 lr=2.77e-04 tok=4.960B dt=4836 +05:21:52 step 37860 loss=3.7359 lr=2.77e-04 tok=4.962B dt=4912 +05:23:08 step 37880 loss=3.7771 lr=2.77e-04 tok=4.965B dt=4988 +05:24:24 step 37900 loss=3.6558 lr=2.77e-04 tok=4.968B dt=5064 +05:25:40 step 37920 loss=3.7569 lr=2.77e-04 tok=4.970B dt=5140 +05:26:55 step 37940 loss=3.6916 lr=2.77e-04 tok=4.973B dt=5215 +05:28:11 step 37960 loss=3.5739 lr=2.77e-04 tok=4.975B dt=5291 +05:29:27 step 37980 loss=3.7557 lr=2.77e-04 tok=4.978B dt=5367 +05:30:43 step 38000 loss=3.7994 lr=2.77e-04 tok=4.981B dt=5443 +05:31:59 step 38020 loss=3.7720 lr=2.77e-04 tok=4.983B dt=5519 +05:33:14 step 38040 loss=3.7465 lr=2.77e-04 tok=4.986B dt=5594 +05:34:30 step 38060 loss=3.6641 lr=2.77e-04 tok=4.989B dt=5670 +05:35:46 step 38080 loss=3.7294 lr=2.77e-04 tok=4.991B dt=5746 +05:37:02 step 38100 loss=3.7475 lr=2.77e-04 tok=4.994B dt=5821 +05:37:16 pushed ckpt_w6.pt,log_w6.txt +05:38:43 step 38120 loss=3.6817 lr=2.77e-04 tok=4.996B dt=5923 +05:39:59 step 38140 loss=3.7064 lr=2.77e-04 tok=4.999B dt=5999 +05:41:15 step 38160 loss=3.8412 lr=2.77e-04 tok=5.002B dt=6075 +05:42:31 step 38180 loss=3.7859 lr=2.77e-04 tok=5.004B dt=6151 +05:43:46 step 38200 loss=3.7418 lr=2.77e-04 tok=5.007B dt=6226 +05:45:02 step 38220 loss=3.7950 lr=2.77e-04 tok=5.010B dt=6302 +05:46:18 step 38240 loss=3.7284 lr=2.77e-04 tok=5.012B dt=6378 +05:47:34 step 38260 loss=3.8280 lr=2.77e-04 tok=5.015B dt=6454 +05:48:49 step 38280 loss=3.7128 lr=2.77e-04 tok=5.017B dt=6529 +05:50:05 step 38300 loss=3.7021 lr=2.77e-04 tok=5.020B dt=6605 +05:51:21 step 38320 loss=3.7311 lr=2.77e-04 tok=5.023B dt=6681 +05:52:37 step 38340 loss=3.7881 lr=2.77e-04 tok=5.025B dt=6757 +05:53:53 step 38360 loss=3.7183 lr=2.77e-04 tok=5.028B dt=6833 +05:55:08 step 38380 loss=3.7137 lr=2.77e-04 tok=5.031B dt=6908 +05:56:24 step 38400 loss=3.6593 lr=2.77e-04 tok=5.033B dt=6984 +05:56:40 pushed ckpt_w6.pt,log_w6.txt +05:58:07 step 38420 loss=3.8131 lr=2.77e-04 tok=5.036B dt=7087 +05:59:23 step 38440 loss=3.7810 lr=2.77e-04 tok=5.038B dt=7163 +06:00:38 step 38460 loss=3.7842 lr=2.77e-04 tok=5.041B dt=7238 +06:01:54 step 38480 loss=3.7092 lr=2.77e-04 tok=5.044B dt=7314 +06:03:10 step 38500 loss=3.7613 lr=2.77e-04 tok=5.046B dt=7390 +06:04:26 step 38520 loss=3.7429 lr=2.77e-04 tok=5.049B dt=7466 +06:05:41 step 38540 loss=3.7162 lr=2.77e-04 tok=5.052B dt=7541 +06:06:57 step 38560 loss=3.7750 lr=2.77e-04 tok=5.054B dt=7617 +06:08:13 step 38580 loss=3.8305 lr=2.77e-04 tok=5.057B dt=7693 +06:09:29 step 38600 loss=3.7271 lr=2.77e-04 tok=5.059B dt=7769 +06:10:45 step 38620 loss=3.8086 lr=2.77e-04 tok=5.062B dt=7845 +06:12:01 step 38640 loss=3.8156 lr=2.76e-04 tok=5.065B dt=7920 +06:13:16 step 38660 loss=3.7205 lr=2.76e-04 tok=5.067B dt=7996 +06:14:32 step 38680 loss=3.6164 lr=2.76e-04 tok=5.070B dt=8072 +06:15:48 step 38700 loss=3.7239 lr=2.76e-04 tok=5.072B dt=8148 +06:16:00 pushed ckpt_w6.pt,log_w6.txt +06:17:23 step 38720 loss=3.7443 lr=2.76e-04 tok=5.075B dt=8243 +06:18:39 step 38740 loss=3.7196 lr=2.76e-04 tok=5.078B dt=8319 +06:19:55 step 38760 loss=3.7438 lr=2.76e-04 tok=5.080B dt=8395 +06:21:11 step 38780 loss=3.7265 lr=2.76e-04 tok=5.083B dt=8471 +06:22:26 step 38800 loss=3.7095 lr=2.76e-04 tok=5.086B dt=8546 +06:23:42 step 38820 loss=3.7607 lr=2.76e-04 tok=5.088B dt=8622 +06:24:58 step 38840 loss=3.7053 lr=2.76e-04 tok=5.091B dt=8698 +06:26:14 step 38860 loss=3.8298 lr=2.76e-04 tok=5.093B dt=8774 +06:27:30 step 38880 loss=3.7976 lr=2.76e-04 tok=5.096B dt=8850 +06:28:45 step 38900 loss=3.6936 lr=2.76e-04 tok=5.099B dt=8925 +06:30:01 step 38920 loss=3.7576 lr=2.76e-04 tok=5.101B dt=9001 +06:31:17 step 38940 loss=3.7564 lr=2.76e-04 tok=5.104B dt=9077 +06:32:33 step 38960 loss=3.7735 lr=2.76e-04 tok=5.107B dt=9153 +06:33:49 step 38980 loss=3.8042 lr=2.76e-04 tok=5.109B dt=9229 +06:35:04 step 39000 loss=3.6806 lr=2.76e-04 tok=5.112B dt=9304 diff --git a/log_w60.txt b/log_w60.txt new file mode 100644 index 0000000000000000000000000000000000000000..de35a2aacddb659ed8b1db26eadc32344af05d84 --- /dev/null +++ b/log_w60.txt @@ -0,0 +1,132 @@ +launch +12:41:07 name=w60 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 7/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:41:13 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:41:23 params 1.08B +12:41:24 resumed step=94350 seen=12.37B round=260 +12:41:24 stream salt=94350 +12:43:21 start: accum=4, 131072 tokens per optimiser step +12:44:59 step 94351 loss=2.9232 lr=8.90e-05 tok=12.367B dt=97 wait=65 +12:45:04 step 94352 loss=3.1364 lr=8.90e-05 tok=12.367B dt=103 wait=65 +12:45:44 step 94360 loss=3.2511 lr=8.90e-05 tok=12.368B dt=142 wait=66 +12:47:15 step 94380 loss=3.2476 lr=8.90e-05 tok=12.371B dt=233 wait=66 +12:48:31 step 94400 loss=3.3621 lr=8.90e-05 tok=12.373B dt=309 wait=66 +12:49:46 step 94420 loss=3.2958 lr=8.89e-05 tok=12.376B dt=385 wait=67 +12:51:02 step 94440 loss=3.2910 lr=8.89e-05 tok=12.378B dt=460 wait=67 +12:52:17 step 94460 loss=3.2748 lr=8.89e-05 tok=12.381B dt=536 wait=68 +12:53:33 step 94480 loss=3.2602 lr=8.89e-05 tok=12.384B dt=611 wait=68 +12:54:48 step 94500 loss=3.3352 lr=8.89e-05 tok=12.386B dt=687 wait=68 +12:55:06 pushed ckpt_w60.pt,log_w60.txt +12:56:52 waited 106s for global r=261 +12:57:04 global r=261 load 4.234 -> 4.218 +12:58:24 step 94520 loss=3.2770 lr=8.88e-05 tok=12.389B dt=903 wait=69 +12:59:39 step 94540 loss=3.2255 lr=8.88e-05 tok=12.392B dt=978 wait=69 +13:00:55 step 94560 loss=3.2465 lr=8.88e-05 tok=12.394B dt=1053 wait=70 +13:02:10 step 94580 loss=3.2954 lr=8.88e-05 tok=12.397B dt=1129 wait=70 +13:03:26 step 94600 loss=3.3802 lr=8.87e-05 tok=12.399B dt=1204 wait=70 +13:04:41 step 94620 loss=3.3385 lr=8.87e-05 tok=12.402B dt=1280 wait=71 +13:05:56 step 94640 loss=3.2784 lr=8.87e-05 tok=12.405B dt=1355 wait=71 +13:06:48 pushed ckpt_w60.pt,log_w60.txt +13:08:18 waited 91s for global r=262 +13:08:26 global r=262 load 4.247 -> 4.222 +13:09:10 step 94660 loss=3.3178 lr=8.87e-05 tok=12.407B dt=1548 wait=72 +13:10:25 step 94680 loss=3.2252 lr=8.87e-05 tok=12.410B dt=1623 wait=72 +13:11:40 step 94700 loss=3.2504 lr=8.86e-05 tok=12.413B dt=1699 wait=72 +13:12:55 step 94720 loss=3.2280 lr=8.86e-05 tok=12.415B dt=1774 wait=73 +13:14:11 step 94740 loss=3.3640 lr=8.86e-05 tok=12.418B dt=1849 wait=73 +13:15:27 step 94760 loss=3.3838 lr=8.86e-05 tok=12.420B dt=1925 wait=74 +13:16:42 step 94780 loss=3.3852 lr=8.86e-05 tok=12.423B dt=2001 wait=74 +13:17:58 step 94800 loss=3.3358 lr=8.85e-05 tok=12.426B dt=2076 wait=74 +13:18:14 pushed ckpt_w60.pt,log_w60.txt +13:19:59 waited 106s for global r=263 +13:20:08 global r=263 load 4.229 -> 4.216 +13:21:30 step 94820 loss=3.2580 lr=8.85e-05 tok=12.428B dt=2289 wait=75 +13:22:45 step 94840 loss=3.1853 lr=8.85e-05 tok=12.431B dt=2364 wait=75 +13:24:01 step 94860 loss=3.2195 lr=8.85e-05 tok=12.433B dt=2439 wait=76 +13:25:16 step 94880 loss=3.3229 lr=8.85e-05 tok=12.436B dt=2515 wait=76 +13:26:32 step 94900 loss=3.3915 lr=8.84e-05 tok=12.439B dt=2590 wait=76 +13:27:47 step 94920 loss=3.3164 lr=8.84e-05 tok=12.441B dt=2666 wait=77 +13:29:02 step 94940 loss=3.2406 lr=8.84e-05 tok=12.444B dt=2741 wait=77 +13:29:55 pushed ckpt_w60.pt,log_w60.txt +13:31:41 waited 106s for global r=264 +13:31:47 global r=264 load 4.230 -> 4.220 +13:32:28 step 94960 loss=3.3484 lr=8.84e-05 tok=12.447B dt=2947 wait=78 +13:33:44 step 94980 loss=3.2445 lr=8.83e-05 tok=12.449B dt=3022 wait=78 +13:34:59 step 95000 loss=3.2908 lr=8.83e-05 tok=12.452B dt=3098 wait=78 +13:36:15 step 95020 loss=3.2801 lr=8.83e-05 tok=12.454B dt=3173 wait=79 +13:37:30 step 95040 loss=3.3860 lr=8.83e-05 tok=12.457B dt=3249 wait=79 +13:38:46 step 95060 loss=3.3644 lr=8.83e-05 tok=12.460B dt=3324 wait=80 +13:40:01 step 95080 loss=3.3887 lr=8.82e-05 tok=12.462B dt=3400 wait=80 +13:41:17 step 95100 loss=3.2905 lr=8.82e-05 tok=12.465B dt=3475 wait=80 +13:41:28 pushed ckpt_w60.pt,log_w60.txt +13:42:59 waited 91s for global r=265 +13:43:06 global r=265 load 4.239 -> 4.216 +13:44:26 step 95120 loss=3.2496 lr=8.82e-05 tok=12.468B dt=3665 wait=81 +13:45:41 step 95140 loss=3.2081 lr=8.82e-05 tok=12.470B dt=3740 wait=81 +13:46:57 step 95160 loss=3.2446 lr=8.82e-05 tok=12.473B dt=3815 wait=82 +13:48:12 step 95180 loss=3.2826 lr=8.81e-05 tok=12.475B dt=3891 wait=82 +13:49:28 step 95200 loss=3.3887 lr=8.81e-05 tok=12.478B dt=3966 wait=82 +13:50:43 step 95220 loss=3.3498 lr=8.81e-05 tok=12.481B dt=4042 wait=83 +13:51:59 step 95240 loss=3.2843 lr=8.81e-05 tok=12.483B dt=4117 wait=83 +13:52:48 pushed ckpt_w60.pt,log_w60.txt +13:54:34 waited 106s for global r=266 +13:54:41 global r=266 load 4.239 -> 4.215 +13:55:26 step 95260 loss=3.3140 lr=8.81e-05 tok=12.486B dt=4325 wait=84 +13:56:42 step 95280 loss=3.2034 lr=8.80e-05 tok=12.489B dt=4400 wait=84 +13:57:57 step 95300 loss=3.2737 lr=8.80e-05 tok=12.491B dt=4475 wait=84 +13:59:12 step 95320 loss=3.2436 lr=8.80e-05 tok=12.494B dt=4551 wait=85 +14:00:28 step 95340 loss=3.3404 lr=8.80e-05 tok=12.496B dt=4626 wait=85 +14:01:43 step 95360 loss=3.3071 lr=8.79e-05 tok=12.499B dt=4702 wait=86 +14:02:58 step 95380 loss=3.3341 lr=8.79e-05 tok=12.502B dt=4777 wait=86 +14:04:14 step 95400 loss=3.2860 lr=8.79e-05 tok=12.504B dt=4852 wait=86 +14:04:28 pushed ckpt_w60.pt,log_w60.txt +14:06:14 waited 106s for global r=267 +14:06:20 global r=267 load 4.235 -> 4.219 +14:07:43 step 95420 loss=3.2490 lr=8.79e-05 tok=12.507B dt=5062 wait=87 +14:08:58 step 95440 loss=3.2409 lr=8.79e-05 tok=12.510B dt=5137 wait=87 +14:10:14 step 95460 loss=3.2661 lr=8.78e-05 tok=12.512B dt=5212 wait=88 +14:11:29 step 95480 loss=3.3122 lr=8.78e-05 tok=12.515B dt=5288 wait=88 +14:12:45 step 95500 loss=3.3865 lr=8.78e-05 tok=12.517B dt=5364 wait=88 +14:14:00 step 95520 loss=3.3146 lr=8.78e-05 tok=12.520B dt=5439 wait=89 +14:15:16 step 95540 loss=3.2484 lr=8.78e-05 tok=12.523B dt=5514 wait=89 +14:16:07 pushed ckpt_w60.pt,log_w60.txt +14:17:38 waited 91s for global r=268 +14:17:47 global r=268 load 4.230 -> 4.216 +14:18:30 step 95560 loss=3.3555 lr=8.77e-05 tok=12.525B dt=5709 wait=90 +14:19:46 step 95580 loss=3.2347 lr=8.77e-05 tok=12.528B dt=5784 wait=90 +14:21:01 step 95600 loss=3.3351 lr=8.77e-05 tok=12.530B dt=5860 wait=90 +14:22:17 step 95620 loss=3.2589 lr=8.77e-05 tok=12.533B dt=5935 wait=91 +14:23:32 step 95640 loss=3.3817 lr=8.76e-05 tok=12.536B dt=6011 wait=91 +14:24:47 step 95660 loss=3.3272 lr=8.76e-05 tok=12.538B dt=6086 wait=92 +14:26:03 step 95680 loss=3.3630 lr=8.76e-05 tok=12.541B dt=6161 wait=92 +14:27:18 step 95700 loss=3.2986 lr=8.76e-05 tok=12.544B dt=6237 wait=92 +14:27:31 pushed ckpt_w60.pt,log_w60.txt +14:29:01 waited 91s for global r=269 +14:29:09 global r=269 load 4.229 -> 4.216 +14:30:29 step 95720 loss=3.2457 lr=8.76e-05 tok=12.546B dt=6428 wait=93 +14:31:45 step 95740 loss=3.2229 lr=8.75e-05 tok=12.549B dt=6503 wait=93 +14:33:00 step 95760 loss=3.2421 lr=8.75e-05 tok=12.551B dt=6578 wait=94 +14:34:15 step 95780 loss=3.2844 lr=8.75e-05 tok=12.554B dt=6654 wait=94 +14:35:31 step 95800 loss=3.4051 lr=8.75e-05 tok=12.557B dt=6730 wait=94 +14:36:47 step 95820 loss=3.3506 lr=8.75e-05 tok=12.559B dt=6805 wait=95 +14:38:02 step 95840 loss=3.2587 lr=8.74e-05 tok=12.562B dt=6881 wait=95 +14:38:52 pushed ckpt_w60.pt,log_w60.txt +14:40:53 waited 121s for global r=270 +14:41:02 global r=270 load 4.225 -> 4.212 +14:41:45 step 95860 loss=3.3390 lr=8.74e-05 tok=12.565B dt=7104 wait=96 +14:43:00 step 95880 loss=3.2204 lr=8.74e-05 tok=12.567B dt=7179 wait=96 +14:44:16 step 95900 loss=3.2729 lr=8.74e-05 tok=12.570B dt=7254 wait=96 +14:45:31 step 95920 loss=3.2331 lr=8.74e-05 tok=12.572B dt=7330 wait=97 +14:46:47 step 95940 loss=3.3708 lr=8.73e-05 tok=12.575B dt=7405 wait=97 +14:48:02 step 95960 loss=3.3343 lr=8.73e-05 tok=12.578B dt=7481 wait=98 +14:49:18 step 95980 loss=3.3505 lr=8.73e-05 tok=12.580B dt=7556 wait=98 +14:50:33 step 96000 loss=3.2809 lr=8.73e-05 tok=12.583B dt=7632 wait=98 +14:50:45 pushed ckpt_w60.pt,log_w60.txt +14:52:46 waited 121s for global r=271 +14:52:54 global r=271 load 4.237 -> 4.216 +14:54:15 step 96020 loss=3.2537 lr=8.72e-05 tok=12.586B dt=7854 wait=99 +14:55:30 step 96040 loss=3.2048 lr=8.72e-05 tok=12.588B dt=7929 wait=99 +14:56:46 step 96060 loss=3.2714 lr=8.72e-05 tok=12.591B dt=8004 wait=100 +14:58:01 step 96080 loss=3.3213 lr=8.72e-05 tok=12.593B dt=8080 wait=100 +14:59:17 step 96100 loss=3.4103 lr=8.72e-05 tok=12.596B dt=8156 wait=100 +15:00:33 step 96120 loss=3.3491 lr=8.71e-05 tok=12.599B dt=8231 wait=101 +15:01:48 step 96140 loss=3.2986 lr=8.71e-05 tok=12.601B dt=8307 wait=101 diff --git a/log_w63.txt b/log_w63.txt new file mode 100644 index 0000000000000000000000000000000000000000..7774681b5dcdacc58de1d9bdf1aaa66fd8f8566a --- /dev/null +++ b/log_w63.txt @@ -0,0 +1,200 @@ +launch +18:47:40 name=w63 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 10/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:47:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:47:46 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:47:56 params 1.08B +18:47:57 resumed step=97950 seen=12.84B round=281 +18:47:57 stream salt=97950 +18:50:19 blocks compiled +18:50:19 start: accum=4, 131072 tokens per optimiser step +launch +18:52:41 name=w63 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 10/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:52:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:52:46 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:53:00 params 1.08B +18:53:01 resumed step=97950 seen=12.84B round=281 +18:53:01 stream salt=97950 +18:55:01 start: accum=4, 131072 tokens per optimiser step +18:58:52 step 97951 loss=3.2675 lr=8.52e-05 tok=12.839B dt=231 wait=196 +18:58:57 step 97952 loss=3.3397 lr=8.52e-05 tok=12.839B dt=236 wait=196 +18:59:35 step 97960 loss=3.5817 lr=8.52e-05 tok=12.840B dt=274 wait=196 +19:00:58 step 97980 loss=3.5515 lr=8.52e-05 tok=12.842B dt=357 wait=196 +19:02:15 step 98000 loss=3.4931 lr=8.52e-05 tok=12.845B dt=434 wait=197 +19:03:32 step 98020 loss=3.5126 lr=8.51e-05 tok=12.848B dt=511 wait=197 +19:04:49 step 98040 loss=3.5441 lr=8.51e-05 tok=12.850B dt=588 wait=197 +19:06:07 step 98060 loss=3.6999 lr=8.51e-05 tok=12.853B dt=666 wait=197 +launch +19:06:32 name=w63 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 10/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:06:36 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:06:36 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +19:06:46 params 1.08B +19:06:47 resumed step=97950 seen=12.84B round=281 +19:06:47 stream salt=97950 +19:08:50 blocks compiled +19:08:50 start: accum=4, 131072 tokens per optimiser step +19:12:10 step 97951 loss=3.2676 lr=8.52e-05 tok=12.839B dt=199 wait=174 +19:12:13 step 97952 loss=3.3397 lr=8.52e-05 tok=12.839B dt=203 wait=174 +19:12:41 step 97960 loss=3.5816 lr=8.52e-05 tok=12.840B dt=230 wait=174 +19:13:50 step 97980 loss=3.5515 lr=8.52e-05 tok=12.842B dt=300 wait=175 +19:14:55 step 98000 loss=3.4932 lr=8.52e-05 tok=12.845B dt=365 wait=175 +19:15:59 step 98020 loss=3.5126 lr=8.51e-05 tok=12.848B dt=428 wait=175 +19:17:03 step 98040 loss=3.5441 lr=8.51e-05 tok=12.850B dt=492 wait=176 +19:18:07 step 98060 loss=3.6998 lr=8.51e-05 tok=12.853B dt=556 wait=176 +19:19:10 step 98080 loss=3.5077 lr=8.51e-05 tok=12.856B dt=620 wait=176 +19:20:14 step 98100 loss=3.6049 lr=8.50e-05 tok=12.858B dt=683 wait=176 +19:20:28 pushed ckpt_w63.pt,log_w63.txt +19:20:28 waited 0s for global r=282 +19:20:53 global r=282 load 4.265 -> 4.262 +19:22:04 step 98120 loss=3.4824 lr=8.50e-05 tok=12.861B dt=794 wait=177 +19:23:08 step 98140 loss=3.5576 lr=8.50e-05 tok=12.863B dt=857 wait=177 +19:24:11 step 98160 loss=3.5300 lr=8.50e-05 tok=12.866B dt=921 wait=177 +19:25:15 step 98180 loss=3.4464 lr=8.50e-05 tok=12.869B dt=985 wait=178 +19:26:19 step 98200 loss=3.5137 lr=8.49e-05 tok=12.871B dt=1048 wait=178 +19:27:22 step 98220 loss=3.5992 lr=8.49e-05 tok=12.874B dt=1112 wait=179 +19:28:26 step 98240 loss=3.4702 lr=8.49e-05 tok=12.877B dt=1176 wait=179 +19:29:10 pushed ckpt_w63.pt,log_w63.txt +19:29:11 waited 0s for global r=283 +19:29:20 global r=283 load 4.270 -> 4.271 +19:29:58 step 98260 loss=3.5902 lr=8.49e-05 tok=12.879B dt=1267 wait=179 +19:31:01 step 98280 loss=3.4637 lr=8.49e-05 tok=12.882B dt=1331 wait=179 +19:32:06 step 98300 loss=3.4837 lr=8.48e-05 tok=12.884B dt=1395 wait=180 +19:33:09 step 98320 loss=3.5060 lr=8.48e-05 tok=12.887B dt=1459 wait=180 +19:34:13 step 98340 loss=3.4406 lr=8.48e-05 tok=12.890B dt=1522 wait=181 +19:35:16 step 98360 loss=3.5806 lr=8.48e-05 tok=12.892B dt=1586 wait=181 +19:36:20 step 98380 loss=3.5111 lr=8.47e-05 tok=12.895B dt=1649 wait=181 +19:37:23 step 98400 loss=3.4664 lr=8.47e-05 tok=12.897B dt=1713 wait=181 +19:37:36 pushed ckpt_w63.pt,log_w63.txt +19:39:06 waited 91s for global r=284 +19:39:16 global r=284 load 4.279 -> 4.280 +19:40:23 step 98420 loss=3.4672 lr=8.47e-05 tok=12.900B dt=1893 wait=182 +19:41:27 step 98440 loss=3.5112 lr=8.47e-05 tok=12.903B dt=1956 wait=182 +19:42:30 step 98460 loss=3.4792 lr=8.47e-05 tok=12.905B dt=2020 wait=183 +19:43:34 step 98480 loss=3.4640 lr=8.46e-05 tok=12.908B dt=2084 wait=183 +19:44:39 step 98500 loss=3.4783 lr=8.46e-05 tok=12.911B dt=2148 wait=183 +19:45:42 step 98520 loss=3.5707 lr=8.46e-05 tok=12.913B dt=2212 wait=184 +19:46:46 step 98540 loss=3.4071 lr=8.46e-05 tok=12.916B dt=2276 wait=184 +19:47:32 pushed ckpt_w63.pt,log_w63.txt +19:51:11 step 98560 loss=3.6037 lr=8.46e-05 tok=12.918B dt=2541 wait=185 +19:52:15 step 98580 loss=3.5091 lr=8.45e-05 tok=12.921B dt=2605 wait=185 +19:53:19 step 98600 loss=3.4467 lr=8.45e-05 tok=12.924B dt=2668 wait=185 +19:54:22 step 98620 loss=3.4095 lr=8.45e-05 tok=12.926B dt=2732 wait=186 +19:55:27 step 98640 loss=3.4811 lr=8.45e-05 tok=12.929B dt=2796 wait=186 +19:56:31 step 98660 loss=3.5007 lr=8.45e-05 tok=12.932B dt=2860 wait=187 +19:57:34 step 98680 loss=3.4414 lr=8.44e-05 tok=12.934B dt=2924 wait=187 +19:58:38 step 98700 loss=3.6778 lr=8.44e-05 tok=12.937B dt=2988 wait=187 +19:58:52 pushed ckpt_w63.pt,log_w63.txt +19:58:52 waited 0s for global r=285 +19:59:00 global r=285 load 4.293 -> 4.280 +20:00:08 step 98720 loss=3.4372 lr=8.44e-05 tok=12.939B dt=3078 wait=188 +20:01:12 step 98740 loss=3.5417 lr=8.44e-05 tok=12.942B dt=3142 wait=188 +20:02:16 step 98760 loss=3.5768 lr=8.43e-05 tok=12.945B dt=3206 wait=189 +20:03:21 step 98780 loss=3.4525 lr=8.43e-05 tok=12.947B dt=3270 wait=189 +20:04:25 step 98800 loss=3.4327 lr=8.43e-05 tok=12.950B dt=3335 wait=189 +20:05:29 step 98820 loss=3.6149 lr=8.43e-05 tok=12.953B dt=3399 wait=190 +20:06:33 step 98840 loss=3.4411 lr=8.43e-05 tok=12.955B dt=3463 wait=190 +20:07:18 pushed ckpt_w63.pt,log_w63.txt +20:08:34 waited 76s for global r=286 +20:08:45 global r=286 load 4.290 -> 4.293 +20:09:23 step 98860 loss=3.6875 lr=8.42e-05 tok=12.958B dt=3632 wait=190 +20:10:27 step 98880 loss=3.4591 lr=8.42e-05 tok=12.960B dt=3696 wait=191 +20:11:31 step 98900 loss=3.4514 lr=8.42e-05 tok=12.963B dt=3761 wait=191 +20:12:35 step 98920 loss=3.6198 lr=8.42e-05 tok=12.966B dt=3825 wait=192 +20:13:39 step 98940 loss=3.4653 lr=8.42e-05 tok=12.968B dt=3889 wait=192 +20:14:43 step 98960 loss=3.5974 lr=8.41e-05 tok=12.971B dt=3953 wait=192 +20:15:47 step 98980 loss=3.4726 lr=8.41e-05 tok=12.974B dt=4016 wait=193 +20:16:51 step 99000 loss=3.5259 lr=8.41e-05 tok=12.976B dt=4080 wait=193 +20:17:03 pushed ckpt_w63.pt,log_w63.txt +20:21:15 step 99020 loss=3.5090 lr=8.41e-05 tok=12.979B dt=4345 wait=193 +20:22:19 step 99040 loss=3.4349 lr=8.40e-05 tok=12.981B dt=4408 wait=194 +20:23:22 step 99060 loss=3.4034 lr=8.40e-05 tok=12.984B dt=4472 wait=194 +20:24:26 step 99080 loss=3.4134 lr=8.40e-05 tok=12.987B dt=4535 wait=195 +20:25:29 step 99100 loss=3.3476 lr=8.40e-05 tok=12.989B dt=4598 wait=195 +20:26:32 step 99120 loss=3.5266 lr=8.40e-05 tok=12.992B dt=4661 wait=195 +20:27:35 step 99140 loss=3.4823 lr=8.39e-05 tok=12.994B dt=4725 wait=196 +20:28:21 pushed ckpt_w63.pt,log_w63.txt +20:32:00 step 99160 loss=3.5583 lr=8.39e-05 tok=12.997B dt=4989 wait=196 +20:33:03 step 99180 loss=3.5640 lr=8.39e-05 tok=13.000B dt=5053 wait=197 +20:34:06 step 99200 loss=3.4669 lr=8.39e-05 tok=13.002B dt=5116 wait=197 +20:35:10 step 99220 loss=3.5014 lr=8.39e-05 tok=13.005B dt=5179 wait=198 +20:36:13 step 99240 loss=3.4617 lr=8.38e-05 tok=13.008B dt=5242 wait=198 +20:37:17 step 99260 loss=3.4701 lr=8.38e-05 tok=13.010B dt=5306 wait=198 +20:38:20 step 99280 loss=3.4228 lr=8.38e-05 tok=13.013B dt=5369 wait=199 +20:39:22 step 99300 loss=3.4824 lr=8.38e-05 tok=13.015B dt=5432 wait=199 +20:39:35 pushed ckpt_w63.pt,log_w63.txt +20:39:35 waited 0s for global r=287 +20:39:45 global r=287 load 4.300 -> 4.274 +20:40:53 step 99320 loss=3.4375 lr=8.38e-05 tok=13.018B dt=5523 wait=199 +20:41:57 step 99340 loss=3.4025 lr=8.37e-05 tok=13.021B dt=5586 wait=200 +20:42:59 step 99360 loss=3.4635 lr=8.37e-05 tok=13.023B dt=5649 wait=200 +20:44:03 step 99380 loss=3.4230 lr=8.37e-05 tok=13.026B dt=5712 wait=201 +20:45:06 step 99400 loss=3.4127 lr=8.37e-05 tok=13.029B dt=5776 wait=201 +20:46:09 step 99420 loss=3.5048 lr=8.36e-05 tok=13.031B dt=5839 wait=201 +20:47:13 step 99440 loss=3.3753 lr=8.36e-05 tok=13.034B dt=5902 wait=202 +20:47:59 pushed ckpt_w63.pt,log_w63.txt +20:49:15 waited 76s for global r=288 +20:49:24 global r=288 load 4.303 -> 4.281 +20:49:59 step 99460 loss=3.5195 lr=8.36e-05 tok=13.036B dt=6069 wait=202 +20:51:03 step 99480 loss=3.4495 lr=8.36e-05 tok=13.039B dt=6132 wait=203 +20:52:06 step 99500 loss=3.3952 lr=8.36e-05 tok=13.042B dt=6196 wait=203 +20:53:10 step 99520 loss=3.4620 lr=8.35e-05 tok=13.044B dt=6259 wait=203 +20:54:13 step 99540 loss=3.3680 lr=8.35e-05 tok=13.047B dt=6323 wait=204 +20:55:17 step 99560 loss=3.5118 lr=8.35e-05 tok=13.050B dt=6387 wait=204 +20:56:21 step 99580 loss=3.4172 lr=8.35e-05 tok=13.052B dt=6450 wait=205 +20:57:24 step 99600 loss=3.4112 lr=8.35e-05 tok=13.055B dt=6513 wait=205 +20:57:39 pushed ckpt_w63.pt,log_w63.txt +21:01:50 step 99620 loss=3.4941 lr=8.34e-05 tok=13.057B dt=6780 wait=205 +21:02:54 step 99640 loss=3.4432 lr=8.34e-05 tok=13.060B dt=6843 wait=206 +21:03:57 step 99660 loss=3.3078 lr=8.34e-05 tok=13.063B dt=6906 wait=206 +21:05:00 step 99680 loss=3.3891 lr=8.34e-05 tok=13.065B dt=6970 wait=207 +21:06:04 step 99700 loss=3.3455 lr=8.33e-05 tok=13.068B dt=7033 wait=207 +21:07:07 step 99720 loss=3.5274 lr=8.33e-05 tok=13.070B dt=7097 wait=207 +21:08:11 step 99740 loss=3.4078 lr=8.33e-05 tok=13.073B dt=7160 wait=208 +21:09:00 pushed ckpt_w63.pt,log_w63.txt +21:09:00 waited 0s for global r=289 +21:09:11 global r=289 load 4.333 -> 4.284 +21:09:50 step 99760 loss=3.5807 lr=8.33e-05 tok=13.076B dt=7259 wait=208 +21:10:53 step 99780 loss=3.4131 lr=8.33e-05 tok=13.078B dt=7322 wait=209 +21:11:56 step 99800 loss=3.4495 lr=8.32e-05 tok=13.081B dt=7386 wait=209 +21:13:00 step 99820 loss=3.4871 lr=8.32e-05 tok=13.084B dt=7450 wait=209 +21:14:04 step 99840 loss=3.4053 lr=8.32e-05 tok=13.086B dt=7513 wait=210 +21:15:07 step 99860 loss=3.5241 lr=8.32e-05 tok=13.089B dt=7577 wait=210 +21:16:10 step 99880 loss=3.4292 lr=8.32e-05 tok=13.091B dt=7640 wait=211 +21:17:14 step 99900 loss=3.4218 lr=8.31e-05 tok=13.094B dt=7703 wait=211 +21:17:29 pushed ckpt_w63.pt,log_w63.txt +21:17:29 waited 0s for global r=290 +21:17:37 global r=290 load 4.311 -> 4.283 +21:18:45 step 99920 loss=3.4242 lr=8.31e-05 tok=13.097B dt=7794 wait=211 +21:19:48 step 99940 loss=3.4382 lr=8.31e-05 tok=13.099B dt=7857 wait=212 +21:20:51 step 99960 loss=3.5032 lr=8.31e-05 tok=13.102B dt=7921 wait=212 +21:21:55 step 99980 loss=3.3648 lr=8.31e-05 tok=13.105B dt=7984 wait=213 +21:22:58 step 100000 loss=3.4067 lr=8.30e-05 tok=13.107B dt=8048 wait=213 +launch +21:23:47 name=w63 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 10/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:23:52 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:23:52 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +21:24:03 params 1.08B +21:24:07 resumed step=99900 seen=13.09B round=290 +21:24:07 stream salt=99900 +21:26:06 blocks compiled +21:26:06 start: accum=4, 131072 tokens per optimiser step +21:28:22 step 99901 loss=3.2730 lr=8.31e-05 tok=13.094B dt=136 wait=112 +21:28:25 step 99902 loss=3.2109 lr=8.31e-05 tok=13.094B dt=140 wait=112 +21:29:05 global r=291 load 4.267 -> 4.285 +21:30:15 step 100220 loss=3.4030 lr=8.28e-05 tok=13.098B dt=249 wait=113 +21:31:20 step 100240 loss=3.4452 lr=8.28e-05 tok=13.101B dt=314 wait=114 +21:32:24 step 100260 loss=3.4751 lr=8.28e-05 tok=13.103B dt=378 wait=114 +21:33:28 step 100280 loss=3.4270 lr=8.27e-05 tok=13.106B dt=442 wait=114 +21:34:31 step 100300 loss=3.4028 lr=8.27e-05 tok=13.109B dt=505 wait=115 +21:35:35 step 100320 loss=3.5025 lr=8.27e-05 tok=13.111B dt=569 wait=115 +21:36:38 step 100340 loss=3.3474 lr=8.27e-05 tok=13.114B dt=633 wait=116 +21:37:25 pushed ckpt_w63.pt,log_w63.txt +21:38:55 waited 91s for global r=292 +21:39:07 global r=292 load 4.288 -> 4.260 +21:39:42 step 100360 loss=3.5172 lr=8.26e-05 tok=13.116B dt=816 wait=116 +21:40:45 step 100380 loss=3.3888 lr=8.26e-05 tok=13.119B dt=879 wait=116 +21:41:48 step 100400 loss=3.3866 lr=8.26e-05 tok=13.122B dt=943 wait=117 +21:42:52 step 100420 loss=3.4400 lr=8.26e-05 tok=13.124B dt=1006 wait=117 +21:43:55 step 100440 loss=3.3785 lr=8.26e-05 tok=13.127B dt=1069 wait=118 +21:44:59 step 100460 loss=3.4244 lr=8.25e-05 tok=13.129B dt=1133 wait=118 +21:46:02 step 100480 loss=3.4150 lr=8.25e-05 tok=13.132B dt=1196 wait=119 +21:47:06 step 100500 loss=3.4001 lr=8.25e-05 tok=13.135B dt=1260 wait=119 diff --git a/log_w64.txt b/log_w64.txt new file mode 100644 index 0000000000000000000000000000000000000000..d1be7ca31cebb86a3f866d93d571779bc14d0cbd --- /dev/null +++ b/log_w64.txt @@ -0,0 +1,238 @@ +launch +18:50:15 name=w64 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 11/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:50:21 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:50:21 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:50:32 params 1.08B +18:50:34 resumed step=97950 seen=12.84B round=281 +18:50:34 stream salt=97950 +launch +18:52:41 name=w64 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 11/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:52:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:52:46 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:52:56 params 1.08B +18:52:57 resumed step=97950 seen=12.84B round=281 +18:52:57 stream salt=97950 +18:54:57 blocks compiled +18:54:57 start: accum=4, 131072 tokens per optimiser step +18:56:41 step 97951 loss=3.1623 lr=8.52e-05 tok=12.839B dt=104 wait=67 +18:56:44 step 97952 loss=3.1886 lr=8.52e-05 tok=12.839B dt=108 wait=67 +18:57:13 step 97960 loss=3.4775 lr=8.52e-05 tok=12.840B dt=136 wait=68 +18:58:22 step 97980 loss=3.3894 lr=8.52e-05 tok=12.842B dt=205 wait=68 +18:59:24 step 98000 loss=3.4161 lr=8.52e-05 tok=12.845B dt=267 wait=69 +19:00:27 step 98020 loss=3.3555 lr=8.51e-05 tok=12.848B dt=330 wait=69 +19:01:29 step 98040 loss=3.3440 lr=8.51e-05 tok=12.850B dt=392 wait=70 +19:02:31 step 98060 loss=3.4725 lr=8.51e-05 tok=12.853B dt=454 wait=70 +19:03:33 step 98080 loss=3.2932 lr=8.51e-05 tok=12.856B dt=517 wait=70 +19:04:36 step 98100 loss=3.3752 lr=8.50e-05 tok=12.858B dt=579 wait=71 +19:04:50 pushed ckpt_w64.pt,log_w64.txt +19:04:50 waited 0s for global r=282 +19:05:18 global r=282 load 4.255 -> 4.262 +launch +19:06:34 name=w64 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 11/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:06:39 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:06:39 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +19:06:48 params 1.08B +19:06:52 resumed step=98100 seen=12.86B round=282 +19:06:52 stream salt=98100 +19:08:54 blocks compiled +19:08:54 start: accum=4, 131072 tokens per optimiser step +19:12:04 step 98101 loss=3.1189 lr=8.50e-05 tok=12.858B dt=191 wait=176 +19:12:08 step 98102 loss=3.1562 lr=8.50e-05 tok=12.858B dt=194 wait=176 +19:13:10 step 98120 loss=3.4949 lr=8.50e-05 tok=12.861B dt=256 wait=176 +19:14:18 step 98140 loss=3.4714 lr=8.50e-05 tok=12.863B dt=325 wait=177 +19:15:22 step 98160 loss=3.5208 lr=8.50e-05 tok=12.866B dt=388 wait=177 +19:16:25 step 98180 loss=3.4730 lr=8.50e-05 tok=12.869B dt=451 wait=178 +19:17:29 step 98200 loss=3.5936 lr=8.49e-05 tok=12.871B dt=515 wait=178 +19:18:32 step 98220 loss=3.5251 lr=8.49e-05 tok=12.874B dt=578 wait=178 +19:19:35 step 98240 loss=3.4743 lr=8.49e-05 tok=12.877B dt=642 wait=179 +19:20:23 pushed ckpt_w64.pt,log_w64.txt +19:24:01 step 98260 loss=3.4984 lr=8.49e-05 tok=12.879B dt=908 wait=179 +19:25:05 step 98280 loss=3.5717 lr=8.49e-05 tok=12.882B dt=971 wait=179 +19:26:08 step 98300 loss=3.4565 lr=8.48e-05 tok=12.884B dt=1034 wait=180 +19:27:11 step 98320 loss=3.6014 lr=8.48e-05 tok=12.887B dt=1097 wait=180 +19:28:29 step 98340 loss=3.4647 lr=8.48e-05 tok=12.890B dt=1175 wait=180 +19:30:00 step 98360 loss=3.4115 lr=8.48e-05 tok=12.892B dt=1266 wait=181 +19:31:30 step 98380 loss=3.5259 lr=8.47e-05 tok=12.895B dt=1356 wait=181 +19:33:00 step 98400 loss=3.3588 lr=8.47e-05 tok=12.897B dt=1446 wait=181 +19:33:16 pushed ckpt_w64.pt,log_w64.txt +19:33:16 waited 0s for global r=283 +19:33:32 global r=283 load 4.304 -> 4.271 +19:35:02 step 98420 loss=3.4100 lr=8.47e-05 tok=12.900B dt=1568 wait=181 +19:36:32 step 98440 loss=3.4673 lr=8.47e-05 tok=12.903B dt=1659 wait=182 +19:38:03 step 98460 loss=3.4838 lr=8.47e-05 tok=12.905B dt=1749 wait=182 +19:39:33 step 98480 loss=3.4253 lr=8.46e-05 tok=12.908B dt=1839 wait=182 +19:41:04 step 98500 loss=3.4446 lr=8.46e-05 tok=12.911B dt=1930 wait=183 +19:42:30 step 98520 loss=3.4726 lr=8.46e-05 tok=12.913B dt=2017 wait=183 +19:44:01 step 98540 loss=3.3960 lr=8.46e-05 tok=12.916B dt=2107 wait=183 +19:45:00 pushed ckpt_w64.pt,log_w64.txt +19:45:00 waited 0s for global r=284 +19:45:09 global r=284 load 4.337 -> 4.280 +19:46:03 step 98560 loss=3.5312 lr=8.46e-05 tok=12.918B dt=2230 wait=184 +19:47:33 step 98580 loss=3.4153 lr=8.45e-05 tok=12.921B dt=2319 wait=184 +19:48:59 step 98600 loss=3.4408 lr=8.45e-05 tok=12.924B dt=2406 wait=184 +19:50:29 step 98620 loss=3.5292 lr=8.45e-05 tok=12.926B dt=2496 wait=185 +19:51:59 step 98640 loss=3.3800 lr=8.45e-05 tok=12.929B dt=2585 wait=185 +19:53:29 step 98660 loss=3.5321 lr=8.45e-05 tok=12.932B dt=2676 wait=185 +19:55:00 step 98680 loss=3.3823 lr=8.44e-05 tok=12.934B dt=2766 wait=186 +19:56:22 step 98700 loss=3.5037 lr=8.44e-05 tok=12.937B dt=2849 wait=186 +19:56:36 pushed ckpt_w64.pt,log_w64.txt +19:56:36 waited 0s for global r=285 +19:56:44 global r=285 load 4.288 -> 4.280 +19:57:54 step 98720 loss=3.3986 lr=8.44e-05 tok=12.939B dt=2940 wait=186 +19:58:57 step 98740 loss=3.4970 lr=8.44e-05 tok=12.942B dt=3003 wait=187 +20:00:01 step 98760 loss=3.4866 lr=8.43e-05 tok=12.945B dt=3067 wait=187 +20:01:04 step 98780 loss=3.4535 lr=8.43e-05 tok=12.947B dt=3130 wait=187 +20:02:07 step 98800 loss=3.4073 lr=8.43e-05 tok=12.950B dt=3193 wait=187 +20:03:10 step 98820 loss=3.5194 lr=8.43e-05 tok=12.953B dt=3256 wait=188 +20:04:13 step 98840 loss=3.4309 lr=8.43e-05 tok=12.955B dt=3319 wait=188 +20:04:56 pushed ckpt_w64.pt,log_w64.txt +20:08:35 step 98860 loss=3.5383 lr=8.42e-05 tok=12.958B dt=3581 wait=188 +20:09:38 step 98880 loss=3.4238 lr=8.42e-05 tok=12.960B dt=3645 wait=189 +20:10:41 step 98900 loss=3.4329 lr=8.42e-05 tok=12.963B dt=3708 wait=189 +20:11:44 step 98920 loss=3.3393 lr=8.42e-05 tok=12.966B dt=3771 wait=189 +20:12:47 step 98940 loss=3.3842 lr=8.42e-05 tok=12.968B dt=3834 wait=190 +20:13:51 step 98960 loss=3.4244 lr=8.41e-05 tok=12.971B dt=3897 wait=190 +20:14:54 step 98980 loss=3.4413 lr=8.41e-05 tok=12.974B dt=3960 wait=190 +20:15:57 step 99000 loss=3.5654 lr=8.41e-05 tok=12.976B dt=4023 wait=191 +20:16:09 pushed ckpt_w64.pt,log_w64.txt +20:16:09 waited 0s for global r=286 +20:16:24 global r=286 load 4.279 -> 4.293 +20:17:41 step 99020 loss=3.4013 lr=8.41e-05 tok=12.979B dt=4127 wait=191 +20:18:45 step 99040 loss=3.3899 lr=8.40e-05 tok=12.981B dt=4192 wait=191 +20:19:49 step 99060 loss=3.4422 lr=8.40e-05 tok=12.984B dt=4255 wait=192 +20:20:52 step 99080 loss=3.3964 lr=8.40e-05 tok=12.987B dt=4318 wait=192 +20:21:55 step 99100 loss=3.4637 lr=8.40e-05 tok=12.989B dt=4381 wait=192 +20:22:58 step 99120 loss=3.4999 lr=8.40e-05 tok=12.992B dt=4444 wait=193 +20:24:01 step 99140 loss=3.3947 lr=8.39e-05 tok=12.994B dt=4507 wait=193 +20:24:44 pushed ckpt_w64.pt,log_w64.txt +20:28:23 step 99160 loss=3.5076 lr=8.39e-05 tok=12.997B dt=4770 wait=193 +20:29:27 step 99180 loss=3.4337 lr=8.39e-05 tok=13.000B dt=4833 wait=193 +20:30:30 step 99200 loss=3.3715 lr=8.39e-05 tok=13.002B dt=4896 wait=194 +20:31:33 step 99220 loss=3.2917 lr=8.39e-05 tok=13.005B dt=4959 wait=194 +20:32:36 step 99240 loss=3.3193 lr=8.38e-05 tok=13.008B dt=5023 wait=194 +20:33:40 step 99260 loss=3.4216 lr=8.38e-05 tok=13.010B dt=5086 wait=195 +20:34:42 step 99280 loss=3.4080 lr=8.38e-05 tok=13.013B dt=5149 wait=195 +20:35:46 step 99300 loss=3.5963 lr=8.38e-05 tok=13.015B dt=5212 wait=195 +20:35:58 pushed ckpt_w64.pt,log_w64.txt +20:37:29 waited 91s for global r=287 +20:37:36 global r=287 load 4.281 -> 4.274 +20:38:46 step 99320 loss=3.4192 lr=8.38e-05 tok=13.018B dt=5393 wait=196 +20:39:49 step 99340 loss=3.4408 lr=8.37e-05 tok=13.021B dt=5455 wait=196 +20:40:53 step 99360 loss=3.3809 lr=8.37e-05 tok=13.023B dt=5519 wait=196 +20:41:56 step 99380 loss=3.3967 lr=8.37e-05 tok=13.026B dt=5582 wait=197 +20:42:59 step 99400 loss=3.4130 lr=8.37e-05 tok=13.029B dt=5646 wait=197 +20:44:03 step 99420 loss=3.4327 lr=8.36e-05 tok=13.031B dt=5709 wait=197 +20:45:06 step 99440 loss=3.3876 lr=8.36e-05 tok=13.034B dt=5773 wait=198 +20:45:52 pushed ckpt_w64.pt,log_w64.txt +20:49:33 step 99460 loss=3.5525 lr=8.36e-05 tok=13.036B dt=6039 wait=198 +20:50:36 step 99480 loss=3.3876 lr=8.36e-05 tok=13.039B dt=6102 wait=198 +20:51:40 step 99500 loss=3.3865 lr=8.36e-05 tok=13.042B dt=6166 wait=199 +20:52:43 step 99520 loss=3.3372 lr=8.35e-05 tok=13.044B dt=6230 wait=199 +20:53:46 step 99540 loss=3.3899 lr=8.35e-05 tok=13.047B dt=6293 wait=199 +20:54:50 step 99560 loss=3.4501 lr=8.35e-05 tok=13.050B dt=6356 wait=200 +20:55:53 step 99580 loss=3.4290 lr=8.35e-05 tok=13.052B dt=6419 wait=200 +20:56:56 step 99600 loss=3.6053 lr=8.35e-05 tok=13.055B dt=6483 wait=200 +20:57:11 pushed ckpt_w64.pt,log_w64.txt +20:57:11 waited 0s for global r=288 +20:57:20 global r=288 load 4.309 -> 4.281 +20:58:28 step 99620 loss=3.3646 lr=8.34e-05 tok=13.057B dt=6574 wait=201 +20:59:32 step 99640 loss=3.4393 lr=8.34e-05 tok=13.060B dt=6638 wait=201 +21:00:35 step 99660 loss=3.4590 lr=8.34e-05 tok=13.063B dt=6701 wait=201 +21:01:38 step 99680 loss=3.3241 lr=8.34e-05 tok=13.065B dt=6764 wait=201 +21:02:41 step 99700 loss=3.4380 lr=8.33e-05 tok=13.068B dt=6827 wait=202 +21:03:45 step 99720 loss=3.5029 lr=8.33e-05 tok=13.070B dt=6891 wait=202 +21:04:48 step 99740 loss=3.3299 lr=8.33e-05 tok=13.073B dt=6954 wait=202 +21:05:33 pushed ckpt_w64.pt,log_w64.txt +21:05:34 waited 0s for global r=289 +21:05:46 global r=289 load 4.273 -> 4.284 +21:06:21 step 99760 loss=3.5290 lr=8.33e-05 tok=13.076B dt=7047 wait=203 +21:07:24 step 99780 loss=3.3727 lr=8.33e-05 tok=13.078B dt=7110 wait=203 +21:08:27 step 99800 loss=3.3829 lr=8.32e-05 tok=13.081B dt=7174 wait=203 +21:09:31 step 99820 loss=3.4974 lr=8.32e-05 tok=13.084B dt=7237 wait=204 +21:10:35 step 99840 loss=3.3574 lr=8.32e-05 tok=13.086B dt=7301 wait=204 +21:11:38 step 99860 loss=3.4933 lr=8.32e-05 tok=13.089B dt=7364 wait=204 +21:12:42 step 99880 loss=3.4632 lr=8.32e-05 tok=13.091B dt=7428 wait=205 +21:13:45 step 99900 loss=3.5270 lr=8.31e-05 tok=13.094B dt=7491 wait=205 +21:13:57 pushed ckpt_w64.pt,log_w64.txt +21:14:42 waited 45s for global r=290 +21:14:51 global r=290 load 4.313 -> 4.283 +21:16:03 step 99920 loss=3.4021 lr=8.31e-05 tok=13.097B dt=7629 wait=205 +21:17:06 step 99940 loss=3.4519 lr=8.31e-05 tok=13.099B dt=7693 wait=206 +21:18:09 step 99960 loss=3.4417 lr=8.31e-05 tok=13.102B dt=7756 wait=206 +21:19:13 step 99980 loss=3.4222 lr=8.31e-05 tok=13.105B dt=7819 wait=206 +21:20:16 step 100000 loss=3.4031 lr=8.30e-05 tok=13.107B dt=7882 wait=207 +21:21:19 step 100020 loss=3.4494 lr=8.30e-05 tok=13.110B dt=7946 wait=207 +21:22:23 step 100040 loss=3.3486 lr=8.30e-05 tok=13.112B dt=8009 wait=207 +21:23:08 pushed ckpt_w64.pt,log_w64.txt +launch +21:23:46 name=w64 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 11/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:23:51 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:23:51 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +21:24:01 params 1.08B +21:24:05 resumed step=99900 seen=13.09B round=290 +21:24:05 stream salt=99900 +21:26:06 blocks compiled +21:26:06 start: accum=4, 131072 tokens per optimiser step +21:28:55 step 99901 loss=3.3233 lr=8.31e-05 tok=13.094B dt=169 wait=150 +21:28:59 step 99902 loss=3.2442 lr=8.31e-05 tok=13.094B dt=173 wait=150 +21:29:40 global r=291 load 4.288 -> 4.285 +21:30:50 step 100220 loss=3.5141 lr=8.28e-05 tok=13.098B dt=284 wait=151 +21:31:55 step 100240 loss=3.5778 lr=8.28e-05 tok=13.101B dt=349 wait=151 +21:32:59 step 100260 loss=3.5559 lr=8.28e-05 tok=13.103B dt=413 wait=151 +21:34:03 step 100280 loss=3.4278 lr=8.27e-05 tok=13.106B dt=477 wait=151 +21:35:07 step 100300 loss=3.5104 lr=8.27e-05 tok=13.109B dt=541 wait=152 +21:36:12 step 100320 loss=3.6174 lr=8.27e-05 tok=13.111B dt=606 wait=152 +21:37:16 step 100340 loss=3.4551 lr=8.27e-05 tok=13.114B dt=670 wait=152 +21:38:02 pushed ckpt_w64.pt,log_w64.txt +21:38:48 waited 46s for global r=292 +21:38:55 global r=292 load 4.291 -> 4.260 +21:39:30 step 100360 loss=3.5682 lr=8.26e-05 tok=13.116B dt=804 wait=153 +21:40:33 step 100380 loss=3.4622 lr=8.26e-05 tok=13.119B dt=868 wait=153 +21:41:37 step 100400 loss=3.4985 lr=8.26e-05 tok=13.122B dt=932 wait=153 +21:42:41 step 100420 loss=3.5219 lr=8.26e-05 tok=13.124B dt=995 wait=154 +21:43:46 step 100440 loss=3.4520 lr=8.26e-05 tok=13.127B dt=1060 wait=154 +21:44:50 step 100460 loss=3.5113 lr=8.25e-05 tok=13.129B dt=1124 wait=154 +21:45:54 step 100480 loss=3.5370 lr=8.25e-05 tok=13.132B dt=1188 wait=155 +21:46:59 step 100500 loss=3.5571 lr=8.25e-05 tok=13.135B dt=1253 wait=155 +21:47:12 pushed ckpt_w64.pt,log_w64.txt +21:48:28 waited 76s for global r=293 +21:48:37 global r=293 load 4.277 -> 4.271 +21:49:42 step 100520 loss=3.4468 lr=8.25e-05 tok=13.137B dt=1417 wait=155 +21:50:46 step 100540 loss=3.5115 lr=8.25e-05 tok=13.140B dt=1481 wait=156 +21:51:51 step 100560 loss=3.4866 lr=8.24e-05 tok=13.143B dt=1545 wait=156 +21:52:55 step 100580 loss=3.4607 lr=8.24e-05 tok=13.145B dt=1609 wait=156 +21:53:59 step 100600 loss=3.4547 lr=8.24e-05 tok=13.148B dt=1673 wait=157 +21:55:03 step 100620 loss=3.5733 lr=8.24e-05 tok=13.150B dt=1737 wait=157 +21:56:07 step 100640 loss=3.4930 lr=8.24e-05 tok=13.153B dt=1802 wait=157 +21:56:53 pushed ckpt_w64.pt,log_w64.txt +21:57:38 waited 46s for global r=294 +21:57:51 global r=294 load 4.292 -> 4.288 +21:58:24 step 100660 loss=3.5955 lr=8.23e-05 tok=13.156B dt=1939 wait=158 +21:59:28 step 100680 loss=3.4129 lr=8.23e-05 tok=13.158B dt=2003 wait=158 +22:00:32 step 100700 loss=3.4153 lr=8.23e-05 tok=13.161B dt=2067 wait=158 +22:01:36 step 100720 loss=3.5158 lr=8.23e-05 tok=13.164B dt=2130 wait=159 +22:02:40 step 100740 loss=3.4512 lr=8.22e-05 tok=13.166B dt=2194 wait=159 +22:03:44 step 100760 loss=3.5753 lr=8.22e-05 tok=13.169B dt=2259 wait=159 +22:04:49 step 100780 loss=3.4780 lr=8.22e-05 tok=13.171B dt=2323 wait=160 +22:05:54 step 100800 loss=3.5502 lr=8.22e-05 tok=13.174B dt=2388 wait=160 +22:06:09 pushed ckpt_w64.pt,log_w64.txt +22:07:10 waited 61s for global r=295 +22:07:20 global r=295 load 4.281 -> 4.272 +22:08:26 step 100820 loss=3.4747 lr=8.22e-05 tok=13.177B dt=2541 wait=160 +22:09:31 step 100840 loss=3.4935 lr=8.21e-05 tok=13.179B dt=2605 wait=161 +22:10:35 step 100860 loss=3.5263 lr=8.21e-05 tok=13.182B dt=2669 wait=161 +22:11:39 step 100880 loss=3.4532 lr=8.21e-05 tok=13.185B dt=2734 wait=161 +22:12:43 step 100900 loss=3.4575 lr=8.21e-05 tok=13.187B dt=2798 wait=162 +22:13:48 step 100920 loss=3.5793 lr=8.21e-05 tok=13.190B dt=2862 wait=162 +22:14:52 step 100940 loss=3.4629 lr=8.20e-05 tok=13.192B dt=2926 wait=162 +22:15:37 pushed ckpt_w64.pt,log_w64.txt +22:16:38 waited 61s for global r=296 +22:16:44 global r=296 load 4.320 -> 4.302 +22:17:18 step 100960 loss=3.6079 lr=8.20e-05 tok=13.195B dt=3072 wait=163 +22:18:22 step 100980 loss=3.4530 lr=8.20e-05 tok=13.198B dt=3137 wait=163 +22:19:27 step 101000 loss=3.4800 lr=8.20e-05 tok=13.200B dt=3201 wait=163 +22:20:31 step 101020 loss=3.5846 lr=8.19e-05 tok=13.203B dt=3265 wait=164 +22:21:35 step 101040 loss=3.5044 lr=8.19e-05 tok=13.206B dt=3330 wait=164 +22:22:39 step 101060 loss=3.5601 lr=8.19e-05 tok=13.208B dt=3394 wait=164 +22:23:43 step 101080 loss=3.4768 lr=8.19e-05 tok=13.211B dt=3457 wait=165 +22:24:48 step 101100 loss=3.5533 lr=8.19e-05 tok=13.213B dt=3522 wait=165 diff --git a/log_w65.txt b/log_w65.txt new file mode 100644 index 0000000000000000000000000000000000000000..f61e585aa946f23bb52549ff8cdf7d64ad1e6543 --- /dev/null +++ b/log_w65.txt @@ -0,0 +1,256 @@ +launch +20:28:39 name=w65 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 12/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:28:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:28:46 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +20:28:57 params 1.08B +20:28:59 resumed step=99450 seen=13.04B round=286 +20:28:59 stream salt=99450 +20:31:21 blocks compiled +20:31:21 start: accum=4, 131072 tokens per optimiser step +20:34:36 step 99451 loss=3.3399 lr=8.36e-05 tok=13.035B dt=196 wait=141 +20:34:40 step 99452 loss=3.2746 lr=8.36e-05 tok=13.035B dt=200 wait=141 +20:35:08 step 99460 loss=3.5503 lr=8.36e-05 tok=13.036B dt=227 wait=142 +20:36:18 step 99480 loss=3.4438 lr=8.36e-05 tok=13.039B dt=298 wait=142 +20:37:22 step 99500 loss=3.5007 lr=8.36e-05 tok=13.042B dt=362 wait=143 +20:38:26 step 99520 loss=3.5036 lr=8.35e-05 tok=13.044B dt=425 wait=143 +20:39:30 step 99540 loss=3.5567 lr=8.35e-05 tok=13.047B dt=489 wait=143 +20:40:34 step 99560 loss=3.6023 lr=8.35e-05 tok=13.050B dt=553 wait=144 +20:41:37 step 99580 loss=3.4749 lr=8.35e-05 tok=13.052B dt=617 wait=144 +20:42:41 step 99600 loss=3.5647 lr=8.35e-05 tok=13.055B dt=681 wait=145 +20:42:56 pushed ckpt_w65.pt,log_w65.txt +20:42:56 waited 0s for global r=287 +20:43:15 global r=287 load 4.347 -> 4.274 +20:44:23 step 99620 loss=3.4418 lr=8.34e-05 tok=13.057B dt=782 wait=145 +20:45:26 step 99640 loss=3.4426 lr=8.34e-05 tok=13.060B dt=845 wait=145 +20:46:30 step 99660 loss=3.4251 lr=8.34e-05 tok=13.063B dt=909 wait=146 +20:47:33 step 99680 loss=3.4000 lr=8.34e-05 tok=13.065B dt=973 wait=146 +20:48:37 step 99700 loss=3.4095 lr=8.33e-05 tok=13.068B dt=1036 wait=147 +20:49:41 step 99720 loss=3.4960 lr=8.33e-05 tok=13.070B dt=1100 wait=147 +20:50:45 step 99740 loss=3.3534 lr=8.33e-05 tok=13.073B dt=1164 wait=148 +20:51:31 pushed ckpt_w65.pt,log_w65.txt +20:51:31 waited 0s for global r=288 +20:51:39 global r=288 load 4.328 -> 4.281 +20:52:13 step 99760 loss=3.5165 lr=8.33e-05 tok=13.076B dt=1252 wait=148 +20:53:16 step 99780 loss=3.3232 lr=8.33e-05 tok=13.078B dt=1316 wait=148 +20:54:20 step 99800 loss=3.3402 lr=8.32e-05 tok=13.081B dt=1379 wait=149 +20:55:24 step 99820 loss=3.4836 lr=8.32e-05 tok=13.084B dt=1443 wait=149 +20:56:27 step 99840 loss=3.3729 lr=8.32e-05 tok=13.086B dt=1507 wait=150 +20:57:31 step 99860 loss=3.4772 lr=8.32e-05 tok=13.089B dt=1570 wait=150 +20:58:34 step 99880 loss=3.4391 lr=8.32e-05 tok=13.091B dt=1634 wait=150 +20:59:39 step 99900 loss=3.4068 lr=8.31e-05 tok=13.094B dt=1698 wait=151 +20:59:53 pushed ckpt_w65.pt,log_w65.txt +21:01:09 waited 76s for global r=289 +21:01:21 global r=289 load 4.271 -> 4.284 +21:02:27 step 99920 loss=3.4419 lr=8.31e-05 tok=13.097B dt=1866 wait=151 +21:03:30 step 99940 loss=3.4579 lr=8.31e-05 tok=13.099B dt=1930 wait=152 +21:04:34 step 99960 loss=3.4707 lr=8.31e-05 tok=13.102B dt=1993 wait=152 +21:05:37 step 99980 loss=3.3940 lr=8.31e-05 tok=13.105B dt=2057 wait=152 +21:06:41 step 100000 loss=3.3990 lr=8.30e-05 tok=13.107B dt=2120 wait=153 +21:07:45 step 100020 loss=3.4687 lr=8.30e-05 tok=13.110B dt=2184 wait=153 +21:08:48 step 100040 loss=3.3929 lr=8.30e-05 tok=13.112B dt=2248 wait=154 +21:09:33 pushed ckpt_w65.pt,log_w65.txt +21:13:08 step 100060 loss=3.4602 lr=8.30e-05 tok=13.115B dt=2507 wait=154 +21:14:11 step 100080 loss=3.4091 lr=8.29e-05 tok=13.118B dt=2571 wait=154 +21:15:14 step 100100 loss=3.3633 lr=8.29e-05 tok=13.120B dt=2634 wait=155 +21:16:18 step 100120 loss=3.3146 lr=8.29e-05 tok=13.123B dt=2697 wait=155 +21:17:22 step 100140 loss=3.3545 lr=8.29e-05 tok=13.126B dt=2761 wait=155 +21:18:25 step 100160 loss=3.4245 lr=8.29e-05 tok=13.128B dt=2825 wait=156 +21:19:28 step 100180 loss=3.4150 lr=8.28e-05 tok=13.131B dt=2888 wait=156 +21:20:31 step 100200 loss=3.5999 lr=8.28e-05 tok=13.133B dt=2951 wait=156 +21:20:45 pushed ckpt_w65.pt,log_w65.txt +21:20:45 waited 0s for global r=290 +21:20:55 global r=290 load 4.305 -> 4.283 +21:22:01 step 100220 loss=3.3682 lr=8.28e-05 tok=13.136B dt=3041 wait=157 +21:23:05 step 100240 loss=3.5070 lr=8.28e-05 tok=13.139B dt=3104 wait=157 +launch +21:23:48 name=w65 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 12/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:23:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:23:54 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +21:24:06 params 1.08B +21:24:15 resumed step=100200 seen=13.13B round=290 +21:24:15 stream salt=100200 +21:26:03 blocks compiled +21:26:03 start: accum=4, 131072 tokens per optimiser step +21:29:02 step 100201 loss=3.2016 lr=8.28e-05 tok=13.134B dt=179 wait=158 +21:29:05 step 100202 loss=3.2654 lr=8.28e-05 tok=13.134B dt=182 wait=158 +21:29:51 global r=291 load 4.279 -> 4.285 +21:29:55 step 100201 loss=3.5369 lr=8.28e-05 tok=13.135B dt=232 wait=159 +21:29:58 step 100202 loss=3.5842 lr=8.28e-05 tok=13.135B dt=235 wait=159 +21:30:58 step 100220 loss=3.4319 lr=8.28e-05 tok=13.137B dt=295 wait=159 +21:32:01 step 100240 loss=3.4942 lr=8.28e-05 tok=13.140B dt=358 wait=159 +21:33:05 step 100260 loss=3.4619 lr=8.28e-05 tok=13.143B dt=422 wait=160 +21:34:09 step 100280 loss=3.4164 lr=8.27e-05 tok=13.145B dt=486 wait=160 +21:35:12 step 100300 loss=3.4428 lr=8.27e-05 tok=13.148B dt=549 wait=160 +21:36:16 step 100320 loss=3.5121 lr=8.27e-05 tok=13.150B dt=613 wait=161 +21:37:19 step 100340 loss=3.4252 lr=8.27e-05 tok=13.153B dt=676 wait=161 +21:38:05 pushed ckpt_w65.pt,log_w65.txt +21:38:51 waited 46s for global r=292 +21:39:01 global r=292 load 4.275 -> 4.260 +21:39:35 step 100360 loss=3.4883 lr=8.26e-05 tok=13.156B dt=812 wait=161 +21:40:39 step 100380 loss=3.3864 lr=8.26e-05 tok=13.158B dt=876 wait=162 +21:41:43 step 100400 loss=3.3957 lr=8.26e-05 tok=13.161B dt=940 wait=162 +21:42:47 step 100420 loss=3.5084 lr=8.26e-05 tok=13.164B dt=1004 wait=162 +21:43:50 step 100440 loss=3.3713 lr=8.26e-05 tok=13.166B dt=1067 wait=162 +21:44:54 step 100460 loss=3.4720 lr=8.25e-05 tok=13.169B dt=1131 wait=163 +21:45:57 step 100480 loss=3.4363 lr=8.25e-05 tok=13.171B dt=1194 wait=163 +21:47:01 step 100500 loss=3.4500 lr=8.25e-05 tok=13.174B dt=1258 wait=163 +21:47:16 pushed ckpt_w65.pt,log_w65.txt +21:48:17 waited 61s for global r=293 +21:48:27 global r=293 load 4.293 -> 4.271 +21:49:33 step 100520 loss=3.3919 lr=8.25e-05 tok=13.177B dt=1410 wait=164 +21:50:36 step 100540 loss=3.4535 lr=8.25e-05 tok=13.179B dt=1473 wait=164 +21:51:40 step 100560 loss=3.4632 lr=8.24e-05 tok=13.182B dt=1537 wait=164 +21:52:43 step 100580 loss=3.3655 lr=8.24e-05 tok=13.185B dt=1600 wait=165 +21:53:47 step 100600 loss=3.4459 lr=8.24e-05 tok=13.187B dt=1664 wait=165 +21:54:51 step 100620 loss=3.5061 lr=8.24e-05 tok=13.190B dt=1728 wait=165 +21:55:54 step 100640 loss=3.3640 lr=8.24e-05 tok=13.192B dt=1791 wait=166 +21:56:39 pushed ckpt_w65.pt,log_w65.txt +21:57:39 waited 61s for global r=294 +21:57:53 global r=294 load 4.311 -> 4.288 +21:58:26 step 100660 loss=3.5246 lr=8.23e-05 tok=13.195B dt=1943 wait=166 +21:59:30 step 100680 loss=3.3826 lr=8.23e-05 tok=13.198B dt=2007 wait=166 +22:00:33 step 100700 loss=3.3707 lr=8.23e-05 tok=13.200B dt=2070 wait=167 +22:01:37 step 100720 loss=3.4882 lr=8.23e-05 tok=13.203B dt=2134 wait=167 +22:02:41 step 100740 loss=3.3419 lr=8.22e-05 tok=13.206B dt=2199 wait=167 +22:03:45 step 100760 loss=3.4925 lr=8.22e-05 tok=13.208B dt=2262 wait=168 +22:04:50 step 100780 loss=3.4889 lr=8.22e-05 tok=13.211B dt=2327 wait=168 +22:05:53 step 100800 loss=3.4035 lr=8.22e-05 tok=13.213B dt=2390 wait=168 +22:06:09 pushed ckpt_w65.pt,log_w65.txt +22:07:11 waited 62s for global r=295 +22:07:20 global r=295 load 4.286 -> 4.272 +22:08:25 step 100820 loss=3.3949 lr=8.22e-05 tok=13.216B dt=2542 wait=168 +22:09:29 step 100840 loss=3.4957 lr=8.21e-05 tok=13.219B dt=2606 wait=169 +22:10:33 step 100860 loss=3.3761 lr=8.21e-05 tok=13.221B dt=2671 wait=169 +22:11:37 step 100880 loss=3.2832 lr=8.21e-05 tok=13.224B dt=2734 wait=169 +22:12:41 step 100900 loss=3.4568 lr=8.21e-05 tok=13.226B dt=2798 wait=170 +22:13:44 step 100920 loss=3.4911 lr=8.21e-05 tok=13.229B dt=2861 wait=170 +22:14:48 step 100940 loss=3.3702 lr=8.20e-05 tok=13.232B dt=2925 wait=170 +22:15:34 pushed ckpt_w65.pt,log_w65.txt +22:16:35 waited 61s for global r=296 +22:16:44 global r=296 load 4.324 -> 4.302 +22:17:18 step 100960 loss=3.5422 lr=8.20e-05 tok=13.234B dt=3075 wait=171 +22:18:21 step 100980 loss=3.4055 lr=8.20e-05 tok=13.237B dt=3139 wait=171 +22:19:25 step 101000 loss=3.4074 lr=8.20e-05 tok=13.240B dt=3203 wait=171 +22:20:29 step 101020 loss=3.4976 lr=8.19e-05 tok=13.242B dt=3266 wait=172 +22:21:33 step 101040 loss=3.3222 lr=8.19e-05 tok=13.245B dt=3330 wait=172 +22:22:36 step 101060 loss=3.5060 lr=8.19e-05 tok=13.247B dt=3394 wait=172 +22:23:40 step 101080 loss=3.4528 lr=8.19e-05 tok=13.250B dt=3457 wait=173 +22:24:44 step 101100 loss=3.4492 lr=8.19e-05 tok=13.253B dt=3521 wait=173 +22:25:01 pushed ckpt_w65.pt,log_w65.txt +22:26:01 waited 61s for global r=297 +22:26:09 global r=297 load 4.306 -> 4.288 +22:27:14 step 101120 loss=3.4291 lr=8.18e-05 tok=13.255B dt=3671 wait=173 +22:28:17 step 101140 loss=3.4598 lr=8.18e-05 tok=13.258B dt=3735 wait=174 +22:29:21 step 101160 loss=3.5032 lr=8.18e-05 tok=13.261B dt=3798 wait=174 +22:30:25 step 101180 loss=3.4282 lr=8.18e-05 tok=13.263B dt=3862 wait=174 +22:31:28 step 101200 loss=3.3937 lr=8.18e-05 tok=13.266B dt=3925 wait=175 +22:32:32 step 101220 loss=3.5175 lr=8.17e-05 tok=13.268B dt=3989 wait=175 +22:33:35 step 101240 loss=3.3788 lr=8.17e-05 tok=13.271B dt=4052 wait=175 +22:34:28 pushed ckpt_w65.pt,log_w65.txt +22:37:29 waited 181s for global r=298 +22:37:36 global r=298 load 4.289 -> 4.283 +22:38:09 step 101260 loss=3.5062 lr=8.17e-05 tok=13.274B dt=4326 wait=176 +22:39:12 step 101280 loss=3.3550 lr=8.17e-05 tok=13.276B dt=4389 wait=176 +22:40:16 step 101300 loss=3.4065 lr=8.17e-05 tok=13.279B dt=4453 wait=176 +22:41:20 step 101320 loss=3.4469 lr=8.16e-05 tok=13.282B dt=4517 wait=176 +22:42:23 step 101340 loss=3.3619 lr=8.16e-05 tok=13.284B dt=4580 wait=177 +22:43:27 step 101360 loss=3.4884 lr=8.16e-05 tok=13.287B dt=4644 wait=177 +22:44:31 step 101380 loss=3.4022 lr=8.16e-05 tok=13.289B dt=4708 wait=177 +22:45:35 step 101400 loss=3.3936 lr=8.15e-05 tok=13.292B dt=4772 wait=178 +22:45:50 pushed ckpt_w65.pt,log_w65.txt +22:49:33 global r=299 load 4.320 -> 4.282 +22:50:36 step 101420 loss=3.3774 lr=8.15e-05 tok=13.296B dt=5073 wait=178 +22:51:39 step 101440 loss=3.4763 lr=8.15e-05 tok=13.299B dt=5136 wait=179 +22:52:43 step 101460 loss=3.4376 lr=8.15e-05 tok=13.301B dt=5200 wait=179 +22:53:46 step 101480 loss=3.4042 lr=8.15e-05 tok=13.304B dt=5263 wait=179 +22:54:50 step 101500 loss=3.3895 lr=8.14e-05 tok=13.306B dt=5327 wait=180 +22:55:54 step 101520 loss=3.4850 lr=8.14e-05 tok=13.309B dt=5391 wait=180 +22:56:57 step 101540 loss=3.3580 lr=8.14e-05 tok=13.312B dt=5454 wait=180 +22:57:44 pushed ckpt_w65.pt,log_w65.txt +23:00:45 waited 182s for global r=300 +23:00:53 global r=300 load 4.309 -> 4.287 +23:01:28 step 101560 loss=3.4997 lr=8.14e-05 tok=13.314B dt=5725 wait=180 +23:02:32 step 101580 loss=3.3856 lr=8.14e-05 tok=13.317B dt=5789 wait=181 +23:03:35 step 101600 loss=3.4221 lr=8.13e-05 tok=13.320B dt=5852 wait=181 +23:04:39 step 101620 loss=3.4945 lr=8.13e-05 tok=13.322B dt=5916 wait=181 +23:05:43 step 101640 loss=3.4201 lr=8.13e-05 tok=13.325B dt=5980 wait=182 +23:06:46 step 101660 loss=3.4360 lr=8.13e-05 tok=13.327B dt=6043 wait=182 +23:07:50 step 101680 loss=3.3896 lr=8.12e-05 tok=13.330B dt=6107 wait=182 +23:08:55 step 101700 loss=3.4158 lr=8.12e-05 tok=13.333B dt=6172 wait=183 +23:09:08 pushed ckpt_w65.pt,log_w65.txt +23:12:53 global r=301 load 4.299 -> 4.294 +23:13:57 step 101720 loss=3.3551 lr=8.12e-05 tok=13.337B dt=6474 wait=183 +23:15:01 step 101740 loss=3.3715 lr=8.12e-05 tok=13.339B dt=6538 wait=183 +23:16:04 step 101760 loss=3.4424 lr=8.12e-05 tok=13.342B dt=6601 wait=184 +23:17:08 step 101780 loss=3.4188 lr=8.11e-05 tok=13.344B dt=6665 wait=184 +23:18:12 step 101800 loss=3.3720 lr=8.11e-05 tok=13.347B dt=6729 wait=184 +23:19:16 step 101820 loss=3.4568 lr=8.11e-05 tok=13.350B dt=6793 wait=185 +23:20:19 step 101840 loss=3.3815 lr=8.11e-05 tok=13.352B dt=6857 wait=185 +23:21:04 pushed ckpt_w65.pt,log_w65.txt +23:24:39 step 101860 loss=3.4709 lr=8.11e-05 tok=13.355B dt=7116 wait=185 +23:25:42 step 101880 loss=3.4161 lr=8.10e-05 tok=13.358B dt=7180 wait=186 +23:26:46 step 101900 loss=3.3611 lr=8.10e-05 tok=13.360B dt=7243 wait=186 +23:27:49 step 101920 loss=3.2675 lr=8.10e-05 tok=13.363B dt=7307 wait=186 +23:28:53 step 101940 loss=3.3682 lr=8.10e-05 tok=13.365B dt=7370 wait=187 +23:29:57 step 101960 loss=3.4458 lr=8.10e-05 tok=13.368B dt=7434 wait=187 +23:31:00 step 101980 loss=3.4187 lr=8.09e-05 tok=13.371B dt=7497 wait=187 +23:32:04 step 102000 loss=3.5979 lr=8.09e-05 tok=13.373B dt=7561 wait=188 +23:32:20 pushed ckpt_w65.pt,log_w65.txt +23:36:27 step 102020 loss=3.4568 lr=8.09e-05 tok=13.376B dt=7824 wait=188 +23:37:31 step 102040 loss=3.4450 lr=8.09e-05 tok=13.379B dt=7888 wait=188 +23:38:34 step 102060 loss=3.4501 lr=8.08e-05 tok=13.381B dt=7952 wait=189 +23:39:38 step 102080 loss=3.4644 lr=8.08e-05 tok=13.384B dt=8015 wait=189 +23:40:42 step 102100 loss=3.4815 lr=8.08e-05 tok=13.386B dt=8079 wait=189 +23:41:46 step 102120 loss=3.4595 lr=8.08e-05 tok=13.389B dt=8143 wait=190 +23:42:49 step 102140 loss=3.3516 lr=8.08e-05 tok=13.392B dt=8206 wait=190 +23:43:35 pushed ckpt_w65.pt,log_w65.txt +23:47:11 step 102160 loss=3.4594 lr=8.07e-05 tok=13.394B dt=8468 wait=190 +23:48:14 step 102180 loss=3.4304 lr=8.07e-05 tok=13.397B dt=8531 wait=191 +23:49:18 step 102200 loss=3.4956 lr=8.07e-05 tok=13.399B dt=8595 wait=191 +23:50:21 step 102220 loss=3.3968 lr=8.07e-05 tok=13.402B dt=8658 wait=191 +23:51:25 step 102240 loss=3.4733 lr=8.07e-05 tok=13.405B dt=8722 wait=192 +23:52:29 step 102260 loss=3.4453 lr=8.06e-05 tok=13.407B dt=8786 wait=192 +23:53:33 step 102280 loss=3.4627 lr=8.06e-05 tok=13.410B dt=8850 wait=192 +23:54:37 step 102300 loss=3.3994 lr=8.06e-05 tok=13.413B dt=8914 wait=192 +23:54:50 pushed ckpt_w65.pt,log_w65.txt +23:58:58 step 102320 loss=3.3380 lr=8.06e-05 tok=13.415B dt=9175 wait=193 +00:00:01 step 102340 loss=3.4819 lr=8.06e-05 tok=13.418B dt=9238 wait=193 +00:01:05 step 102360 loss=3.4341 lr=8.05e-05 tok=13.420B dt=9302 wait=193 +00:02:08 step 102380 loss=3.3965 lr=8.05e-05 tok=13.423B dt=9366 wait=194 +00:03:12 step 102400 loss=3.4633 lr=8.05e-05 tok=13.426B dt=9429 wait=194 +00:04:16 step 102420 loss=3.4806 lr=8.05e-05 tok=13.428B dt=9493 wait=194 +00:05:19 step 102440 loss=3.4269 lr=8.04e-05 tok=13.431B dt=9556 wait=195 +00:06:09 pushed ckpt_w65.pt,log_w65.txt +00:09:44 step 102460 loss=3.3352 lr=8.04e-05 tok=13.434B dt=9821 wait=195 +00:10:48 step 102480 loss=3.3966 lr=8.04e-05 tok=13.436B dt=9885 wait=195 +00:11:52 step 102500 loss=3.5243 lr=8.04e-05 tok=13.439B dt=9949 wait=196 +00:12:56 step 102520 loss=3.4028 lr=8.04e-05 tok=13.441B dt=10013 wait=196 +00:13:59 step 102540 loss=3.4175 lr=8.03e-05 tok=13.444B dt=10076 wait=196 +00:15:03 step 102560 loss=3.4974 lr=8.03e-05 tok=13.447B dt=10140 wait=196 +00:16:07 step 102580 loss=3.5309 lr=8.03e-05 tok=13.449B dt=10204 wait=197 +00:17:11 step 102600 loss=3.3836 lr=8.03e-05 tok=13.452B dt=10268 wait=197 +00:17:24 pushed ckpt_w65.pt,log_w65.txt +00:21:32 step 102620 loss=3.4690 lr=8.03e-05 tok=13.455B dt=10529 wait=197 +00:22:35 step 102640 loss=3.4673 lr=8.02e-05 tok=13.457B dt=10592 wait=198 +00:23:39 step 102660 loss=3.3032 lr=8.02e-05 tok=13.460B dt=10656 wait=198 +00:24:43 step 102680 loss=3.4119 lr=8.02e-05 tok=13.462B dt=10720 wait=198 +00:25:46 step 102700 loss=3.3793 lr=8.02e-05 tok=13.465B dt=10783 wait=199 +00:26:50 step 102720 loss=3.4934 lr=8.01e-05 tok=13.468B dt=10847 wait=199 +00:27:53 step 102740 loss=3.4192 lr=8.01e-05 tok=13.470B dt=10911 wait=199 +00:28:38 pushed ckpt_w65.pt,log_w65.txt +00:32:13 step 102760 loss=3.5838 lr=8.01e-05 tok=13.473B dt=11170 wait=200 +00:33:16 step 102780 loss=3.4402 lr=8.01e-05 tok=13.476B dt=11233 wait=200 +00:34:20 step 102800 loss=3.5646 lr=8.01e-05 tok=13.478B dt=11297 wait=200 +00:35:24 step 102820 loss=3.4694 lr=8.00e-05 tok=13.481B dt=11361 wait=201 +00:36:27 step 102840 loss=3.4514 lr=8.00e-05 tok=13.483B dt=11424 wait=201 +00:37:31 step 102860 loss=3.3919 lr=8.00e-05 tok=13.486B dt=11488 wait=201 +00:38:34 step 102880 loss=3.4625 lr=8.00e-05 tok=13.489B dt=11551 wait=202 +00:39:38 step 102900 loss=3.4163 lr=8.00e-05 tok=13.491B dt=11615 wait=202 +00:39:52 pushed ckpt_w65.pt,log_w65.txt +00:43:59 step 102920 loss=3.4641 lr=7.99e-05 tok=13.494B dt=11876 wait=202 +00:45:03 step 102940 loss=3.4942 lr=7.99e-05 tok=13.496B dt=11940 wait=203 +00:46:07 step 102960 loss=3.5462 lr=7.99e-05 tok=13.499B dt=12004 wait=203 +00:47:11 step 102980 loss=3.4311 lr=7.99e-05 tok=13.502B dt=12068 wait=203 +00:48:14 step 103000 loss=3.5965 lr=7.99e-05 tok=13.504B dt=12132 wait=204 +00:49:18 step 103020 loss=3.5078 lr=7.98e-05 tok=13.507B dt=12195 wait=204 +00:50:22 step 103040 loss=3.2429 lr=7.98e-05 tok=13.510B dt=12259 wait=204 diff --git a/log_w66.txt b/log_w66.txt new file mode 100644 index 0000000000000000000000000000000000000000..6c2b0c0c684c45973974278f1b435a9306f1af13 --- /dev/null +++ b/log_w66.txt @@ -0,0 +1,338 @@ +launch +18:01:10 name=w66 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 13/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:01:16 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:01:16 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:01:25 params 1.08B +18:01:27 resumed step=97800 seen=12.82B round=280 +18:01:27 stream salt=97800 +18:03:30 blocks compiled +18:03:30 start: accum=4, 131072 tokens per optimiser step +18:07:03 step 97801 loss=3.2974 lr=8.54e-05 tok=12.819B dt=213 wait=177 +18:07:07 step 97802 loss=3.3292 lr=8.54e-05 tok=12.819B dt=217 wait=177 +18:08:07 step 97820 loss=3.5220 lr=8.53e-05 tok=12.821B dt=277 wait=178 +18:09:14 step 97840 loss=3.5093 lr=8.53e-05 tok=12.824B dt=344 wait=178 +18:10:17 step 97860 loss=3.5412 lr=8.53e-05 tok=12.827B dt=407 wait=178 +18:11:20 step 97880 loss=3.5262 lr=8.53e-05 tok=12.829B dt=471 wait=179 +18:12:24 step 97900 loss=3.6180 lr=8.53e-05 tok=12.832B dt=534 wait=179 +18:13:28 step 97920 loss=3.5829 lr=8.52e-05 tok=12.835B dt=598 wait=179 +18:14:31 step 97940 loss=3.5029 lr=8.52e-05 tok=12.837B dt=661 wait=180 +18:15:17 pushed ckpt_w66.pt,log_w66.txt +18:17:03 waited 106s for global r=281 +18:17:16 global r=281 load 4.289 -> 4.263 +18:17:54 step 97960 loss=3.6024 lr=8.52e-05 tok=12.840B dt=864 wait=180 +18:18:58 step 97980 loss=3.4439 lr=8.52e-05 tok=12.842B dt=928 wait=180 +18:20:01 step 98000 loss=3.5109 lr=8.52e-05 tok=12.845B dt=991 wait=181 +18:21:04 step 98020 loss=3.5598 lr=8.51e-05 tok=12.848B dt=1054 wait=181 +18:22:07 step 98040 loss=3.4351 lr=8.51e-05 tok=12.850B dt=1117 wait=181 +18:23:10 step 98060 loss=3.5121 lr=8.51e-05 tok=12.853B dt=1180 wait=182 +18:24:14 step 98080 loss=3.4563 lr=8.51e-05 tok=12.856B dt=1244 wait=182 +18:25:17 step 98100 loss=3.4936 lr=8.50e-05 tok=12.858B dt=1307 wait=182 +18:25:33 pushed ckpt_w66.pt,log_w66.txt +18:29:46 step 98120 loss=3.5032 lr=8.50e-05 tok=12.861B dt=1576 wait=183 +18:30:50 step 98140 loss=3.5385 lr=8.50e-05 tok=12.863B dt=1640 wait=183 +18:31:53 step 98160 loss=3.3909 lr=8.50e-05 tok=12.866B dt=1704 wait=183 +18:32:57 step 98180 loss=3.3572 lr=8.50e-05 tok=12.869B dt=1767 wait=184 +18:34:01 step 98200 loss=3.3489 lr=8.49e-05 tok=12.871B dt=1831 wait=184 +18:35:04 step 98220 loss=3.5491 lr=8.49e-05 tok=12.874B dt=1894 wait=185 +18:36:07 step 98240 loss=3.5139 lr=8.49e-05 tok=12.877B dt=1957 wait=185 +18:36:54 pushed ckpt_w66.pt,log_w66.txt +18:40:32 step 98260 loss=3.5142 lr=8.49e-05 tok=12.879B dt=2222 wait=185 +18:41:36 step 98280 loss=3.6145 lr=8.49e-05 tok=12.882B dt=2286 wait=186 +18:42:40 step 98300 loss=3.4951 lr=8.48e-05 tok=12.884B dt=2350 wait=186 +18:43:44 step 98320 loss=3.5418 lr=8.48e-05 tok=12.887B dt=2414 wait=186 +18:44:47 step 98340 loss=3.5229 lr=8.48e-05 tok=12.890B dt=2477 wait=187 +18:45:51 step 98360 loss=3.5354 lr=8.48e-05 tok=12.892B dt=2541 wait=187 +18:46:54 step 98380 loss=3.4232 lr=8.47e-05 tok=12.895B dt=2604 wait=187 +18:47:58 step 98400 loss=3.5374 lr=8.47e-05 tok=12.897B dt=2668 wait=188 +18:48:10 pushed ckpt_w66.pt,log_w66.txt +18:52:21 step 98420 loss=3.3755 lr=8.47e-05 tok=12.900B dt=2931 wait=188 +launch +18:52:41 name=w66 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 13/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:52:46 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:52:46 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:52:55 params 1.08B +18:52:59 resumed step=98400 seen=12.90B round=281 +18:52:59 stream salt=98400 +18:55:04 blocks compiled +18:55:04 start: accum=4, 131072 tokens per optimiser step +18:58:10 step 98401 loss=3.0326 lr=8.47e-05 tok=12.898B dt=186 wait=169 +18:58:13 step 98402 loss=3.1409 lr=8.47e-05 tok=12.898B dt=189 wait=169 +18:59:15 step 98420 loss=3.4512 lr=8.47e-05 tok=12.900B dt=251 wait=169 +19:00:21 step 98440 loss=3.4738 lr=8.47e-05 tok=12.903B dt=316 wait=170 +19:01:24 step 98460 loss=3.5533 lr=8.47e-05 tok=12.905B dt=380 wait=170 +19:02:27 step 98480 loss=3.4770 lr=8.46e-05 tok=12.908B dt=443 wait=171 +19:03:31 step 98500 loss=3.5701 lr=8.46e-05 tok=12.911B dt=507 wait=171 +19:04:35 step 98520 loss=3.5325 lr=8.46e-05 tok=12.913B dt=571 wait=171 +19:05:38 step 98540 loss=3.4723 lr=8.46e-05 tok=12.916B dt=634 wait=172 +19:06:24 pushed ckpt_w66.pt,log_w66.txt +19:06:24 waited 0s for global r=282 +19:06:39 global r=282 load 4.296 -> 4.262 +19:07:16 step 98560 loss=3.5372 lr=8.46e-05 tok=12.918B dt=732 wait=172 +19:08:19 step 98580 loss=3.4509 lr=8.45e-05 tok=12.921B dt=795 wait=172 +19:09:22 step 98600 loss=3.4521 lr=8.45e-05 tok=12.924B dt=858 wait=173 +19:10:26 step 98620 loss=3.5088 lr=8.45e-05 tok=12.926B dt=922 wait=173 +19:11:29 step 98640 loss=3.4168 lr=8.45e-05 tok=12.929B dt=985 wait=173 +19:12:32 step 98660 loss=3.5191 lr=8.45e-05 tok=12.932B dt=1048 wait=174 +19:13:35 step 98680 loss=3.3961 lr=8.44e-05 tok=12.934B dt=1111 wait=174 +19:14:39 step 98700 loss=3.4655 lr=8.44e-05 tok=12.937B dt=1175 wait=174 +19:14:53 pushed ckpt_w66.pt,log_w66.txt +19:19:09 step 98720 loss=3.4257 lr=8.44e-05 tok=12.939B dt=1445 wait=175 +19:20:12 step 98740 loss=3.4064 lr=8.44e-05 tok=12.942B dt=1508 wait=175 +19:21:15 step 98760 loss=3.3840 lr=8.43e-05 tok=12.945B dt=1571 wait=176 +19:22:18 step 98780 loss=3.4170 lr=8.43e-05 tok=12.947B dt=1634 wait=176 +19:23:21 step 98800 loss=3.3307 lr=8.43e-05 tok=12.950B dt=1697 wait=176 +19:24:24 step 98820 loss=3.5203 lr=8.43e-05 tok=12.953B dt=1760 wait=177 +19:25:28 step 98840 loss=3.4281 lr=8.43e-05 tok=12.955B dt=1824 wait=177 +19:26:15 pushed ckpt_w66.pt,log_w66.txt +19:27:01 waited 45s for global r=283 +19:27:08 global r=283 load 4.282 -> 4.271 +19:27:45 step 98860 loss=3.6220 lr=8.42e-05 tok=12.958B dt=1961 wait=177 +launch +19:28:38 name=w66 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 13/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:28:43 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:28:43 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +19:28:53 fp8 rowwise on block linears +19:28:53 params 1.08B +19:28:57 resumed step=98850 seen=12.96B round=283 +19:28:57 stream salt=98850 +19:30:59 blocks compiled +19:30:59 start: accum=4, 131072 tokens per optimiser step +19:34:15 step 98851 loss=3.1313 lr=8.42e-05 tok=12.957B dt=197 wait=157 +19:34:19 step 98852 loss=3.1661 lr=8.42e-05 tok=12.957B dt=200 wait=157 +19:34:47 step 98860 loss=3.5093 lr=8.42e-05 tok=12.958B dt=229 wait=157 +19:35:59 step 98880 loss=3.4059 lr=8.42e-05 tok=12.960B dt=300 wait=158 +19:37:04 step 98900 loss=3.4942 lr=8.42e-05 tok=12.963B dt=365 wait=158 +19:38:10 step 98920 loss=3.4464 lr=8.42e-05 tok=12.966B dt=431 wait=159 +19:39:16 step 98940 loss=3.5571 lr=8.42e-05 tok=12.968B dt=497 wait=159 +19:40:22 step 98960 loss=3.6677 lr=8.41e-05 tok=12.971B dt=563 wait=159 +19:41:27 step 98980 loss=3.4905 lr=8.41e-05 tok=12.974B dt=629 wait=160 +19:42:33 step 99000 loss=3.5226 lr=8.41e-05 tok=12.976B dt=694 wait=160 +19:42:49 pushed ckpt_w66.pt,log_w66.txt +19:42:49 waited 0s for global r=284 +19:43:07 global r=284 load 4.290 -> 4.281 +19:44:19 step 99020 loss=3.4767 lr=8.41e-05 tok=12.979B dt=800 wait=160 +19:45:24 step 99040 loss=3.4713 lr=8.40e-05 tok=12.981B dt=866 wait=161 +19:46:30 step 99060 loss=3.4601 lr=8.40e-05 tok=12.984B dt=931 wait=161 +19:47:35 step 99080 loss=3.4155 lr=8.40e-05 tok=12.987B dt=997 wait=161 +19:48:42 step 99100 loss=3.3822 lr=8.40e-05 tok=12.989B dt=1063 wait=162 +launch +19:49:08 name=w66 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 13/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +19:49:12 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +19:49:12 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +19:49:23 fp8 rowwise on block linears +19:49:23 params 1.08B +19:49:27 resumed step=99000 seen=12.98B round=284 +19:49:27 stream salt=99000 +19:51:27 blocks compiled +19:51:27 start: accum=4, 131072 tokens per optimiser step +19:54:29 step 99001 loss=3.3725 lr=8.41e-05 tok=12.976B dt=182 wait=159 +19:54:33 step 99002 loss=3.3690 lr=8.41e-05 tok=12.976B dt=186 wait=159 +19:55:37 step 99020 loss=3.4390 lr=8.41e-05 tok=12.979B dt=251 wait=160 +19:56:47 step 99040 loss=3.5024 lr=8.40e-05 tok=12.981B dt=320 wait=160 +19:57:53 step 99060 loss=3.5321 lr=8.40e-05 tok=12.984B dt=386 wait=161 +19:58:59 step 99080 loss=3.5417 lr=8.40e-05 tok=12.987B dt=452 wait=161 +20:00:05 step 99100 loss=3.6345 lr=8.40e-05 tok=12.989B dt=519 wait=161 +launch +20:01:13 name=w66 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 13/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:01:17 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:01:18 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +20:01:27 params 1.08B +20:01:32 resumed step=99000 seen=12.98B round=284 +20:01:32 stream salt=99000 +20:03:38 blocks compiled +20:03:38 start: accum=4, 131072 tokens per optimiser step +20:06:18 step 99001 loss=3.3698 lr=8.41e-05 tok=12.976B dt=160 wait=145 +20:06:22 step 99002 loss=3.3666 lr=8.41e-05 tok=12.976B dt=163 wait=145 +20:07:23 step 99020 loss=3.4363 lr=8.41e-05 tok=12.979B dt=225 wait=146 +20:08:30 step 99040 loss=3.5003 lr=8.40e-05 tok=12.981B dt=291 wait=146 +20:09:34 step 99060 loss=3.5296 lr=8.40e-05 tok=12.984B dt=356 wait=147 +20:10:38 step 99080 loss=3.5391 lr=8.40e-05 tok=12.987B dt=420 wait=147 +20:11:43 step 99100 loss=3.6321 lr=8.40e-05 tok=12.989B dt=484 wait=147 +20:12:47 step 99120 loss=3.5689 lr=8.40e-05 tok=12.992B dt=548 wait=148 +20:13:50 step 99140 loss=3.5331 lr=8.39e-05 tok=12.994B dt=612 wait=148 +20:14:37 pushed ckpt_w66.pt,log_w66.txt +20:14:37 waited 0s for global r=286 +20:14:55 global r=286 load 4.306 -> 4.293 +20:15:30 step 99160 loss=3.5936 lr=8.39e-05 tok=12.997B dt=712 wait=148 +20:16:34 step 99180 loss=3.4670 lr=8.39e-05 tok=13.000B dt=776 wait=149 +20:17:38 step 99200 loss=3.4809 lr=8.39e-05 tok=13.002B dt=839 wait=149 +20:18:42 step 99220 loss=3.5475 lr=8.39e-05 tok=13.005B dt=903 wait=149 +20:19:46 step 99240 loss=3.4534 lr=8.38e-05 tok=13.008B dt=968 wait=150 +20:20:50 step 99260 loss=3.5220 lr=8.38e-05 tok=13.010B dt=1032 wait=150 +20:21:54 step 99280 loss=3.4553 lr=8.38e-05 tok=13.013B dt=1095 wait=150 +20:22:58 step 99300 loss=3.4763 lr=8.38e-05 tok=13.015B dt=1160 wait=151 +20:23:12 pushed ckpt_w66.pt,log_w66.txt +20:27:20 step 99320 loss=3.5186 lr=8.38e-05 tok=13.018B dt=1422 wait=151 +20:28:24 step 99340 loss=3.5131 lr=8.37e-05 tok=13.021B dt=1486 wait=151 +20:29:28 step 99360 loss=3.4428 lr=8.37e-05 tok=13.023B dt=1550 wait=152 +20:30:31 step 99380 loss=3.3847 lr=8.37e-05 tok=13.026B dt=1613 wait=152 +20:31:35 step 99400 loss=3.4115 lr=8.37e-05 tok=13.029B dt=1677 wait=152 +20:32:39 step 99420 loss=3.5561 lr=8.36e-05 tok=13.031B dt=1741 wait=153 +20:33:43 step 99440 loss=3.4187 lr=8.36e-05 tok=13.034B dt=1805 wait=153 +20:34:28 pushed ckpt_w66.pt,log_w66.txt +20:37:30 waited 181s for global r=287 +20:37:40 global r=287 load 4.332 -> 4.274 +20:38:14 step 99460 loss=3.5676 lr=8.36e-05 tok=13.036B dt=2075 wait=154 +20:39:17 step 99480 loss=3.4104 lr=8.36e-05 tok=13.039B dt=2139 wait=154 +20:40:22 step 99500 loss=3.4265 lr=8.36e-05 tok=13.042B dt=2203 wait=154 +20:41:26 step 99520 loss=3.5304 lr=8.35e-05 tok=13.044B dt=2268 wait=154 +20:42:30 step 99540 loss=3.4333 lr=8.35e-05 tok=13.047B dt=2332 wait=155 +20:43:34 step 99560 loss=3.5179 lr=8.35e-05 tok=13.050B dt=2395 wait=155 +20:44:38 step 99580 loss=3.4596 lr=8.35e-05 tok=13.052B dt=2459 wait=155 +20:45:42 step 99600 loss=3.5669 lr=8.35e-05 tok=13.055B dt=2524 wait=156 +20:45:54 pushed ckpt_w66.pt,log_w66.txt +20:50:02 step 99620 loss=3.5376 lr=8.34e-05 tok=13.057B dt=2783 wait=156 +20:51:06 step 99640 loss=3.4587 lr=8.34e-05 tok=13.060B dt=2847 wait=156 +20:52:09 step 99660 loss=3.3468 lr=8.34e-05 tok=13.063B dt=2911 wait=157 +20:53:13 step 99680 loss=3.4507 lr=8.34e-05 tok=13.065B dt=2975 wait=157 +20:54:17 step 99700 loss=3.4309 lr=8.33e-05 tok=13.068B dt=3039 wait=157 +20:55:22 step 99720 loss=3.5605 lr=8.33e-05 tok=13.070B dt=3103 wait=158 +20:56:26 step 99740 loss=3.4760 lr=8.33e-05 tok=13.073B dt=3167 wait=158 +20:57:10 pushed ckpt_w66.pt,log_w66.txt +20:57:10 waited 0s for global r=288 +20:57:19 global r=288 load 4.293 -> 4.281 +20:57:53 step 99760 loss=3.6353 lr=8.33e-05 tok=13.076B dt=3255 wait=159 +20:58:57 step 99780 loss=3.5033 lr=8.33e-05 tok=13.078B dt=3319 wait=159 +21:00:01 step 99800 loss=3.4934 lr=8.32e-05 tok=13.081B dt=3383 wait=159 +21:01:05 step 99820 loss=3.5510 lr=8.32e-05 tok=13.084B dt=3447 wait=160 +21:02:09 step 99840 loss=3.4851 lr=8.32e-05 tok=13.086B dt=3511 wait=160 +21:03:13 step 99860 loss=3.5349 lr=8.32e-05 tok=13.089B dt=3575 wait=160 +21:04:17 step 99880 loss=3.4869 lr=8.32e-05 tok=13.091B dt=3639 wait=161 +21:05:21 step 99900 loss=3.5267 lr=8.31e-05 tok=13.094B dt=3703 wait=161 +21:05:35 pushed ckpt_w66.pt,log_w66.txt +21:05:35 waited 0s for global r=289 +21:05:46 global r=289 load 4.284 -> 4.284 +21:06:51 step 99920 loss=3.4801 lr=8.31e-05 tok=13.097B dt=3793 wait=161 +21:07:56 step 99940 loss=3.4974 lr=8.31e-05 tok=13.099B dt=3857 wait=162 +21:09:00 step 99960 loss=3.5052 lr=8.31e-05 tok=13.102B dt=3921 wait=162 +21:10:03 step 99980 loss=3.4595 lr=8.31e-05 tok=13.105B dt=3985 wait=162 +21:11:07 step 100000 loss=3.4899 lr=8.30e-05 tok=13.107B dt=4049 wait=163 +21:12:12 step 100020 loss=3.6119 lr=8.30e-05 tok=13.110B dt=4113 wait=163 +21:13:16 step 100040 loss=3.4395 lr=8.30e-05 tok=13.112B dt=4178 wait=163 +21:14:01 pushed ckpt_w66.pt,log_w66.txt +21:14:31 waited 30s for global r=290 +21:14:43 global r=290 load 4.278 -> 4.283 +21:15:16 step 100060 loss=3.5919 lr=8.30e-05 tok=13.115B dt=4298 wait=164 +21:16:20 step 100080 loss=3.4595 lr=8.29e-05 tok=13.118B dt=4362 wait=164 +21:17:25 step 100100 loss=3.4496 lr=8.29e-05 tok=13.120B dt=4426 wait=164 +21:18:29 step 100120 loss=3.4282 lr=8.29e-05 tok=13.123B dt=4491 wait=165 +21:19:32 step 100140 loss=3.4330 lr=8.29e-05 tok=13.126B dt=4554 wait=165 +21:20:36 step 100160 loss=3.5045 lr=8.29e-05 tok=13.128B dt=4618 wait=165 +21:21:39 step 100180 loss=3.4241 lr=8.28e-05 tok=13.131B dt=4681 wait=166 +21:22:43 step 100200 loss=3.4082 lr=8.28e-05 tok=13.133B dt=4745 wait=166 +21:22:57 pushed ckpt_w66.pt,log_w66.txt +launch +21:23:46 name=w66 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 13/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:23:51 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:23:51 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +21:24:01 params 1.08B +21:24:05 resumed step=100050 seen=13.11B round=290 +21:24:05 stream salt=100050 +21:25:58 blocks compiled +21:25:58 start: accum=4, 131072 tokens per optimiser step +21:28:15 step 100051 loss=3.3087 lr=8.30e-05 tok=13.114B dt=138 wait=121 +21:28:19 step 100052 loss=3.3484 lr=8.30e-05 tok=13.114B dt=141 wait=121 +21:28:46 step 100060 loss=3.5808 lr=8.30e-05 tok=13.115B dt=168 wait=121 +21:28:59 global r=291 load 4.270 -> 4.285 +21:30:08 step 100220 loss=3.4385 lr=8.28e-05 tok=13.118B dt=250 wait=122 +21:31:12 step 100240 loss=3.5094 lr=8.28e-05 tok=13.120B dt=314 wait=122 +21:32:16 step 100260 loss=3.5447 lr=8.28e-05 tok=13.123B dt=378 wait=122 +21:33:20 step 100280 loss=3.4892 lr=8.27e-05 tok=13.126B dt=443 wait=123 +21:34:25 step 100300 loss=3.4898 lr=8.27e-05 tok=13.128B dt=507 wait=123 +21:35:29 step 100320 loss=3.5759 lr=8.27e-05 tok=13.131B dt=571 wait=123 +21:36:33 step 100340 loss=3.4048 lr=8.27e-05 tok=13.133B dt=636 wait=124 +21:37:25 pushed ckpt_w66.pt,log_w66.txt +21:38:55 waited 91s for global r=292 +21:39:06 global r=292 load 4.262 -> 4.260 +21:39:41 step 100360 loss=3.5257 lr=8.26e-05 tok=13.136B dt=823 wait=124 +21:40:45 step 100380 loss=3.4225 lr=8.26e-05 tok=13.139B dt=888 wait=124 +21:41:50 step 100400 loss=3.3202 lr=8.26e-05 tok=13.141B dt=952 wait=125 +21:42:54 step 100420 loss=3.5788 lr=8.26e-05 tok=13.144B dt=1016 wait=125 +21:43:58 step 100440 loss=3.4099 lr=8.26e-05 tok=13.147B dt=1080 wait=125 +21:45:02 step 100460 loss=3.5303 lr=8.25e-05 tok=13.149B dt=1144 wait=126 +21:46:06 step 100480 loss=3.4739 lr=8.25e-05 tok=13.152B dt=1208 wait=126 +21:47:10 step 100500 loss=3.2274 lr=8.25e-05 tok=13.154B dt=1273 wait=126 +21:47:24 pushed ckpt_w66.pt,log_w66.txt +21:48:24 waited 60s for global r=293 +21:48:31 global r=293 load 4.286 -> 4.271 +21:49:36 step 100520 loss=3.4387 lr=8.25e-05 tok=13.157B dt=1419 wait=127 +21:50:41 step 100540 loss=3.5059 lr=8.25e-05 tok=13.160B dt=1483 wait=127 +21:51:45 step 100560 loss=3.4322 lr=8.24e-05 tok=13.162B dt=1547 wait=127 +21:52:49 step 100580 loss=3.5009 lr=8.24e-05 tok=13.165B dt=1611 wait=128 +21:53:53 step 100600 loss=3.4696 lr=8.24e-05 tok=13.167B dt=1675 wait=128 +21:54:57 step 100620 loss=3.4985 lr=8.24e-05 tok=13.170B dt=1739 wait=128 +21:56:01 step 100640 loss=3.3915 lr=8.24e-05 tok=13.173B dt=1803 wait=129 +21:56:45 pushed ckpt_w66.pt,log_w66.txt +21:57:46 waited 61s for global r=294 +21:57:56 global r=294 load 4.309 -> 4.288 +21:58:30 step 100660 loss=3.5353 lr=8.23e-05 tok=13.175B dt=1952 wait=129 +21:59:34 step 100680 loss=3.4023 lr=8.23e-05 tok=13.178B dt=2016 wait=130 +22:00:38 step 100700 loss=3.4485 lr=8.23e-05 tok=13.181B dt=2080 wait=130 +22:01:42 step 100720 loss=3.5610 lr=8.23e-05 tok=13.183B dt=2144 wait=130 +22:02:47 step 100740 loss=3.4557 lr=8.22e-05 tok=13.186B dt=2209 wait=131 +22:03:51 step 100760 loss=3.5568 lr=8.22e-05 tok=13.188B dt=2273 wait=131 +22:04:55 step 100780 loss=3.4131 lr=8.22e-05 tok=13.191B dt=2337 wait=131 +22:05:59 step 100800 loss=3.5112 lr=8.22e-05 tok=13.194B dt=2401 wait=131 +22:06:11 pushed ckpt_w66.pt,log_w66.txt +22:07:12 waited 61s for global r=295 +22:07:20 global r=295 load 4.299 -> 4.272 +22:08:26 step 100820 loss=3.4446 lr=8.22e-05 tok=13.196B dt=2548 wait=132 +22:09:31 step 100840 loss=3.4191 lr=8.21e-05 tok=13.199B dt=2613 wait=132 +22:10:35 step 100860 loss=3.5049 lr=8.21e-05 tok=13.202B dt=2677 wait=132 +22:11:39 step 100880 loss=3.4014 lr=8.21e-05 tok=13.204B dt=2741 wait=133 +22:12:43 step 100900 loss=3.5256 lr=8.21e-05 tok=13.207B dt=2806 wait=133 +22:13:48 step 100920 loss=3.5819 lr=8.21e-05 tok=13.209B dt=2870 wait=133 +22:14:52 step 100940 loss=3.3710 lr=8.20e-05 tok=13.212B dt=2934 wait=134 +22:15:37 pushed ckpt_w66.pt,log_w66.txt +22:16:38 waited 60s for global r=296 +22:16:45 global r=296 load 4.312 -> 4.302 +22:17:18 step 100960 loss=3.5235 lr=8.20e-05 tok=13.215B dt=3080 wait=134 +22:18:22 step 100980 loss=3.4560 lr=8.20e-05 tok=13.217B dt=3144 wait=134 +22:19:27 step 101000 loss=3.4616 lr=8.20e-05 tok=13.220B dt=3209 wait=135 +22:20:31 step 101020 loss=3.5651 lr=8.19e-05 tok=13.223B dt=3273 wait=135 +22:21:35 step 101040 loss=3.5099 lr=8.19e-05 tok=13.225B dt=3338 wait=135 +22:22:40 step 101060 loss=3.5783 lr=8.19e-05 tok=13.228B dt=3402 wait=136 +22:23:44 step 101080 loss=3.4713 lr=8.19e-05 tok=13.230B dt=3466 wait=136 +22:24:49 step 101100 loss=3.5596 lr=8.19e-05 tok=13.233B dt=3531 wait=136 +22:25:03 pushed ckpt_w66.pt,log_w66.txt +22:25:48 waited 45s for global r=297 +22:25:56 global r=297 load 4.278 -> 4.288 +22:27:01 step 101120 loss=3.4418 lr=8.18e-05 tok=13.236B dt=3664 wait=137 +22:28:06 step 101140 loss=3.4687 lr=8.18e-05 tok=13.238B dt=3728 wait=137 +22:29:10 step 101160 loss=3.5151 lr=8.18e-05 tok=13.241B dt=3792 wait=137 +22:30:15 step 101180 loss=3.4774 lr=8.18e-05 tok=13.244B dt=3857 wait=138 +22:31:19 step 101200 loss=3.4351 lr=8.18e-05 tok=13.246B dt=3922 wait=138 +22:32:23 step 101220 loss=3.5008 lr=8.17e-05 tok=13.249B dt=3986 wait=138 +22:33:27 step 101240 loss=3.4245 lr=8.17e-05 tok=13.251B dt=4050 wait=139 +22:34:13 pushed ckpt_w66.pt,log_w66.txt +22:37:48 step 101260 loss=3.5771 lr=8.17e-05 tok=13.254B dt=4310 wait=139 +22:37:54 global r=298 load 4.291 -> 4.283 +22:38:26 step 101260 loss=3.5113 lr=8.17e-05 tok=13.255B dt=4349 wait=139 +22:39:30 step 101280 loss=3.3688 lr=8.17e-05 tok=13.258B dt=4412 wait=139 +22:40:35 step 101300 loss=3.3381 lr=8.17e-05 tok=13.261B dt=4477 wait=140 +22:41:38 step 101320 loss=3.5237 lr=8.16e-05 tok=13.263B dt=4541 wait=140 +22:42:42 step 101340 loss=3.3569 lr=8.16e-05 tok=13.266B dt=4604 wait=140 +22:43:46 step 101360 loss=3.4822 lr=8.16e-05 tok=13.268B dt=4668 wait=141 +22:44:49 step 101380 loss=3.4103 lr=8.16e-05 tok=13.271B dt=4732 wait=141 +22:45:53 step 101400 loss=3.3983 lr=8.15e-05 tok=13.274B dt=4796 wait=141 +22:46:05 pushed ckpt_w66.pt,log_w66.txt +22:49:06 waited 181s for global r=299 +22:49:12 global r=299 load 4.284 -> 4.282 +22:50:17 step 101420 loss=3.3994 lr=8.15e-05 tok=13.276B dt=5059 wait=142 +22:51:21 step 101440 loss=3.4325 lr=8.15e-05 tok=13.279B dt=5123 wait=142 +22:52:24 step 101460 loss=3.4291 lr=8.15e-05 tok=13.282B dt=5186 wait=142 +22:53:27 step 101480 loss=3.3597 lr=8.15e-05 tok=13.284B dt=5250 wait=143 +22:54:32 step 101500 loss=3.4069 lr=8.14e-05 tok=13.287B dt=5314 wait=143 +22:55:35 step 101520 loss=3.4683 lr=8.14e-05 tok=13.289B dt=5378 wait=143 +22:56:39 step 101540 loss=3.3430 lr=8.14e-05 tok=13.292B dt=5441 wait=144 +22:57:24 pushed ckpt_w66.pt,log_w66.txt +23:00:58 step 101560 loss=3.4923 lr=8.14e-05 tok=13.295B dt=5701 wait=144 +23:01:04 global r=300 load 4.277 -> 4.287 +23:01:35 step 101560 loss=3.4571 lr=8.14e-05 tok=13.296B dt=5738 wait=144 +23:02:38 step 101580 loss=3.2974 lr=8.14e-05 tok=13.299B dt=5801 wait=145 +23:03:42 step 101600 loss=3.2883 lr=8.13e-05 tok=13.301B dt=5864 wait=145 +23:04:45 step 101620 loss=3.4638 lr=8.13e-05 tok=13.304B dt=5928 wait=145 +23:05:49 step 101640 loss=3.3682 lr=8.13e-05 tok=13.306B dt=5991 wait=146 +23:06:53 step 101660 loss=3.4905 lr=8.13e-05 tok=13.309B dt=6055 wait=146 +23:07:56 step 101680 loss=3.3860 lr=8.12e-05 tok=13.312B dt=6118 wait=146 +23:09:00 step 101700 loss=3.4688 lr=8.12e-05 tok=13.314B dt=6182 wait=147 diff --git a/log_w67.txt b/log_w67.txt new file mode 100644 index 0000000000000000000000000000000000000000..c72fe56e232a79f9c3afcf6a7288e13968804164 --- /dev/null +++ b/log_w67.txt @@ -0,0 +1,18 @@ +launch +18:01:11 name=w67 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 14/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:01:18 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:01:18 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:01:27 params 1.08B +18:01:29 resumed step=97800 seen=12.82B round=280 +18:01:29 stream salt=97800 +18:03:38 blocks compiled +18:03:38 start: accum=4, 131072 tokens per optimiser step +18:06:26 step 97801 loss=3.2280 lr=8.54e-05 tok=12.819B dt=168 wait=130 +18:06:29 step 97802 loss=3.2180 lr=8.54e-05 tok=12.819B dt=171 wait=130 +18:07:30 step 97820 loss=3.5582 lr=8.53e-05 tok=12.821B dt=232 wait=131 +18:08:39 step 97840 loss=3.5236 lr=8.53e-05 tok=12.824B dt=301 wait=131 +18:09:43 step 97860 loss=3.5628 lr=8.53e-05 tok=12.827B dt=364 wait=131 +18:10:46 step 97880 loss=3.5184 lr=8.53e-05 tok=12.829B dt=428 wait=132 +18:11:50 step 97900 loss=3.6526 lr=8.53e-05 tok=12.832B dt=491 wait=132 +18:12:54 step 97920 loss=3.5631 lr=8.52e-05 tok=12.835B dt=555 wait=133 +18:13:57 step 97940 loss=3.5272 lr=8.52e-05 tok=12.837B dt=618 wait=133 diff --git a/log_w68.txt b/log_w68.txt new file mode 100644 index 0000000000000000000000000000000000000000..72edf44df7824b5d583fb5d0ecd2c03520ddcae6 --- /dev/null +++ b/log_w68.txt @@ -0,0 +1,19 @@ +launch +18:34:44 name=w68 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 15/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:34:51 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:34:51 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:35:00 params 1.08B +18:35:02 resumed step=97950 seen=12.84B round=281 +18:35:02 stream salt=97950 +18:37:18 blocks compiled +18:37:18 start: accum=4, 131072 tokens per optimiser step +18:40:40 step 97951 loss=3.2022 lr=8.52e-05 tok=12.839B dt=202 wait=165 +18:40:44 step 97952 loss=3.2581 lr=8.52e-05 tok=12.839B dt=205 wait=165 +18:41:11 step 97960 loss=3.5857 lr=8.52e-05 tok=12.840B dt=233 wait=165 +18:42:20 step 97980 loss=3.4717 lr=8.52e-05 tok=12.842B dt=301 wait=166 +18:43:27 step 98000 loss=3.4916 lr=8.52e-05 tok=12.845B dt=368 wait=166 +18:44:30 step 98020 loss=3.4344 lr=8.51e-05 tok=12.848B dt=432 wait=166 +18:45:34 step 98040 loss=3.5171 lr=8.51e-05 tok=12.850B dt=495 wait=167 +18:46:37 step 98060 loss=3.6764 lr=8.51e-05 tok=12.853B dt=559 wait=167 +18:47:41 step 98080 loss=3.4808 lr=8.51e-05 tok=12.856B dt=623 wait=167 +18:48:44 step 98100 loss=3.6047 lr=8.50e-05 tok=12.858B dt=686 wait=168 diff --git a/log_w7.txt b/log_w7.txt new file mode 100644 index 0000000000000000000000000000000000000000..4ed48c457071ad127f4dc2ec6c096a287afc4611 --- /dev/null +++ b/log_w7.txt @@ -0,0 +1,200 @@ +launch +21:01:04 name=w7 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 2/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +21:01:10 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +21:01:20 params 1.08B +21:01:21 resumed step=30600 seen=4.01B round=15 +21:01:21 stream salt=30600 +21:03:29 start: accum=4, 131072 tokens per optimiser step +21:06:13 step 30620 loss=3.7365 lr=2.85e-04 tok=4.013B dt=165 +21:07:43 step 30640 loss=3.8909 lr=2.85e-04 tok=4.016B dt=254 +21:08:58 step 30660 loss=3.8503 lr=2.85e-04 tok=4.019B dt=329 +21:10:14 step 30680 loss=3.8732 lr=2.85e-04 tok=4.021B dt=405 +21:11:29 step 30700 loss=3.7611 lr=2.85e-04 tok=4.024B dt=481 +21:12:45 step 30720 loss=3.7345 lr=2.85e-04 tok=4.027B dt=556 +21:14:01 step 30740 loss=3.7792 lr=2.85e-04 tok=4.029B dt=632 +21:15:16 step 30760 loss=3.7897 lr=2.85e-04 tok=4.032B dt=708 +21:16:32 step 30780 loss=3.7340 lr=2.85e-04 tok=4.034B dt=783 +21:17:47 step 30800 loss=3.8161 lr=2.85e-04 tok=4.037B dt=859 +21:19:03 step 30820 loss=3.7539 lr=2.85e-04 tok=4.040B dt=935 +21:20:19 step 30840 loss=3.7262 lr=2.85e-04 tok=4.042B dt=1010 +21:21:34 step 30860 loss=3.7639 lr=2.85e-04 tok=4.045B dt=1086 +21:22:50 step 30880 loss=3.7249 lr=2.85e-04 tok=4.048B dt=1161 +21:24:05 step 30900 loss=3.8004 lr=2.85e-04 tok=4.050B dt=1237 +21:25:21 step 30920 loss=3.8864 lr=2.85e-04 tok=4.053B dt=1312 +21:26:37 step 30940 loss=3.8405 lr=2.85e-04 tok=4.055B dt=1388 +21:27:52 step 30960 loss=3.8593 lr=2.85e-04 tok=4.058B dt=1464 +21:29:08 step 30980 loss=3.8803 lr=2.85e-04 tok=4.061B dt=1539 +21:30:24 step 31000 loss=3.8358 lr=2.85e-04 tok=4.063B dt=1615 +21:31:39 step 31020 loss=3.7647 lr=2.85e-04 tok=4.066B dt=1691 +21:32:55 step 31040 loss=3.6669 lr=2.85e-04 tok=4.068B dt=1766 +21:34:11 step 31060 loss=3.7366 lr=2.85e-04 tok=4.071B dt=1842 +21:35:26 step 31080 loss=3.6825 lr=2.85e-04 tok=4.074B dt=1917 +21:36:41 step 31100 loss=3.8089 lr=2.85e-04 tok=4.076B dt=1993 +21:37:57 step 31120 loss=3.7374 lr=2.85e-04 tok=4.079B dt=2068 +21:39:12 step 31140 loss=3.7151 lr=2.85e-04 tok=4.082B dt=2144 +21:40:28 step 31160 loss=3.8094 lr=2.85e-04 tok=4.084B dt=2219 +21:41:43 step 31180 loss=3.7239 lr=2.85e-04 tok=4.087B dt=2295 +21:42:59 step 31200 loss=3.8090 lr=2.85e-04 tok=4.089B dt=2371 +21:43:12 pushed ckpt_w7.pt +21:43:13 pushed log_w7.txt +21:44:39 step 31220 loss=3.8245 lr=2.85e-04 tok=4.092B dt=2470 +21:45:54 step 31240 loss=3.7901 lr=2.85e-04 tok=4.095B dt=2545 +21:47:10 step 31260 loss=3.6824 lr=2.85e-04 tok=4.097B dt=2621 +21:48:25 step 31280 loss=3.6857 lr=2.85e-04 tok=4.100B dt=2696 +21:49:41 step 31300 loss=3.7382 lr=2.85e-04 tok=4.103B dt=2772 +21:50:56 step 31320 loss=3.8482 lr=2.84e-04 tok=4.105B dt=2848 +21:52:12 step 31340 loss=3.9178 lr=2.84e-04 tok=4.108B dt=2923 +21:53:28 step 31360 loss=3.8447 lr=2.84e-04 tok=4.110B dt=2999 +21:54:43 step 31380 loss=3.7578 lr=2.84e-04 tok=4.113B dt=3075 +21:55:59 step 31400 loss=3.7527 lr=2.84e-04 tok=4.116B dt=3150 +21:57:15 step 31420 loss=3.8120 lr=2.84e-04 tok=4.118B dt=3226 +21:58:30 step 31440 loss=3.7822 lr=2.84e-04 tok=4.121B dt=3302 +21:59:46 step 31460 loss=3.7536 lr=2.84e-04 tok=4.124B dt=3377 +22:01:02 step 31480 loss=3.8178 lr=2.84e-04 tok=4.126B dt=3453 +22:02:17 step 31500 loss=3.7094 lr=2.84e-04 tok=4.129B dt=3529 +22:03:33 step 31520 loss=3.8160 lr=2.84e-04 tok=4.131B dt=3604 +22:04:48 step 31540 loss=3.7548 lr=2.84e-04 tok=4.134B dt=3680 +22:06:04 step 31560 loss=3.7875 lr=2.84e-04 tok=4.137B dt=3755 +22:07:20 step 31580 loss=3.9070 lr=2.84e-04 tok=4.139B dt=3831 +22:08:35 step 31600 loss=3.7900 lr=2.84e-04 tok=4.142B dt=3907 +22:09:51 step 31620 loss=3.7717 lr=2.84e-04 tok=4.144B dt=3982 +22:11:07 step 31640 loss=3.8544 lr=2.84e-04 tok=4.147B dt=4058 +22:12:22 step 31660 loss=3.7989 lr=2.84e-04 tok=4.150B dt=4134 +22:13:38 step 31680 loss=3.7388 lr=2.84e-04 tok=4.152B dt=4209 +22:14:53 step 31700 loss=3.7483 lr=2.84e-04 tok=4.155B dt=4285 +22:16:09 step 31720 loss=3.7044 lr=2.84e-04 tok=4.158B dt=4360 +22:17:24 step 31740 loss=3.8208 lr=2.84e-04 tok=4.160B dt=4436 +22:18:40 step 31760 loss=3.7577 lr=2.84e-04 tok=4.163B dt=4511 +22:19:55 step 31780 loss=3.7531 lr=2.84e-04 tok=4.165B dt=4587 +22:21:11 step 31800 loss=3.8712 lr=2.84e-04 tok=4.168B dt=4662 +22:21:23 pushed ckpt_w7.pt +22:21:23 pushed log_w7.txt +22:21:38 pulled global r=16 4.315 -> 4.154 +22:23:01 step 31820 loss=3.6537 lr=2.84e-04 tok=4.171B dt=4772 +22:24:16 step 31840 loss=3.6131 lr=2.84e-04 tok=4.173B dt=4848 +22:25:32 step 31860 loss=3.5986 lr=2.84e-04 tok=4.176B dt=4923 +22:26:47 step 31880 loss=3.6716 lr=2.84e-04 tok=4.179B dt=4999 +22:28:03 step 31900 loss=3.7590 lr=2.84e-04 tok=4.181B dt=5075 +22:29:19 step 31920 loss=3.7459 lr=2.84e-04 tok=4.184B dt=5150 +22:30:34 step 31940 loss=3.6862 lr=2.84e-04 tok=4.186B dt=5226 +22:31:50 step 31960 loss=3.7924 lr=2.84e-04 tok=4.189B dt=5301 +22:33:05 step 31980 loss=3.8792 lr=2.84e-04 tok=4.192B dt=5377 +22:34:21 step 32000 loss=3.8701 lr=2.84e-04 tok=4.194B dt=5452 +22:35:37 step 32020 loss=3.8034 lr=2.84e-04 tok=4.197B dt=5528 +22:36:52 step 32040 loss=3.8362 lr=2.84e-04 tok=4.200B dt=5604 +22:38:08 step 32060 loss=3.7098 lr=2.84e-04 tok=4.202B dt=5680 +22:39:24 step 32080 loss=3.7803 lr=2.84e-04 tok=4.205B dt=5755 +22:40:39 step 32100 loss=3.7538 lr=2.84e-04 tok=4.207B dt=5831 +22:41:55 step 32120 loss=3.6604 lr=2.84e-04 tok=4.210B dt=5906 +22:43:10 step 32140 loss=3.6835 lr=2.84e-04 tok=4.213B dt=5982 +22:44:26 step 32160 loss=3.7728 lr=2.84e-04 tok=4.215B dt=6057 +22:45:41 step 32180 loss=3.6844 lr=2.84e-04 tok=4.218B dt=6133 +22:46:57 step 32200 loss=3.7350 lr=2.84e-04 tok=4.221B dt=6208 +22:48:13 step 32220 loss=3.6953 lr=2.84e-04 tok=4.223B dt=6284 +22:49:28 step 32240 loss=3.8418 lr=2.84e-04 tok=4.226B dt=6360 +22:50:44 step 32260 loss=3.8185 lr=2.84e-04 tok=4.228B dt=6435 +22:51:59 step 32280 loss=3.7562 lr=2.84e-04 tok=4.231B dt=6511 +22:53:15 step 32300 loss=3.6722 lr=2.84e-04 tok=4.234B dt=6586 +22:54:30 step 32320 loss=3.7124 lr=2.83e-04 tok=4.236B dt=6662 +22:55:46 step 32340 loss=3.6443 lr=2.83e-04 tok=4.239B dt=6737 +22:57:01 step 32360 loss=3.7122 lr=2.83e-04 tok=4.241B dt=6813 +22:58:17 step 32380 loss=3.7856 lr=2.83e-04 tok=4.244B dt=6888 +22:59:32 step 32400 loss=3.7823 lr=2.83e-04 tok=4.247B dt=6964 +22:59:46 pushed ckpt_w7.pt +22:59:46 pushed log_w7.txt +23:01:09 step 32420 loss=3.7615 lr=2.83e-04 tok=4.249B dt=7060 +23:02:25 step 32440 loss=3.7314 lr=2.83e-04 tok=4.252B dt=7136 +23:03:40 step 32460 loss=3.8342 lr=2.83e-04 tok=4.255B dt=7212 +23:04:56 step 32480 loss=3.7468 lr=2.83e-04 tok=4.257B dt=7287 +23:06:11 step 32500 loss=3.7251 lr=2.83e-04 tok=4.260B dt=7363 +23:07:27 step 32520 loss=3.8379 lr=2.83e-04 tok=4.262B dt=7438 +23:08:43 step 32540 loss=3.6880 lr=2.83e-04 tok=4.265B dt=7514 +23:09:58 step 32560 loss=3.7608 lr=2.83e-04 tok=4.268B dt=7590 +23:11:13 step 32580 loss=3.6996 lr=2.83e-04 tok=4.270B dt=7665 +23:12:29 step 32600 loss=3.7359 lr=2.83e-04 tok=4.273B dt=7740 +23:13:45 step 32620 loss=3.6941 lr=2.83e-04 tok=4.276B dt=7816 +23:15:00 step 32640 loss=3.5835 lr=2.83e-04 tok=4.278B dt=7892 +23:16:16 step 32660 loss=3.7407 lr=2.83e-04 tok=4.281B dt=7967 +23:17:31 step 32680 loss=3.7422 lr=2.83e-04 tok=4.283B dt=8043 +23:18:47 step 32700 loss=3.7367 lr=2.83e-04 tok=4.286B dt=8118 +23:20:02 step 32720 loss=3.7782 lr=2.83e-04 tok=4.289B dt=8194 +23:21:18 step 32740 loss=3.6957 lr=2.83e-04 tok=4.291B dt=8269 +23:22:34 step 32760 loss=3.6982 lr=2.83e-04 tok=4.294B dt=8345 +23:23:49 step 32780 loss=3.8707 lr=2.83e-04 tok=4.297B dt=8421 +23:25:05 step 32800 loss=3.7813 lr=2.83e-04 tok=4.299B dt=8496 +23:26:20 step 32820 loss=3.7477 lr=2.83e-04 tok=4.302B dt=8572 +23:27:36 step 32840 loss=3.7265 lr=2.83e-04 tok=4.304B dt=8647 +23:28:51 step 32860 loss=3.7593 lr=2.83e-04 tok=4.307B dt=8723 +23:30:07 step 32880 loss=3.7269 lr=2.83e-04 tok=4.310B dt=8798 +23:31:23 step 32900 loss=3.7517 lr=2.83e-04 tok=4.312B dt=8874 +23:32:38 step 32920 loss=3.7413 lr=2.83e-04 tok=4.315B dt=8950 +23:33:54 step 32940 loss=3.6969 lr=2.83e-04 tok=4.318B dt=9025 +23:35:09 step 32960 loss=3.6957 lr=2.83e-04 tok=4.320B dt=9101 +23:36:25 step 32980 loss=3.7814 lr=2.83e-04 tok=4.323B dt=9176 +23:37:40 step 33000 loss=3.7618 lr=2.83e-04 tok=4.325B dt=9252 +23:37:52 pushed ckpt_w7.pt +23:37:53 pushed log_w7.txt +23:39:14 step 33020 loss=3.8296 lr=2.83e-04 tok=4.328B dt=9346 +23:40:30 step 33040 loss=3.7049 lr=2.83e-04 tok=4.331B dt=9421 +23:41:45 step 33060 loss=3.6975 lr=2.83e-04 tok=4.333B dt=9497 +23:43:01 step 33080 loss=3.8117 lr=2.83e-04 tok=4.336B dt=9573 +23:44:17 step 33100 loss=3.7225 lr=2.83e-04 tok=4.338B dt=9648 +23:45:32 step 33120 loss=3.7274 lr=2.83e-04 tok=4.341B dt=9724 +23:46:48 step 33140 loss=3.6287 lr=2.83e-04 tok=4.344B dt=9799 +23:48:04 step 33160 loss=3.7639 lr=2.83e-04 tok=4.346B dt=9875 +23:49:19 step 33180 loss=3.8167 lr=2.83e-04 tok=4.349B dt=9951 +23:50:35 step 33200 loss=3.7180 lr=2.83e-04 tok=4.352B dt=10026 +23:51:50 step 33220 loss=3.7435 lr=2.83e-04 tok=4.354B dt=10102 +23:53:06 step 33240 loss=3.6332 lr=2.83e-04 tok=4.357B dt=10177 +23:54:22 step 33260 loss=3.6698 lr=2.83e-04 tok=4.359B dt=10253 +23:55:37 step 33280 loss=3.6573 lr=2.82e-04 tok=4.362B dt=10328 +23:56:53 step 33300 loss=3.7239 lr=2.82e-04 tok=4.365B dt=10404 +23:58:08 step 33320 loss=3.6816 lr=2.82e-04 tok=4.367B dt=10480 +23:59:24 step 33340 loss=3.6797 lr=2.82e-04 tok=4.370B dt=10555 +00:00:39 step 33360 loss=3.7830 lr=2.82e-04 tok=4.373B dt=10631 +00:01:55 step 33380 loss=3.8047 lr=2.82e-04 tok=4.375B dt=10707 +00:03:11 step 33400 loss=3.8166 lr=2.82e-04 tok=4.378B dt=10782 +00:04:26 step 33420 loss=3.6590 lr=2.82e-04 tok=4.380B dt=10858 +00:05:42 step 33440 loss=3.7624 lr=2.82e-04 tok=4.383B dt=10933 +00:06:58 step 33460 loss=3.7182 lr=2.82e-04 tok=4.386B dt=11009 +00:08:13 step 33480 loss=3.7976 lr=2.82e-04 tok=4.388B dt=11085 +00:09:29 step 33500 loss=3.7657 lr=2.82e-04 tok=4.391B dt=11160 +00:10:44 step 33520 loss=3.7743 lr=2.82e-04 tok=4.394B dt=11236 +00:12:00 step 33540 loss=3.8886 lr=2.82e-04 tok=4.396B dt=11311 +00:13:16 step 33560 loss=3.7030 lr=2.82e-04 tok=4.399B dt=11387 +00:14:31 step 33580 loss=3.7467 lr=2.82e-04 tok=4.401B dt=11463 +00:15:47 step 33600 loss=3.7058 lr=2.82e-04 tok=4.404B dt=11538 +00:16:03 pushed ckpt_w7.pt +00:16:04 pushed log_w7.txt +00:16:10 pulled global r=17 4.260 -> 4.127 +00:17:32 step 33620 loss=3.6107 lr=2.82e-04 tok=4.407B dt=11644 +00:18:48 step 33640 loss=3.5589 lr=2.82e-04 tok=4.409B dt=11719 +00:20:03 step 33660 loss=3.6279 lr=2.82e-04 tok=4.412B dt=11795 +00:21:19 step 33680 loss=3.6721 lr=2.82e-04 tok=4.415B dt=11870 +00:22:35 step 33700 loss=3.7926 lr=2.82e-04 tok=4.417B dt=11946 +00:23:50 step 33720 loss=3.7215 lr=2.82e-04 tok=4.420B dt=12022 +00:25:06 step 33740 loss=3.6585 lr=2.82e-04 tok=4.422B dt=12097 +00:26:21 step 33760 loss=3.7469 lr=2.82e-04 tok=4.425B dt=12173 +00:27:37 step 33780 loss=3.8314 lr=2.82e-04 tok=4.428B dt=12249 +00:28:53 step 33800 loss=3.7832 lr=2.82e-04 tok=4.430B dt=12324 +00:30:08 step 33820 loss=3.7330 lr=2.82e-04 tok=4.433B dt=12400 +00:31:24 step 33840 loss=3.7793 lr=2.82e-04 tok=4.435B dt=12475 +00:32:40 step 33860 loss=3.6867 lr=2.82e-04 tok=4.438B dt=12551 +00:33:55 step 33880 loss=3.7445 lr=2.82e-04 tok=4.441B dt=12627 +00:35:11 step 33900 loss=3.6820 lr=2.82e-04 tok=4.443B dt=12702 +00:36:26 step 33920 loss=3.6266 lr=2.82e-04 tok=4.446B dt=12778 +00:37:42 step 33940 loss=3.6661 lr=2.82e-04 tok=4.449B dt=12853 +00:38:58 step 33960 loss=3.7657 lr=2.82e-04 tok=4.451B dt=12929 +00:40:13 step 33980 loss=3.6443 lr=2.82e-04 tok=4.454B dt=13005 +00:41:29 step 34000 loss=3.7055 lr=2.82e-04 tok=4.456B dt=13080 +00:42:44 step 34020 loss=3.6455 lr=2.82e-04 tok=4.459B dt=13156 +00:44:00 step 34040 loss=3.7783 lr=2.82e-04 tok=4.462B dt=13231 +00:45:16 step 34060 loss=3.7734 lr=2.82e-04 tok=4.464B dt=13307 +00:46:31 step 34080 loss=3.7079 lr=2.82e-04 tok=4.467B dt=13382 +00:47:46 step 34100 loss=3.6552 lr=2.82e-04 tok=4.470B dt=13458 +00:49:02 step 34120 loss=3.7215 lr=2.82e-04 tok=4.472B dt=13533 +00:50:18 step 34140 loss=3.6520 lr=2.82e-04 tok=4.475B dt=13609 +00:51:33 step 34160 loss=3.7336 lr=2.82e-04 tok=4.477B dt=13685 +00:52:49 step 34180 loss=3.7663 lr=2.82e-04 tok=4.480B dt=13760 +00:54:04 step 34200 loss=3.6940 lr=2.82e-04 tok=4.483B dt=13836 +00:54:16 pushed ckpt_w7.pt diff --git a/log_w71.txt b/log_w71.txt new file mode 100644 index 0000000000000000000000000000000000000000..5f6bd5c63b006762a7d0fda8b09ec628fdbfb73d --- /dev/null +++ b/log_w71.txt @@ -0,0 +1,29 @@ +launch +20:26:07 name=w71 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 18/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +20:26:13 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +20:26:13 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +20:26:23 params 1.08B +20:26:24 resumed step=99450 seen=13.04B round=286 +20:26:24 stream salt=99450 +20:28:29 blocks compiled +20:28:29 start: accum=4, 131072 tokens per optimiser step +20:31:38 step 99451 loss=3.0424 lr=8.36e-05 tok=13.035B dt=189 wait=149 +20:31:41 step 99452 loss=3.1611 lr=8.36e-05 tok=13.035B dt=192 wait=149 +20:32:08 step 99460 loss=3.5314 lr=8.36e-05 tok=13.036B dt=219 wait=149 +20:33:16 step 99480 loss=3.4171 lr=8.36e-05 tok=13.039B dt=287 wait=150 +20:34:19 step 99500 loss=3.4658 lr=8.36e-05 tok=13.042B dt=351 wait=150 +20:35:23 step 99520 loss=3.4673 lr=8.35e-05 tok=13.044B dt=414 wait=150 +20:36:26 step 99540 loss=3.4839 lr=8.35e-05 tok=13.047B dt=478 wait=151 +20:37:30 step 99560 loss=3.6544 lr=8.35e-05 tok=13.050B dt=541 wait=151 +20:38:33 step 99580 loss=3.4797 lr=8.35e-05 tok=13.052B dt=604 wait=151 +20:39:36 step 99600 loss=3.5020 lr=8.35e-05 tok=13.055B dt=668 wait=151 +20:39:51 pushed ckpt_w71.pt,log_w71.txt +20:39:51 waited 0s for global r=287 +20:40:08 global r=287 load 4.294 -> 4.274 +20:41:14 step 99620 loss=3.4405 lr=8.34e-05 tok=13.057B dt=766 wait=152 +20:42:18 step 99640 loss=3.4431 lr=8.34e-05 tok=13.060B dt=829 wait=152 +20:43:36 step 99660 loss=3.3774 lr=8.34e-05 tok=13.063B dt=908 wait=153 +20:44:39 step 99680 loss=3.3669 lr=8.34e-05 tok=13.065B dt=971 wait=153 +20:45:42 step 99700 loss=3.3755 lr=8.33e-05 tok=13.068B dt=1033 wait=153 +20:46:45 step 99720 loss=3.5175 lr=8.33e-05 tok=13.070B dt=1097 wait=153 +20:47:49 step 99740 loss=3.3950 lr=8.33e-05 tok=13.073B dt=1160 wait=154 diff --git a/log_w72.txt b/log_w72.txt new file mode 100644 index 0000000000000000000000000000000000000000..efa772e1a197e33c37fb0c58bbdcadc071cd9ecc --- /dev/null +++ b/log_w72.txt @@ -0,0 +1,19 @@ +launch +06:16:10 name=w72 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 19/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:16:16 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:16:16 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:16:26 params 1.08B +06:16:27 resumed step=103050 seen=13.51B round=302 +06:16:27 stream salt=103050 +06:18:12 blocks compiled +06:18:12 start: accum=4, 131072 tokens per optimiser step +06:20:34 step 103051 loss=3.1104 lr=7.98e-05 tok=13.511B dt=141 wait=96 +06:20:37 step 103052 loss=3.3712 lr=7.98e-05 tok=13.511B dt=145 wait=96 +06:21:06 step 103060 loss=3.6633 lr=7.98e-05 tok=13.512B dt=173 wait=96 +06:22:18 step 103080 loss=3.5071 lr=7.98e-05 tok=13.515B dt=245 wait=97 +06:23:23 step 103100 loss=3.5291 lr=7.97e-05 tok=13.517B dt=310 wait=97 +06:24:27 step 103120 loss=3.5174 lr=7.97e-05 tok=13.520B dt=375 wait=97 +06:25:32 step 103140 loss=3.5314 lr=7.97e-05 tok=13.523B dt=439 wait=98 +06:26:36 step 103160 loss=3.6405 lr=7.97e-05 tok=13.525B dt=504 wait=98 +06:27:41 step 103180 loss=3.5411 lr=7.97e-05 tok=13.528B dt=568 wait=98 +06:28:45 step 103200 loss=3.5286 lr=7.96e-05 tok=13.531B dt=633 wait=99 diff --git a/log_w74.txt b/log_w74.txt new file mode 100644 index 0000000000000000000000000000000000000000..b528784ef3bba53a6838d5b88080ae46a90f0101 --- /dev/null +++ b/log_w74.txt @@ -0,0 +1,190 @@ +launch +06:16:12 name=w74 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 21/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:16:18 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:16:18 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:16:27 params 1.08B +06:16:29 resumed step=103050 seen=13.51B round=302 +06:16:29 stream salt=103050 +06:18:16 blocks compiled +06:18:16 start: accum=4, 131072 tokens per optimiser step +06:21:07 step 103051 loss=3.0937 lr=7.98e-05 tok=13.511B dt=171 wait=129 +06:21:10 step 103052 loss=3.1373 lr=7.98e-05 tok=13.511B dt=175 wait=129 +06:21:39 step 103060 loss=3.5437 lr=7.98e-05 tok=13.512B dt=203 wait=129 +06:22:50 step 103080 loss=3.4422 lr=7.98e-05 tok=13.515B dt=274 wait=130 +06:23:53 step 103100 loss=3.5001 lr=7.97e-05 tok=13.517B dt=337 wait=130 +06:24:57 step 103120 loss=3.3988 lr=7.97e-05 tok=13.520B dt=401 wait=131 +06:26:01 step 103140 loss=3.4934 lr=7.97e-05 tok=13.523B dt=465 wait=131 +06:27:04 step 103160 loss=3.6592 lr=7.97e-05 tok=13.525B dt=529 wait=131 +06:28:08 step 103180 loss=3.4629 lr=7.97e-05 tok=13.528B dt=593 wait=132 +06:29:12 step 103200 loss=3.4818 lr=7.96e-05 tok=13.531B dt=657 wait=132 +06:29:27 pushed ckpt_w74.pt,log_w74.txt +06:33:36 step 103220 loss=3.4993 lr=7.96e-05 tok=13.533B dt=920 wait=132 +06:34:39 step 103240 loss=3.4526 lr=7.96e-05 tok=13.536B dt=984 wait=133 +06:35:43 step 103260 loss=3.4688 lr=7.96e-05 tok=13.538B dt=1047 wait=133 +06:36:46 step 103280 loss=3.4430 lr=7.96e-05 tok=13.541B dt=1110 wait=133 +06:37:50 step 103300 loss=3.4335 lr=7.95e-05 tok=13.544B dt=1174 wait=134 +06:38:53 step 103320 loss=3.4863 lr=7.95e-05 tok=13.546B dt=1238 wait=134 +06:39:57 step 103340 loss=3.4543 lr=7.95e-05 tok=13.549B dt=1301 wait=134 +06:40:42 pushed ckpt_w74.pt,log_w74.txt +06:44:19 step 103360 loss=3.3469 lr=7.95e-05 tok=13.552B dt=1563 wait=135 +06:45:23 step 103380 loss=3.2757 lr=7.94e-05 tok=13.554B dt=1627 wait=135 +06:46:27 step 103400 loss=3.5286 lr=7.94e-05 tok=13.557B dt=1691 wait=135 +06:47:30 step 103420 loss=3.4760 lr=7.94e-05 tok=13.559B dt=1754 wait=136 +06:48:34 step 103440 loss=3.4464 lr=7.94e-05 tok=13.562B dt=1818 wait=136 +06:49:38 step 103460 loss=3.3854 lr=7.94e-05 tok=13.565B dt=1882 wait=136 +06:50:24 global r=303 load 4.309 -> 4.274 +06:50:56 step 103360 loss=3.4673 lr=7.95e-05 tok=13.567B dt=1960 wait=137 +06:51:59 step 103380 loss=3.3632 lr=7.94e-05 tok=13.570B dt=2023 wait=137 +06:53:03 step 103400 loss=3.3977 lr=7.94e-05 tok=13.573B dt=2087 wait=137 +06:54:06 step 103420 loss=3.4866 lr=7.94e-05 tok=13.575B dt=2151 wait=138 +06:55:11 step 103440 loss=3.3522 lr=7.94e-05 tok=13.578B dt=2215 wait=138 +06:56:14 step 103460 loss=3.4229 lr=7.94e-05 tok=13.580B dt=2278 wait=138 +06:57:18 step 103480 loss=3.4172 lr=7.93e-05 tok=13.583B dt=2342 wait=139 +06:58:22 step 103500 loss=3.4070 lr=7.93e-05 tok=13.586B dt=2406 wait=139 +06:58:34 pushed ckpt_w74.pt,log_w74.txt +06:59:50 waited 76s for global r=304 +07:00:00 global r=304 load 4.288 -> 4.273 +07:01:06 step 103520 loss=3.4128 lr=7.93e-05 tok=13.588B dt=2570 wait=139 +07:02:09 step 103540 loss=3.4386 lr=7.93e-05 tok=13.591B dt=2633 wait=140 +07:03:13 step 103560 loss=3.4222 lr=7.93e-05 tok=13.593B dt=2697 wait=140 +07:04:16 step 103580 loss=3.3735 lr=7.92e-05 tok=13.596B dt=2761 wait=140 +07:05:20 step 103600 loss=3.4320 lr=7.92e-05 tok=13.599B dt=2824 wait=140 +07:06:24 step 103620 loss=3.5507 lr=7.92e-05 tok=13.601B dt=2888 wait=141 +07:07:28 step 103640 loss=3.2816 lr=7.92e-05 tok=13.604B dt=2953 wait=141 +07:08:14 pushed ckpt_w74.pt,log_w74.txt +07:09:00 waited 45s for global r=305 +07:09:06 global r=305 load 4.329 -> 4.288 +07:09:39 step 103660 loss=3.4244 lr=7.92e-05 tok=13.607B dt=3083 wait=141 +07:10:43 step 103680 loss=3.4041 lr=7.91e-05 tok=13.609B dt=3147 wait=142 +07:11:46 step 103700 loss=3.3743 lr=7.91e-05 tok=13.612B dt=3210 wait=142 +07:12:50 step 103720 loss=3.4270 lr=7.91e-05 tok=13.614B dt=3274 wait=143 +07:13:54 step 103740 loss=3.3500 lr=7.91e-05 tok=13.617B dt=3338 wait=143 +07:14:58 step 103760 loss=3.4525 lr=7.90e-05 tok=13.620B dt=3402 wait=143 +07:16:01 step 103780 loss=3.4207 lr=7.90e-05 tok=13.622B dt=3466 wait=144 +07:17:05 step 103800 loss=3.4081 lr=7.90e-05 tok=13.625B dt=3529 wait=144 +07:17:18 pushed ckpt_w74.pt,log_w74.txt +07:18:19 waited 60s for global r=306 +07:18:25 global r=306 load 4.292 -> 4.286 +07:19:30 step 103820 loss=3.4362 lr=7.90e-05 tok=13.628B dt=3674 wait=144 +07:20:33 step 103840 loss=3.4533 lr=7.90e-05 tok=13.630B dt=3737 wait=145 +07:21:37 step 103860 loss=3.4252 lr=7.89e-05 tok=13.633B dt=3801 wait=145 +07:22:40 step 103880 loss=3.4048 lr=7.89e-05 tok=13.635B dt=3864 wait=145 +07:23:44 step 103900 loss=3.4962 lr=7.89e-05 tok=13.638B dt=3928 wait=146 +07:24:48 step 103920 loss=3.4662 lr=7.89e-05 tok=13.641B dt=3992 wait=146 +07:25:51 step 103940 loss=3.3569 lr=7.89e-05 tok=13.643B dt=4055 wait=146 +07:26:37 pushed ckpt_w74.pt,log_w74.txt +07:27:37 waited 60s for global r=307 +07:27:43 global r=307 load 4.272 -> 4.288 +07:28:16 step 103960 loss=3.5070 lr=7.88e-05 tok=13.646B dt=4201 wait=147 +07:29:20 step 103980 loss=3.3639 lr=7.88e-05 tok=13.649B dt=4264 wait=147 +07:30:23 step 104000 loss=3.3527 lr=7.88e-05 tok=13.651B dt=4328 wait=147 +07:31:27 step 104020 loss=3.4847 lr=7.88e-05 tok=13.654B dt=4391 wait=148 +07:32:31 step 104040 loss=3.3415 lr=7.87e-05 tok=13.656B dt=4455 wait=148 +07:33:34 step 104060 loss=3.4436 lr=7.87e-05 tok=13.659B dt=4518 wait=148 +07:34:37 step 104080 loss=3.3982 lr=7.87e-05 tok=13.662B dt=4582 wait=149 +07:35:41 step 104100 loss=3.4252 lr=7.87e-05 tok=13.664B dt=4645 wait=149 +07:35:57 pushed ckpt_w74.pt,log_w74.txt +07:36:57 waited 60s for global r=308 +07:37:03 global r=308 load 4.298 -> 4.280 +07:38:08 step 104120 loss=3.4039 lr=7.87e-05 tok=13.667B dt=4792 wait=149 +07:39:11 step 104140 loss=3.4255 lr=7.86e-05 tok=13.669B dt=4855 wait=150 +07:40:15 step 104160 loss=3.4037 lr=7.86e-05 tok=13.672B dt=4919 wait=150 +07:41:18 step 104180 loss=3.3711 lr=7.86e-05 tok=13.675B dt=4982 wait=150 +07:42:21 step 104200 loss=3.3816 lr=7.86e-05 tok=13.677B dt=5046 wait=151 +07:43:25 step 104220 loss=3.4941 lr=7.86e-05 tok=13.680B dt=5110 wait=151 +07:44:29 step 104240 loss=3.3997 lr=7.85e-05 tok=13.683B dt=5173 wait=151 +07:45:13 pushed ckpt_w74.pt,log_w74.txt +07:46:14 waited 61s for global r=309 +07:46:21 global r=309 load 4.288 -> 4.285 +07:46:54 step 104260 loss=3.4581 lr=7.85e-05 tok=13.685B dt=5318 wait=152 +07:47:57 step 104280 loss=3.3617 lr=7.85e-05 tok=13.688B dt=5382 wait=152 +07:49:01 step 104300 loss=3.3395 lr=7.85e-05 tok=13.690B dt=5445 wait=152 +07:50:04 step 104320 loss=3.4597 lr=7.85e-05 tok=13.693B dt=5508 wait=153 +07:51:07 step 104340 loss=3.3644 lr=7.84e-05 tok=13.696B dt=5572 wait=153 +07:52:11 step 104360 loss=3.3878 lr=7.84e-05 tok=13.698B dt=5635 wait=153 +07:53:15 step 104380 loss=3.3711 lr=7.84e-05 tok=13.701B dt=5699 wait=154 +07:54:19 step 104400 loss=3.3956 lr=7.84e-05 tok=13.704B dt=5763 wait=154 +07:54:31 pushed ckpt_w74.pt,log_w74.txt +07:55:47 waited 76s for global r=310 +07:55:53 global r=310 load 4.314 -> 4.292 +07:56:59 step 104420 loss=3.4241 lr=7.83e-05 tok=13.706B dt=5923 wait=154 +07:58:02 step 104440 loss=3.4005 lr=7.83e-05 tok=13.709B dt=5987 wait=155 +07:59:09 step 104460 loss=2.9154 lr=7.83e-05 tok=13.711B dt=6053 wait=155 +08:00:13 step 104480 loss=3.3507 lr=7.83e-05 tok=13.714B dt=6117 wait=155 +08:01:17 step 104500 loss=3.4109 lr=7.83e-05 tok=13.717B dt=6181 wait=155 +08:02:21 step 104520 loss=3.4821 lr=7.82e-05 tok=13.719B dt=6245 wait=156 +08:03:24 step 104540 loss=3.3486 lr=7.82e-05 tok=13.722B dt=6308 wait=156 +08:04:08 pushed ckpt_w74.pt,log_w74.txt +08:04:53 waited 45s for global r=311 +08:05:01 global r=311 load 4.311 -> 4.290 +08:05:34 step 104560 loss=3.5198 lr=7.82e-05 tok=13.725B dt=6438 wait=156 +08:06:37 step 104580 loss=3.3478 lr=7.82e-05 tok=13.727B dt=6501 wait=157 +08:07:41 step 104600 loss=3.3761 lr=7.82e-05 tok=13.730B dt=6565 wait=157 +08:08:45 step 104620 loss=3.4514 lr=7.81e-05 tok=13.732B dt=6629 wait=157 +08:09:48 step 104640 loss=3.3551 lr=7.81e-05 tok=13.735B dt=6692 wait=158 +08:10:52 step 104660 loss=3.4308 lr=7.81e-05 tok=13.738B dt=6756 wait=158 +08:11:55 step 104680 loss=3.3145 lr=7.81e-05 tok=13.740B dt=6820 wait=158 +08:12:59 step 104700 loss=3.3669 lr=7.81e-05 tok=13.743B dt=6883 wait=159 +08:13:11 pushed ckpt_w74.pt,log_w74.txt +08:14:11 waited 61s for global r=312 +08:14:18 global r=312 load 4.294 -> 4.287 +08:15:23 step 104720 loss=3.2957 lr=7.80e-05 tok=13.746B dt=7027 wait=159 +08:16:26 step 104740 loss=3.4468 lr=7.80e-05 tok=13.748B dt=7090 wait=159 +08:17:30 step 104760 loss=3.4169 lr=7.80e-05 tok=13.751B dt=7154 wait=160 +08:18:33 step 104780 loss=3.4064 lr=7.80e-05 tok=13.753B dt=7218 wait=160 +08:19:37 step 104800 loss=3.3169 lr=7.79e-05 tok=13.756B dt=7281 wait=160 +08:20:40 step 104820 loss=3.4693 lr=7.79e-05 tok=13.759B dt=7344 wait=161 +08:21:44 step 104840 loss=3.3935 lr=7.79e-05 tok=13.761B dt=7408 wait=161 +08:22:31 pushed ckpt_w74.pt,log_w74.txt +08:23:31 waited 61s for global r=313 +08:23:37 global r=313 load 4.335 -> 4.303 +08:24:10 step 104860 loss=3.4845 lr=7.79e-05 tok=13.764B dt=7554 wait=161 +08:25:14 step 104880 loss=3.3710 lr=7.79e-05 tok=13.766B dt=7618 wait=162 +08:26:17 step 104900 loss=3.3857 lr=7.78e-05 tok=13.769B dt=7682 wait=162 +08:27:21 step 104920 loss=3.4716 lr=7.78e-05 tok=13.772B dt=7745 wait=162 +08:28:24 step 104940 loss=3.3682 lr=7.78e-05 tok=13.774B dt=7809 wait=162 +08:29:28 step 104960 loss=3.4562 lr=7.78e-05 tok=13.777B dt=7873 wait=163 +08:30:32 step 104980 loss=3.3780 lr=7.78e-05 tok=13.780B dt=7936 wait=163 +08:31:35 step 105000 loss=3.4550 lr=7.77e-05 tok=13.782B dt=7999 wait=163 +08:31:49 pushed ckpt_w74.pt,log_w74.txt +08:32:50 waited 60s for global r=314 +08:32:55 global r=314 load 4.302 -> 4.287 +08:34:00 step 105020 loss=3.3776 lr=7.77e-05 tok=13.785B dt=8144 wait=164 +08:35:03 step 105040 loss=3.4047 lr=7.77e-05 tok=13.787B dt=8208 wait=164 +08:36:07 step 105060 loss=3.4247 lr=7.77e-05 tok=13.790B dt=8271 wait=164 +08:37:10 step 105080 loss=3.3935 lr=7.76e-05 tok=13.793B dt=8335 wait=165 +08:38:14 step 105100 loss=3.3665 lr=7.76e-05 tok=13.795B dt=8398 wait=165 +08:39:18 step 105120 loss=3.4968 lr=7.76e-05 tok=13.798B dt=8462 wait=165 +08:40:21 step 105140 loss=3.3575 lr=7.76e-05 tok=13.801B dt=8526 wait=166 +08:41:07 pushed ckpt_w74.pt,log_w74.txt +08:42:08 waited 61s for global r=315 +08:42:17 global r=315 load 4.309 -> 4.284 +08:42:50 step 105160 loss=3.4559 lr=7.76e-05 tok=13.803B dt=8674 wait=166 +08:43:53 step 105180 loss=3.3729 lr=7.75e-05 tok=13.806B dt=8738 wait=166 +08:44:57 step 105200 loss=3.3828 lr=7.75e-05 tok=13.808B dt=8802 wait=167 +08:46:01 step 105220 loss=3.5024 lr=7.75e-05 tok=13.811B dt=8865 wait=167 +08:47:04 step 105240 loss=3.3483 lr=7.75e-05 tok=13.814B dt=8929 wait=167 +08:48:08 step 105260 loss=3.4805 lr=7.75e-05 tok=13.816B dt=8992 wait=168 +08:49:11 step 105280 loss=3.3960 lr=7.74e-05 tok=13.819B dt=9055 wait=168 +08:50:15 step 105300 loss=3.4115 lr=7.74e-05 tok=13.822B dt=9119 wait=168 +08:50:28 pushed ckpt_w74.pt,log_w74.txt +08:51:28 waited 60s for global r=316 +08:51:34 global r=316 load 4.296 -> 4.281 +08:52:39 step 105320 loss=3.4027 lr=7.74e-05 tok=13.824B dt=9263 wait=168 +08:53:43 step 105340 loss=3.3749 lr=7.74e-05 tok=13.827B dt=9327 wait=169 +08:54:47 step 105360 loss=3.4655 lr=7.74e-05 tok=13.829B dt=9391 wait=169 +08:55:50 step 105380 loss=3.3144 lr=7.73e-05 tok=13.832B dt=9454 wait=169 +08:56:53 step 105400 loss=3.3683 lr=7.73e-05 tok=13.835B dt=9517 wait=170 +08:57:56 step 105420 loss=3.4801 lr=7.73e-05 tok=13.837B dt=9581 wait=170 +08:59:00 step 105440 loss=3.3732 lr=7.73e-05 tok=13.840B dt=9644 wait=170 +08:59:45 pushed ckpt_w74.pt,log_w74.txt +09:00:45 waited 61s for global r=317 +09:00:51 global r=317 load 4.305 -> 4.298 +09:01:24 step 105460 loss=3.4369 lr=7.72e-05 tok=13.843B dt=9788 wait=171 +09:02:27 step 105480 loss=3.3286 lr=7.72e-05 tok=13.845B dt=9851 wait=171 +09:03:31 step 105500 loss=3.3650 lr=7.72e-05 tok=13.848B dt=9915 wait=171 +09:04:34 step 105520 loss=3.4522 lr=7.72e-05 tok=13.850B dt=9978 wait=172 +09:05:37 step 105540 loss=3.3391 lr=7.72e-05 tok=13.853B dt=10042 wait=172 +09:06:41 step 105560 loss=3.4358 lr=7.71e-05 tok=13.856B dt=10106 wait=172 +09:07:45 step 105580 loss=3.3913 lr=7.71e-05 tok=13.858B dt=10169 wait=173 +09:08:49 step 105600 loss=3.4078 lr=7.71e-05 tok=13.861B dt=10233 wait=173 diff --git a/log_w75.txt b/log_w75.txt new file mode 100644 index 0000000000000000000000000000000000000000..d0154358b23776951a6334227b3a2dc245fad40d --- /dev/null +++ b/log_w75.txt @@ -0,0 +1,235 @@ +launch +06:16:12 name=w75 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 22/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:16:18 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:16:19 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:16:28 params 1.08B +06:16:30 resumed step=103050 seen=13.51B round=302 +06:16:30 stream salt=103050 +06:18:16 blocks compiled +06:18:16 start: accum=4, 131072 tokens per optimiser step +06:20:47 step 103051 loss=3.2529 lr=7.98e-05 tok=13.511B dt=151 wait=110 +06:20:50 step 103052 loss=3.2336 lr=7.98e-05 tok=13.511B dt=155 wait=110 +06:21:19 step 103060 loss=3.5229 lr=7.98e-05 tok=13.512B dt=183 wait=110 +06:22:27 step 103080 loss=3.4080 lr=7.98e-05 tok=13.515B dt=251 wait=111 +06:23:30 step 103100 loss=3.4496 lr=7.97e-05 tok=13.517B dt=315 wait=111 +06:24:34 step 103120 loss=3.4652 lr=7.97e-05 tok=13.520B dt=378 wait=111 +06:25:37 step 103140 loss=3.3865 lr=7.97e-05 tok=13.523B dt=442 wait=112 +06:26:41 step 103160 loss=3.6160 lr=7.97e-05 tok=13.525B dt=505 wait=112 +06:27:44 step 103180 loss=3.4287 lr=7.97e-05 tok=13.528B dt=569 wait=112 +06:28:47 step 103200 loss=3.5293 lr=7.96e-05 tok=13.531B dt=632 wait=113 +06:29:03 pushed ckpt_w75.pt,log_w75.txt +06:33:12 step 103220 loss=3.5131 lr=7.96e-05 tok=13.533B dt=896 wait=113 +06:34:15 step 103240 loss=3.3852 lr=7.96e-05 tok=13.536B dt=960 wait=113 +06:35:19 step 103260 loss=3.4856 lr=7.96e-05 tok=13.538B dt=1023 wait=114 +06:36:22 step 103280 loss=3.3398 lr=7.96e-05 tok=13.541B dt=1087 wait=114 +06:37:25 step 103300 loss=3.3423 lr=7.95e-05 tok=13.544B dt=1150 wait=114 +06:38:29 step 103320 loss=3.4647 lr=7.95e-05 tok=13.546B dt=1213 wait=115 +06:39:32 step 103340 loss=3.3986 lr=7.95e-05 tok=13.549B dt=1277 wait=115 +06:40:16 pushed ckpt_w75.pt,log_w75.txt +06:43:51 step 103360 loss=3.3818 lr=7.95e-05 tok=13.552B dt=1535 wait=115 +06:44:54 step 103380 loss=3.4774 lr=7.94e-05 tok=13.554B dt=1598 wait=116 +06:45:57 step 103400 loss=3.5245 lr=7.94e-05 tok=13.557B dt=1662 wait=116 +06:47:01 step 103420 loss=3.4310 lr=7.94e-05 tok=13.559B dt=1725 wait=116 +06:48:04 step 103440 loss=3.4077 lr=7.94e-05 tok=13.562B dt=1788 wait=117 +06:49:07 step 103460 loss=3.3828 lr=7.94e-05 tok=13.565B dt=1851 wait=117 +06:50:12 step 103480 loss=3.4279 lr=7.93e-05 tok=13.567B dt=1917 wait=117 +06:50:28 global r=303 load 4.279 -> 4.274 +06:50:59 step 103360 loss=3.4680 lr=7.95e-05 tok=13.569B dt=1964 wait=118 +06:52:04 step 103380 loss=3.1321 lr=7.94e-05 tok=13.571B dt=2028 wait=118 +06:53:07 step 103400 loss=3.3571 lr=7.94e-05 tok=13.574B dt=2091 wait=118 +06:54:10 step 103420 loss=3.4814 lr=7.94e-05 tok=13.576B dt=2154 wait=118 +06:55:13 step 103440 loss=3.3918 lr=7.94e-05 tok=13.579B dt=2218 wait=119 +06:56:17 step 103460 loss=3.4851 lr=7.94e-05 tok=13.582B dt=2282 wait=119 +06:57:21 step 103480 loss=3.3894 lr=7.93e-05 tok=13.584B dt=2345 wait=119 +06:58:24 step 103500 loss=3.5019 lr=7.93e-05 tok=13.587B dt=2409 wait=120 +06:58:37 pushed ckpt_w75.pt,log_w75.txt +06:59:37 waited 61s for global r=304 +06:59:50 global r=304 load 4.315 -> 4.273 +07:00:58 step 103520 loss=3.3764 lr=7.93e-05 tok=13.590B dt=2562 wait=120 +07:02:01 step 103540 loss=3.4615 lr=7.93e-05 tok=13.592B dt=2625 wait=120 +07:03:04 step 103560 loss=3.3980 lr=7.93e-05 tok=13.595B dt=2689 wait=121 +07:04:07 step 103580 loss=3.3396 lr=7.92e-05 tok=13.597B dt=2752 wait=121 +07:05:11 step 103600 loss=3.3458 lr=7.92e-05 tok=13.600B dt=2815 wait=121 +07:06:14 step 103620 loss=3.4407 lr=7.92e-05 tok=13.603B dt=2878 wait=122 +07:07:17 step 103640 loss=3.4002 lr=7.92e-05 tok=13.605B dt=2941 wait=122 +07:08:04 pushed ckpt_w75.pt,log_w75.txt +07:09:05 waited 61s for global r=305 +07:09:13 global r=305 load 4.317 -> 4.288 +07:09:46 step 103660 loss=3.5067 lr=7.92e-05 tok=13.608B dt=3090 wait=122 +07:10:49 step 103680 loss=3.3891 lr=7.91e-05 tok=13.611B dt=3153 wait=123 +07:11:52 step 103700 loss=3.3811 lr=7.91e-05 tok=13.613B dt=3217 wait=123 +07:12:55 step 103720 loss=3.4592 lr=7.91e-05 tok=13.616B dt=3280 wait=123 +07:13:59 step 103740 loss=3.3645 lr=7.91e-05 tok=13.618B dt=3343 wait=124 +07:15:02 step 103760 loss=3.4026 lr=7.90e-05 tok=13.621B dt=3406 wait=124 +07:16:05 step 103780 loss=3.3730 lr=7.90e-05 tok=13.624B dt=3469 wait=124 +07:17:08 step 103800 loss=3.4493 lr=7.90e-05 tok=13.626B dt=3533 wait=124 +07:17:29 pushed ckpt_w75.pt,log_w75.txt +07:18:15 waited 46s for global r=306 +07:18:27 global r=306 load 4.307 -> 4.286 +07:19:31 step 103820 loss=3.3640 lr=7.90e-05 tok=13.629B dt=3676 wait=125 +07:20:35 step 103840 loss=3.4225 lr=7.90e-05 tok=13.631B dt=3739 wait=125 +07:21:38 step 103860 loss=3.4256 lr=7.89e-05 tok=13.634B dt=3803 wait=125 +07:22:41 step 103880 loss=3.3854 lr=7.89e-05 tok=13.637B dt=3866 wait=126 +07:23:44 step 103900 loss=3.4078 lr=7.89e-05 tok=13.639B dt=3929 wait=126 +07:24:48 step 103920 loss=3.4682 lr=7.89e-05 tok=13.642B dt=3992 wait=126 +07:25:50 step 103940 loss=3.3593 lr=7.89e-05 tok=13.645B dt=4055 wait=127 +07:26:34 pushed ckpt_w75.pt,log_w75.txt +07:27:36 waited 61s for global r=307 +07:27:44 global r=307 load 4.319 -> 4.288 +07:28:17 step 103960 loss=3.4720 lr=7.88e-05 tok=13.647B dt=4201 wait=127 +07:29:20 step 103980 loss=3.4070 lr=7.88e-05 tok=13.650B dt=4265 wait=127 +07:30:24 step 104000 loss=3.4156 lr=7.88e-05 tok=13.652B dt=4328 wait=128 +07:31:27 step 104020 loss=3.5011 lr=7.88e-05 tok=13.655B dt=4392 wait=128 +07:32:31 step 104040 loss=3.3231 lr=7.87e-05 tok=13.658B dt=4455 wait=128 +07:33:34 step 104060 loss=3.4527 lr=7.87e-05 tok=13.660B dt=4518 wait=129 +07:34:37 step 104080 loss=3.4364 lr=7.87e-05 tok=13.663B dt=4581 wait=129 +07:35:40 step 104100 loss=3.4315 lr=7.87e-05 tok=13.666B dt=4644 wait=129 +07:35:52 pushed ckpt_w75.pt,log_w75.txt +07:37:08 waited 77s for global r=308 +07:37:15 global r=308 load 4.294 -> 4.280 +07:38:20 step 104120 loss=3.3995 lr=7.87e-05 tok=13.668B dt=4805 wait=130 +07:39:24 step 104140 loss=3.4724 lr=7.86e-05 tok=13.671B dt=4868 wait=130 +07:40:27 step 104160 loss=3.3923 lr=7.86e-05 tok=13.673B dt=4932 wait=130 +07:41:31 step 104180 loss=3.3600 lr=7.86e-05 tok=13.676B dt=4995 wait=131 +07:42:34 step 104200 loss=3.4248 lr=7.86e-05 tok=13.679B dt=5059 wait=131 +07:43:38 step 104220 loss=3.4531 lr=7.86e-05 tok=13.681B dt=5122 wait=131 +07:44:41 step 104240 loss=3.3468 lr=7.85e-05 tok=13.684B dt=5185 wait=131 +07:45:35 pushed ckpt_w75.pt,log_w75.txt +07:46:21 waited 46s for global r=309 +07:46:27 global r=309 load 4.297 -> 4.285 +07:47:00 step 104260 loss=3.4943 lr=7.85e-05 tok=13.687B dt=5324 wait=132 +07:48:03 step 104280 loss=3.3785 lr=7.85e-05 tok=13.689B dt=5387 wait=132 +07:49:06 step 104300 loss=3.3507 lr=7.85e-05 tok=13.692B dt=5451 wait=132 +07:50:10 step 104320 loss=3.4294 lr=7.85e-05 tok=13.694B dt=5514 wait=133 +07:51:13 step 104340 loss=3.3624 lr=7.84e-05 tok=13.697B dt=5578 wait=133 +07:52:17 step 104360 loss=3.4221 lr=7.84e-05 tok=13.700B dt=5641 wait=133 +07:53:20 step 104380 loss=3.3386 lr=7.84e-05 tok=13.702B dt=5705 wait=134 +07:54:24 step 104400 loss=3.4539 lr=7.84e-05 tok=13.705B dt=5768 wait=134 +07:54:36 pushed ckpt_w75.pt,log_w75.txt +07:55:44 waited 68s for global r=310 +07:55:53 global r=310 load 4.313 -> 4.292 +07:56:58 step 104420 loss=3.3635 lr=7.83e-05 tok=13.708B dt=5922 wait=134 +07:58:01 step 104440 loss=3.3619 lr=7.83e-05 tok=13.710B dt=5986 wait=135 +07:59:05 step 104460 loss=3.3891 lr=7.83e-05 tok=13.713B dt=6049 wait=135 +08:00:08 step 104480 loss=3.4164 lr=7.83e-05 tok=13.715B dt=6112 wait=135 +08:01:12 step 104500 loss=3.4035 lr=7.83e-05 tok=13.718B dt=6176 wait=136 +08:02:15 step 104520 loss=3.4695 lr=7.82e-05 tok=13.721B dt=6239 wait=136 +08:03:18 step 104540 loss=3.3416 lr=7.82e-05 tok=13.723B dt=6302 wait=136 +08:04:01 pushed ckpt_w75.pt,log_w75.txt +08:05:01 waited 61s for global r=311 +08:05:08 global r=311 load 4.323 -> 4.290 +08:05:41 step 104560 loss=3.4681 lr=7.82e-05 tok=13.726B dt=6445 wait=137 +08:06:44 step 104580 loss=3.3692 lr=7.82e-05 tok=13.728B dt=6509 wait=137 +08:07:48 step 104600 loss=3.4321 lr=7.82e-05 tok=13.731B dt=6572 wait=137 +08:08:51 step 104620 loss=3.4679 lr=7.81e-05 tok=13.734B dt=6635 wait=138 +08:09:54 step 104640 loss=3.3803 lr=7.81e-05 tok=13.736B dt=6699 wait=138 +08:10:58 step 104660 loss=3.4711 lr=7.81e-05 tok=13.739B dt=6762 wait=138 +08:12:01 step 104680 loss=3.4416 lr=7.81e-05 tok=13.742B dt=6826 wait=138 +08:13:05 step 104700 loss=3.4121 lr=7.81e-05 tok=13.744B dt=6889 wait=139 +08:13:22 pushed ckpt_w75.pt,log_w75.txt +08:14:14 waited 52s for global r=312 +08:14:24 global r=312 load 4.313 -> 4.287 +08:15:28 step 104720 loss=3.3791 lr=7.80e-05 tok=13.747B dt=7033 wait=139 +08:16:32 step 104740 loss=3.3600 lr=7.80e-05 tok=13.749B dt=7097 wait=140 +08:17:35 step 104760 loss=3.4071 lr=7.80e-05 tok=13.752B dt=7160 wait=140 +08:18:39 step 104780 loss=3.3831 lr=7.80e-05 tok=13.755B dt=7223 wait=140 +08:19:42 step 104800 loss=3.3802 lr=7.79e-05 tok=13.757B dt=7286 wait=141 +08:20:45 step 104820 loss=3.5183 lr=7.79e-05 tok=13.760B dt=7349 wait=141 +08:21:48 step 104840 loss=3.2877 lr=7.79e-05 tok=13.763B dt=7413 wait=141 +08:22:36 pushed ckpt_w75.pt,log_w75.txt +08:23:37 waited 61s for global r=313 +08:23:45 global r=313 load 4.332 -> 4.303 +08:24:18 step 104860 loss=3.4757 lr=7.79e-05 tok=13.765B dt=7562 wait=141 +08:25:21 step 104880 loss=3.3472 lr=7.79e-05 tok=13.768B dt=7625 wait=142 +08:26:24 step 104900 loss=3.4024 lr=7.78e-05 tok=13.770B dt=7689 wait=142 +08:27:28 step 104920 loss=3.4873 lr=7.78e-05 tok=13.773B dt=7752 wait=142 +08:28:31 step 104940 loss=3.3472 lr=7.78e-05 tok=13.776B dt=7816 wait=143 +08:29:35 step 104960 loss=3.4337 lr=7.78e-05 tok=13.778B dt=7879 wait=143 +08:30:38 step 104980 loss=3.4071 lr=7.78e-05 tok=13.781B dt=7942 wait=143 +08:31:42 step 105000 loss=3.4287 lr=7.77e-05 tok=13.784B dt=8006 wait=144 +08:32:02 pushed ckpt_w75.pt,log_w75.txt +08:32:54 waited 51s for global r=314 +08:33:02 global r=314 load 4.323 -> 4.287 +08:34:06 step 105020 loss=3.3611 lr=7.77e-05 tok=13.786B dt=8150 wait=144 +08:35:09 step 105040 loss=3.4449 lr=7.77e-05 tok=13.789B dt=8213 wait=144 +08:36:12 step 105060 loss=3.4132 lr=7.77e-05 tok=13.791B dt=8277 wait=145 +08:37:16 step 105080 loss=3.3775 lr=7.76e-05 tok=13.794B dt=8340 wait=145 +08:38:19 step 105100 loss=3.4115 lr=7.76e-05 tok=13.797B dt=8403 wait=145 +08:39:22 step 105120 loss=3.4725 lr=7.76e-05 tok=13.799B dt=8467 wait=146 +08:40:25 step 105140 loss=3.3874 lr=7.76e-05 tok=13.802B dt=8530 wait=146 +08:41:10 pushed ckpt_w75.pt,log_w75.txt +08:42:11 waited 61s for global r=315 +08:42:18 global r=315 load 4.270 -> 4.284 +08:42:51 step 105160 loss=3.4782 lr=7.76e-05 tok=13.805B dt=8675 wait=146 +08:43:54 step 105180 loss=3.3412 lr=7.75e-05 tok=13.807B dt=8738 wait=147 +08:44:57 step 105200 loss=3.3185 lr=7.75e-05 tok=13.810B dt=8801 wait=147 +08:46:00 step 105220 loss=3.4082 lr=7.75e-05 tok=13.812B dt=8865 wait=147 +08:47:04 step 105240 loss=3.3303 lr=7.75e-05 tok=13.815B dt=8928 wait=148 +08:48:07 step 105260 loss=3.4433 lr=7.75e-05 tok=13.818B dt=8991 wait=148 +08:49:10 step 105280 loss=3.3691 lr=7.74e-05 tok=13.820B dt=9055 wait=148 +08:50:14 step 105300 loss=3.3949 lr=7.74e-05 tok=13.823B dt=9118 wait=149 +08:50:27 pushed ckpt_w75.pt,log_w75.txt +08:51:29 waited 61s for global r=316 +08:51:36 global r=316 load 4.301 -> 4.281 +08:52:40 step 105320 loss=3.3366 lr=7.74e-05 tok=13.825B dt=9264 wait=149 +08:53:43 step 105340 loss=3.4390 lr=7.74e-05 tok=13.828B dt=9327 wait=149 +08:54:46 step 105360 loss=3.4180 lr=7.74e-05 tok=13.831B dt=9391 wait=149 +08:55:49 step 105380 loss=3.3571 lr=7.73e-05 tok=13.833B dt=9454 wait=150 +08:56:53 step 105400 loss=3.3559 lr=7.73e-05 tok=13.836B dt=9517 wait=150 +08:57:56 step 105420 loss=3.4110 lr=7.73e-05 tok=13.839B dt=9580 wait=150 +08:59:00 step 105440 loss=3.3140 lr=7.73e-05 tok=13.841B dt=9644 wait=151 +08:59:43 pushed ckpt_w75.pt,log_w75.txt +09:00:50 waited 67s for global r=317 +09:00:56 global r=317 load 4.331 -> 4.298 +09:01:28 step 105460 loss=3.4757 lr=7.72e-05 tok=13.844B dt=9792 wait=151 +09:02:31 step 105480 loss=3.3710 lr=7.72e-05 tok=13.846B dt=9856 wait=151 +09:03:35 step 105500 loss=3.3745 lr=7.72e-05 tok=13.849B dt=9919 wait=152 +09:04:38 step 105520 loss=3.4401 lr=7.72e-05 tok=13.852B dt=9983 wait=152 +09:05:42 step 105540 loss=3.3182 lr=7.72e-05 tok=13.854B dt=10046 wait=152 +09:06:46 step 105560 loss=3.3943 lr=7.71e-05 tok=13.857B dt=10110 wait=153 +09:07:49 step 105580 loss=3.3646 lr=7.71e-05 tok=13.860B dt=10174 wait=153 +09:08:52 step 105600 loss=3.3727 lr=7.71e-05 tok=13.862B dt=10237 wait=153 +09:09:04 pushed ckpt_w75.pt,log_w75.txt +09:10:05 waited 61s for global r=318 +09:10:12 global r=318 load 4.280 -> 4.280 +09:11:17 step 105620 loss=3.3295 lr=7.71e-05 tok=13.865B dt=10381 wait=154 +09:12:20 step 105640 loss=3.3772 lr=7.71e-05 tok=13.867B dt=10444 wait=154 +09:13:23 step 105660 loss=3.4026 lr=7.70e-05 tok=13.870B dt=10508 wait=154 +09:14:26 step 105680 loss=3.3535 lr=7.70e-05 tok=13.873B dt=10571 wait=154 +09:15:29 step 105700 loss=3.3810 lr=7.70e-05 tok=13.875B dt=10634 wait=155 +09:16:33 step 105720 loss=3.4315 lr=7.70e-05 tok=13.878B dt=10698 wait=155 +09:17:36 step 105740 loss=3.3248 lr=7.70e-05 tok=13.881B dt=10761 wait=155 +09:18:23 pushed ckpt_w75.pt,log_w75.txt +09:21:57 step 105760 loss=3.5010 lr=7.69e-05 tok=13.883B dt=11021 wait=156 +09:23:00 step 105780 loss=3.3211 lr=7.69e-05 tok=13.886B dt=11085 wait=156 +09:24:04 step 105800 loss=3.3091 lr=7.69e-05 tok=13.888B dt=11148 wait=156 +09:25:08 step 105820 loss=3.1541 lr=7.69e-05 tok=13.891B dt=11212 wait=157 +09:26:11 step 105840 loss=3.3674 lr=7.68e-05 tok=13.894B dt=11276 wait=157 +09:27:14 step 105860 loss=3.3789 lr=7.68e-05 tok=13.896B dt=11339 wait=157 +09:28:17 step 105880 loss=3.4016 lr=7.68e-05 tok=13.899B dt=11401 wait=158 +09:29:21 step 105900 loss=3.5361 lr=7.68e-05 tok=13.901B dt=11465 wait=158 +09:29:34 pushed ckpt_w75.pt,log_w75.txt +09:33:39 step 105920 loss=3.4485 lr=7.68e-05 tok=13.904B dt=11724 wait=158 +09:34:43 step 105940 loss=3.5120 lr=7.67e-05 tok=13.907B dt=11787 wait=159 +09:35:46 step 105960 loss=3.4226 lr=7.67e-05 tok=13.909B dt=11850 wait=159 +09:36:49 step 105980 loss=3.4007 lr=7.67e-05 tok=13.912B dt=11914 wait=159 +09:37:52 step 106000 loss=3.3787 lr=7.67e-05 tok=13.915B dt=11977 wait=160 +09:38:56 step 106020 loss=3.4070 lr=7.67e-05 tok=13.917B dt=12040 wait=160 +09:39:59 step 106040 loss=3.3680 lr=7.66e-05 tok=13.920B dt=12103 wait=160 +09:40:42 pushed ckpt_w75.pt,log_w75.txt +09:44:15 step 106060 loss=3.3925 lr=7.66e-05 tok=13.922B dt=12360 wait=161 +09:45:18 step 106080 loss=3.3432 lr=7.66e-05 tok=13.925B dt=12422 wait=161 +09:46:21 step 106100 loss=3.4940 lr=7.66e-05 tok=13.928B dt=12485 wait=161 +09:47:24 step 106120 loss=3.4144 lr=7.65e-05 tok=13.930B dt=12548 wait=161 +09:48:27 step 106140 loss=3.4071 lr=7.65e-05 tok=13.933B dt=12611 wait=162 +09:49:30 step 106160 loss=3.4135 lr=7.65e-05 tok=13.936B dt=12675 wait=162 +09:50:34 step 106180 loss=3.4324 lr=7.65e-05 tok=13.938B dt=12738 wait=162 +09:51:37 step 106200 loss=3.3884 lr=7.65e-05 tok=13.941B dt=12801 wait=163 +09:51:50 pushed ckpt_w75.pt,log_w75.txt +09:55:55 step 106220 loss=3.3512 lr=7.64e-05 tok=13.943B dt=13060 wait=163 +09:56:59 step 106240 loss=3.4383 lr=7.64e-05 tok=13.946B dt=13123 wait=163 +09:58:01 step 106260 loss=3.3962 lr=7.64e-05 tok=13.949B dt=13186 wait=164 +09:59:05 step 106280 loss=3.3653 lr=7.64e-05 tok=13.951B dt=13249 wait=164 +10:00:08 step 106300 loss=3.3955 lr=7.64e-05 tok=13.954B dt=13312 wait=164 +10:01:11 step 106320 loss=3.4195 lr=7.63e-05 tok=13.957B dt=13375 wait=165 +10:02:15 step 106340 loss=3.3320 lr=7.63e-05 tok=13.959B dt=13439 wait=165 diff --git a/log_w76.txt b/log_w76.txt new file mode 100644 index 0000000000000000000000000000000000000000..871b412b7bf9ea744c0a43ea761ec4fe9b9e1f9f --- /dev/null +++ b/log_w76.txt @@ -0,0 +1,191 @@ +launch +06:16:15 name=w76 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 23/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:16:21 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:16:21 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:16:31 params 1.08B +06:16:32 resumed step=103050 seen=13.51B round=302 +06:16:32 stream salt=103050 +06:18:19 blocks compiled +06:18:19 start: accum=4, 131072 tokens per optimiser step +06:20:52 step 103051 loss=3.0115 lr=7.98e-05 tok=13.511B dt=152 wait=106 +06:20:55 step 103052 loss=3.2552 lr=7.98e-05 tok=13.511B dt=156 wait=107 +06:21:24 step 103060 loss=3.6156 lr=7.98e-05 tok=13.512B dt=185 wait=107 +06:22:37 step 103080 loss=3.5164 lr=7.98e-05 tok=13.515B dt=257 wait=108 +06:23:41 step 103100 loss=3.4772 lr=7.97e-05 tok=13.517B dt=322 wait=108 +06:24:45 step 103120 loss=3.4319 lr=7.97e-05 tok=13.520B dt=386 wait=108 +06:25:50 step 103140 loss=3.5304 lr=7.97e-05 tok=13.523B dt=450 wait=109 +06:26:54 step 103160 loss=3.6408 lr=7.97e-05 tok=13.525B dt=515 wait=109 +06:27:58 step 103180 loss=3.5050 lr=7.97e-05 tok=13.528B dt=579 wait=109 +06:29:03 step 103200 loss=3.5672 lr=7.96e-05 tok=13.531B dt=644 wait=109 +06:29:20 pushed ckpt_w76.pt,log_w76.txt +06:33:29 step 103220 loss=3.6043 lr=7.96e-05 tok=13.533B dt=910 wait=110 +06:34:34 step 103240 loss=3.4568 lr=7.96e-05 tok=13.536B dt=975 wait=110 +06:35:38 step 103260 loss=3.4888 lr=7.96e-05 tok=13.538B dt=1039 wait=110 +06:36:42 step 103280 loss=3.4750 lr=7.96e-05 tok=13.541B dt=1103 wait=111 +06:37:46 step 103300 loss=3.4092 lr=7.95e-05 tok=13.544B dt=1167 wait=111 +06:38:50 step 103320 loss=3.4766 lr=7.95e-05 tok=13.546B dt=1231 wait=111 +06:39:54 step 103340 loss=3.5673 lr=7.95e-05 tok=13.549B dt=1295 wait=112 +06:40:40 pushed ckpt_w76.pt,log_w76.txt +06:44:15 step 103360 loss=3.4869 lr=7.95e-05 tok=13.552B dt=1556 wait=112 +06:45:19 step 103380 loss=3.5713 lr=7.94e-05 tok=13.554B dt=1620 wait=112 +06:46:23 step 103400 loss=3.5646 lr=7.94e-05 tok=13.557B dt=1684 wait=113 +06:47:27 step 103420 loss=3.4861 lr=7.94e-05 tok=13.559B dt=1748 wait=113 +06:48:32 step 103440 loss=3.4771 lr=7.94e-05 tok=13.562B dt=1813 wait=113 +06:49:36 step 103460 loss=3.4349 lr=7.94e-05 tok=13.565B dt=1876 wait=114 +06:50:40 step 103480 loss=3.4895 lr=7.93e-05 tok=13.567B dt=1941 wait=114 +06:50:49 global r=303 load 4.296 -> 4.274 +06:51:21 step 103360 loss=3.4577 lr=7.95e-05 tok=13.569B dt=1982 wait=114 +06:52:25 step 103380 loss=3.4024 lr=7.94e-05 tok=13.571B dt=2046 wait=115 +06:53:30 step 103400 loss=3.4322 lr=7.94e-05 tok=13.574B dt=2111 wait=115 +06:54:34 step 103420 loss=3.5310 lr=7.94e-05 tok=13.576B dt=2175 wait=115 +06:55:41 step 103440 loss=3.3552 lr=7.94e-05 tok=13.579B dt=2242 wait=115 +06:56:45 step 103460 loss=3.5089 lr=7.94e-05 tok=13.582B dt=2306 wait=116 +06:57:50 step 103480 loss=3.3757 lr=7.93e-05 tok=13.584B dt=2371 wait=116 +06:58:54 step 103500 loss=3.5343 lr=7.93e-05 tok=13.587B dt=2435 wait=116 +06:59:08 pushed ckpt_w76.pt,log_w76.txt +06:59:39 waited 30s for global r=304 +06:59:48 global r=304 load 4.289 -> 4.273 +07:00:54 step 103520 loss=3.4001 lr=7.93e-05 tok=13.590B dt=2555 wait=117 +07:01:58 step 103540 loss=3.5130 lr=7.93e-05 tok=13.592B dt=2619 wait=117 +07:03:03 step 103560 loss=3.4902 lr=7.93e-05 tok=13.595B dt=2683 wait=117 +07:04:07 step 103580 loss=3.4410 lr=7.92e-05 tok=13.597B dt=2748 wait=118 +07:05:12 step 103600 loss=3.4693 lr=7.92e-05 tok=13.600B dt=2812 wait=118 +07:06:16 step 103620 loss=3.5417 lr=7.92e-05 tok=13.603B dt=2877 wait=118 +07:07:20 step 103640 loss=3.4114 lr=7.92e-05 tok=13.605B dt=2941 wait=119 +07:08:05 pushed ckpt_w76.pt,log_w76.txt +07:09:05 waited 60s for global r=305 +07:09:11 global r=305 load 4.287 -> 4.288 +07:09:45 step 103660 loss=3.5838 lr=7.92e-05 tok=13.608B dt=3085 wait=119 +07:10:49 step 103680 loss=3.4045 lr=7.91e-05 tok=13.611B dt=3150 wait=119 +07:11:53 step 103700 loss=3.4613 lr=7.91e-05 tok=13.613B dt=3214 wait=120 +07:12:57 step 103720 loss=3.5552 lr=7.91e-05 tok=13.616B dt=3278 wait=120 +07:14:02 step 103740 loss=3.3877 lr=7.91e-05 tok=13.618B dt=3342 wait=120 +07:15:06 step 103760 loss=3.5796 lr=7.90e-05 tok=13.621B dt=3407 wait=121 +07:16:10 step 103780 loss=3.4532 lr=7.90e-05 tok=13.624B dt=3471 wait=121 +07:17:15 step 103800 loss=3.5100 lr=7.90e-05 tok=13.626B dt=3536 wait=121 +07:17:27 pushed ckpt_w76.pt,log_w76.txt +07:18:12 waited 45s for global r=306 +07:18:20 global r=306 load 4.285 -> 4.286 +07:19:27 step 103820 loss=3.4839 lr=7.90e-05 tok=13.629B dt=3668 wait=122 +07:20:31 step 103840 loss=3.5325 lr=7.90e-05 tok=13.631B dt=3732 wait=122 +07:21:36 step 103860 loss=3.5287 lr=7.89e-05 tok=13.634B dt=3796 wait=122 +07:22:40 step 103880 loss=3.4848 lr=7.89e-05 tok=13.637B dt=3861 wait=123 +07:23:45 step 103900 loss=3.5211 lr=7.89e-05 tok=13.639B dt=3925 wait=123 +07:24:49 step 103920 loss=3.5889 lr=7.89e-05 tok=13.642B dt=3990 wait=123 +07:25:53 step 103940 loss=3.4448 lr=7.89e-05 tok=13.645B dt=4054 wait=124 +07:26:38 pushed ckpt_w76.pt,log_w76.txt +07:27:38 waited 60s for global r=307 +07:27:43 global r=307 load 4.283 -> 4.288 +07:28:17 step 103960 loss=3.5626 lr=7.88e-05 tok=13.647B dt=4197 wait=124 +07:29:21 step 103980 loss=3.4517 lr=7.88e-05 tok=13.650B dt=4262 wait=124 +07:30:25 step 104000 loss=3.4074 lr=7.88e-05 tok=13.652B dt=4326 wait=125 +07:31:29 step 104020 loss=3.5235 lr=7.88e-05 tok=13.655B dt=4390 wait=125 +07:32:33 step 104040 loss=3.4165 lr=7.87e-05 tok=13.658B dt=4454 wait=125 +07:33:37 step 104060 loss=3.4785 lr=7.87e-05 tok=13.660B dt=4518 wait=126 +07:34:41 step 104080 loss=3.3814 lr=7.87e-05 tok=13.663B dt=4582 wait=126 +07:35:44 step 104100 loss=3.3955 lr=7.87e-05 tok=13.666B dt=4645 wait=126 +07:35:56 pushed ckpt_w76.pt,log_w76.txt +07:36:57 waited 61s for global r=308 +07:37:03 global r=308 load 4.282 -> 4.280 +07:38:08 step 104120 loss=3.4170 lr=7.87e-05 tok=13.668B dt=4789 wait=127 +07:39:12 step 104140 loss=3.4726 lr=7.86e-05 tok=13.671B dt=4853 wait=127 +07:40:16 step 104160 loss=3.4083 lr=7.86e-05 tok=13.673B dt=4916 wait=127 +07:41:19 step 104180 loss=3.3884 lr=7.86e-05 tok=13.676B dt=4980 wait=128 +07:42:23 step 104200 loss=3.3898 lr=7.86e-05 tok=13.679B dt=5044 wait=128 +07:43:26 step 104220 loss=3.4875 lr=7.86e-05 tok=13.681B dt=5107 wait=128 +07:44:30 step 104240 loss=3.3534 lr=7.85e-05 tok=13.684B dt=5171 wait=129 +07:45:14 pushed ckpt_w76.pt,log_w76.txt +07:46:14 waited 61s for global r=309 +07:46:21 global r=309 load 4.310 -> 4.285 +07:46:54 step 104260 loss=3.4293 lr=7.85e-05 tok=13.687B dt=5315 wait=129 +07:47:58 step 104280 loss=3.3484 lr=7.85e-05 tok=13.689B dt=5379 wait=129 +07:49:02 step 104300 loss=3.3658 lr=7.85e-05 tok=13.692B dt=5443 wait=130 +07:50:05 step 104320 loss=3.4539 lr=7.85e-05 tok=13.694B dt=5506 wait=130 +07:51:10 step 104340 loss=3.3363 lr=7.84e-05 tok=13.697B dt=5571 wait=130 +07:52:14 step 104360 loss=3.4329 lr=7.84e-05 tok=13.700B dt=5635 wait=130 +07:53:17 step 104380 loss=3.3924 lr=7.84e-05 tok=13.702B dt=5698 wait=131 +07:54:21 step 104400 loss=3.3507 lr=7.84e-05 tok=13.705B dt=5762 wait=131 +07:54:36 pushed ckpt_w76.pt,log_w76.txt +07:55:36 waited 61s for global r=310 +07:55:42 global r=310 load 4.307 -> 4.292 +07:56:48 step 104420 loss=3.3934 lr=7.83e-05 tok=13.708B dt=5908 wait=131 +07:57:51 step 104440 loss=3.4268 lr=7.83e-05 tok=13.710B dt=5972 wait=132 +07:58:56 step 104460 loss=3.3535 lr=7.83e-05 tok=13.713B dt=6036 wait=132 +07:59:59 step 104480 loss=3.3670 lr=7.83e-05 tok=13.715B dt=6100 wait=132 +08:01:03 step 104500 loss=3.4321 lr=7.83e-05 tok=13.718B dt=6164 wait=133 +08:02:07 step 104520 loss=3.4880 lr=7.82e-05 tok=13.721B dt=6228 wait=133 +08:03:11 step 104540 loss=3.3653 lr=7.82e-05 tok=13.723B dt=6292 wait=133 +08:03:57 pushed ckpt_w76.pt,log_w76.txt +08:04:58 waited 61s for global r=311 +08:05:04 global r=311 load 4.313 -> 4.290 +08:05:37 step 104560 loss=3.4681 lr=7.82e-05 tok=13.726B dt=6438 wait=134 +08:06:40 step 104580 loss=3.3886 lr=7.82e-05 tok=13.728B dt=6501 wait=134 +08:07:43 step 104600 loss=3.3136 lr=7.82e-05 tok=13.731B dt=6564 wait=134 +08:08:47 step 104620 loss=3.4842 lr=7.81e-05 tok=13.734B dt=6628 wait=135 +08:09:52 step 104640 loss=3.3719 lr=7.81e-05 tok=13.736B dt=6692 wait=135 +08:10:55 step 104660 loss=3.4471 lr=7.81e-05 tok=13.739B dt=6756 wait=135 +08:11:59 step 104680 loss=3.3876 lr=7.81e-05 tok=13.742B dt=6820 wait=135 +08:13:03 step 104700 loss=3.4687 lr=7.81e-05 tok=13.744B dt=6884 wait=136 +08:13:16 pushed ckpt_w76.pt,log_w76.txt +08:14:17 waited 61s for global r=312 +08:14:23 global r=312 load 4.304 -> 4.287 +08:15:28 step 104720 loss=3.3876 lr=7.80e-05 tok=13.747B dt=7029 wait=136 +08:16:32 step 104740 loss=3.4415 lr=7.80e-05 tok=13.749B dt=7092 wait=136 +08:17:35 step 104760 loss=3.4181 lr=7.80e-05 tok=13.752B dt=7156 wait=137 +08:18:39 step 104780 loss=3.3828 lr=7.80e-05 tok=13.755B dt=7220 wait=137 +08:19:43 step 104800 loss=3.3877 lr=7.79e-05 tok=13.757B dt=7283 wait=137 +08:20:46 step 104820 loss=3.4722 lr=7.79e-05 tok=13.760B dt=7347 wait=138 +08:21:50 step 104840 loss=3.3495 lr=7.79e-05 tok=13.763B dt=7411 wait=138 +08:22:34 pushed ckpt_w76.pt,log_w76.txt +08:23:34 waited 60s for global r=313 +08:23:43 global r=313 load 4.311 -> 4.303 +08:24:16 step 104860 loss=3.4235 lr=7.79e-05 tok=13.765B dt=7557 wait=138 +08:25:19 step 104880 loss=3.3398 lr=7.79e-05 tok=13.768B dt=7620 wait=139 +08:26:23 step 104900 loss=3.3705 lr=7.78e-05 tok=13.770B dt=7684 wait=139 +08:27:26 step 104920 loss=3.4638 lr=7.78e-05 tok=13.773B dt=7747 wait=139 +08:28:30 step 104940 loss=3.3558 lr=7.78e-05 tok=13.776B dt=7811 wait=139 +08:29:33 step 104960 loss=3.4510 lr=7.78e-05 tok=13.778B dt=7874 wait=140 +08:30:37 step 104980 loss=3.4236 lr=7.78e-05 tok=13.781B dt=7938 wait=140 +08:31:41 step 105000 loss=3.4258 lr=7.77e-05 tok=13.784B dt=8002 wait=140 +08:31:55 pushed ckpt_w76.pt,log_w76.txt +08:32:56 waited 61s for global r=314 +08:33:01 global r=314 load 4.278 -> 4.287 +08:34:06 step 105020 loss=3.3764 lr=7.77e-05 tok=13.786B dt=8147 wait=141 +08:35:10 step 105040 loss=3.4225 lr=7.77e-05 tok=13.789B dt=8211 wait=141 +08:36:14 step 105060 loss=3.4230 lr=7.77e-05 tok=13.791B dt=8274 wait=141 +08:37:17 step 105080 loss=3.3372 lr=7.76e-05 tok=13.794B dt=8338 wait=142 +08:38:21 step 105100 loss=3.3731 lr=7.76e-05 tok=13.797B dt=8402 wait=142 +08:39:25 step 105120 loss=3.4834 lr=7.76e-05 tok=13.799B dt=8466 wait=142 +08:40:29 step 105140 loss=3.3728 lr=7.76e-05 tok=13.802B dt=8529 wait=143 +08:41:12 pushed ckpt_w76.pt,log_w76.txt +08:42:13 waited 61s for global r=315 +08:42:22 global r=315 load 4.325 -> 4.284 +08:42:55 step 105160 loss=3.4374 lr=7.76e-05 tok=13.805B dt=8676 wait=143 +08:43:59 step 105180 loss=3.4238 lr=7.75e-05 tok=13.807B dt=8740 wait=143 +08:45:03 step 105200 loss=3.4147 lr=7.75e-05 tok=13.810B dt=8804 wait=144 +08:46:07 step 105220 loss=3.4375 lr=7.75e-05 tok=13.812B dt=8868 wait=144 +08:47:11 step 105240 loss=3.3835 lr=7.75e-05 tok=13.815B dt=8932 wait=144 +08:48:15 step 105260 loss=3.4615 lr=7.75e-05 tok=13.818B dt=8995 wait=144 +08:49:18 step 105280 loss=3.3807 lr=7.74e-05 tok=13.820B dt=9059 wait=145 +08:50:22 step 105300 loss=3.3432 lr=7.74e-05 tok=13.823B dt=9123 wait=145 +08:50:34 pushed ckpt_w76.pt,log_w76.txt +08:51:35 waited 60s for global r=316 +08:51:41 global r=316 load 4.274 -> 4.281 +08:52:46 step 105320 loss=3.3274 lr=7.74e-05 tok=13.825B dt=9267 wait=145 +08:53:50 step 105340 loss=3.3856 lr=7.74e-05 tok=13.828B dt=9331 wait=146 +08:54:54 step 105360 loss=3.4149 lr=7.74e-05 tok=13.831B dt=9394 wait=146 +08:55:57 step 105380 loss=3.3858 lr=7.73e-05 tok=13.833B dt=9458 wait=146 +08:57:01 step 105400 loss=3.4047 lr=7.73e-05 tok=13.836B dt=9522 wait=147 +08:58:04 step 105420 loss=3.4507 lr=7.73e-05 tok=13.839B dt=9585 wait=147 +08:59:08 step 105440 loss=3.3773 lr=7.73e-05 tok=13.841B dt=9649 wait=147 +08:59:52 pushed ckpt_w76.pt,log_w76.txt +09:00:52 waited 61s for global r=317 +09:01:00 global r=317 load 4.309 -> 4.298 +09:01:34 step 105460 loss=3.5309 lr=7.72e-05 tok=13.844B dt=9795 wait=148 +09:02:37 step 105480 loss=3.3169 lr=7.72e-05 tok=13.846B dt=9858 wait=148 +09:03:41 step 105500 loss=3.4021 lr=7.72e-05 tok=13.849B dt=9922 wait=148 +09:04:45 step 105520 loss=3.4881 lr=7.72e-05 tok=13.852B dt=9985 wait=148 +09:05:48 step 105540 loss=3.3740 lr=7.72e-05 tok=13.854B dt=10049 wait=149 +09:06:52 step 105560 loss=3.4840 lr=7.71e-05 tok=13.857B dt=10113 wait=149 +09:07:56 step 105580 loss=3.3737 lr=7.71e-05 tok=13.860B dt=10177 wait=149 +09:09:00 step 105600 loss=3.3849 lr=7.71e-05 tok=13.862B dt=10241 wait=150 diff --git a/log_w77.txt b/log_w77.txt new file mode 100644 index 0000000000000000000000000000000000000000..3075e3a0f3bbfd3d0bf1ad1dfea36638d7f8b8a5 --- /dev/null +++ b/log_w77.txt @@ -0,0 +1,191 @@ +launch +06:16:14 name=w77 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 24/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +06:16:20 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +06:16:20 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +06:16:30 params 1.08B +06:16:31 resumed step=103050 seen=13.51B round=302 +06:16:31 stream salt=103050 +06:18:15 blocks compiled +06:18:15 start: accum=4, 131072 tokens per optimiser step +06:20:36 step 103051 loss=3.1125 lr=7.98e-05 tok=13.511B dt=141 wait=95 +06:20:39 step 103052 loss=3.1500 lr=7.98e-05 tok=13.511B dt=144 wait=95 +06:21:07 step 103060 loss=3.4464 lr=7.98e-05 tok=13.512B dt=172 wait=95 +06:22:17 step 103080 loss=3.3967 lr=7.98e-05 tok=13.515B dt=242 wait=95 +06:23:20 step 103100 loss=3.4211 lr=7.97e-05 tok=13.517B dt=305 wait=96 +06:24:23 step 103120 loss=3.4075 lr=7.97e-05 tok=13.520B dt=368 wait=96 +06:25:26 step 103140 loss=3.4361 lr=7.97e-05 tok=13.523B dt=432 wait=96 +06:26:30 step 103160 loss=3.5983 lr=7.97e-05 tok=13.525B dt=496 wait=97 +06:27:34 step 103180 loss=3.4090 lr=7.97e-05 tok=13.528B dt=559 wait=97 +06:28:38 step 103200 loss=3.4788 lr=7.96e-05 tok=13.531B dt=623 wait=97 +06:28:52 pushed ckpt_w77.pt,log_w77.txt +06:33:00 step 103220 loss=3.4310 lr=7.96e-05 tok=13.533B dt=885 wait=98 +06:34:04 step 103240 loss=3.4140 lr=7.96e-05 tok=13.536B dt=949 wait=98 +06:35:07 step 103260 loss=3.4120 lr=7.96e-05 tok=13.538B dt=1012 wait=98 +06:36:11 step 103280 loss=3.3933 lr=7.96e-05 tok=13.541B dt=1077 wait=99 +06:37:15 step 103300 loss=3.3740 lr=7.95e-05 tok=13.544B dt=1140 wait=99 +06:38:18 step 103320 loss=3.4394 lr=7.95e-05 tok=13.546B dt=1203 wait=99 +06:39:21 step 103340 loss=3.4664 lr=7.95e-05 tok=13.549B dt=1266 wait=99 +06:40:06 pushed ckpt_w77.pt,log_w77.txt +06:43:42 step 103360 loss=3.3344 lr=7.95e-05 tok=13.552B dt=1527 wait=100 +06:44:45 step 103380 loss=3.4380 lr=7.94e-05 tok=13.554B dt=1590 wait=100 +06:45:49 step 103400 loss=3.5382 lr=7.94e-05 tok=13.557B dt=1654 wait=100 +06:46:52 step 103420 loss=3.4385 lr=7.94e-05 tok=13.559B dt=1717 wait=101 +06:47:55 step 103440 loss=3.4115 lr=7.94e-05 tok=13.562B dt=1781 wait=101 +06:48:59 step 103460 loss=3.3609 lr=7.94e-05 tok=13.565B dt=1844 wait=101 +06:50:03 step 103480 loss=3.4393 lr=7.93e-05 tok=13.567B dt=1908 wait=102 +06:50:47 global r=303 load 4.284 -> 4.274 +06:51:18 step 103360 loss=3.3972 lr=7.95e-05 tok=13.570B dt=1983 wait=102 +06:52:22 step 103380 loss=3.3624 lr=7.94e-05 tok=13.573B dt=2047 wait=102 +06:53:25 step 103400 loss=3.3777 lr=7.94e-05 tok=13.575B dt=2110 wait=103 +06:54:28 step 103420 loss=3.4779 lr=7.94e-05 tok=13.578B dt=2173 wait=103 +06:55:32 step 103440 loss=3.3932 lr=7.94e-05 tok=13.580B dt=2237 wait=103 +06:56:35 step 103460 loss=3.4264 lr=7.94e-05 tok=13.583B dt=2301 wait=104 +06:57:39 step 103480 loss=3.3933 lr=7.93e-05 tok=13.586B dt=2364 wait=104 +06:58:43 step 103500 loss=3.4526 lr=7.93e-05 tok=13.588B dt=2428 wait=104 +06:58:55 pushed ckpt_w77.pt,log_w77.txt +06:59:41 waited 45s for global r=304 +06:59:47 global r=304 load 4.275 -> 4.273 +07:00:52 step 103520 loss=3.3267 lr=7.93e-05 tok=13.591B dt=2557 wait=104 +07:01:56 step 103540 loss=3.4596 lr=7.93e-05 tok=13.593B dt=2621 wait=105 +07:02:59 step 103560 loss=3.4448 lr=7.93e-05 tok=13.596B dt=2684 wait=105 +07:04:02 step 103580 loss=3.4085 lr=7.92e-05 tok=13.599B dt=2747 wait=105 +07:05:06 step 103600 loss=3.4146 lr=7.92e-05 tok=13.601B dt=2811 wait=106 +07:06:09 step 103620 loss=3.4751 lr=7.92e-05 tok=13.604B dt=2875 wait=106 +07:07:12 step 103640 loss=3.3489 lr=7.92e-05 tok=13.607B dt=2938 wait=106 +07:07:58 pushed ckpt_w77.pt,log_w77.txt +07:08:59 waited 61s for global r=305 +07:09:07 global r=305 load 4.304 -> 4.288 +07:09:40 step 103660 loss=3.4506 lr=7.92e-05 tok=13.609B dt=3085 wait=107 +07:10:44 step 103680 loss=3.3099 lr=7.91e-05 tok=13.612B dt=3150 wait=107 +07:11:48 step 103700 loss=3.3323 lr=7.91e-05 tok=13.614B dt=3213 wait=107 +07:12:52 step 103720 loss=3.4385 lr=7.91e-05 tok=13.617B dt=3277 wait=108 +07:13:55 step 103740 loss=3.3515 lr=7.91e-05 tok=13.620B dt=3340 wait=108 +07:14:59 step 103760 loss=3.4770 lr=7.90e-05 tok=13.622B dt=3404 wait=108 +07:16:02 step 103780 loss=3.3903 lr=7.90e-05 tok=13.625B dt=3467 wait=109 +07:17:05 step 103800 loss=3.4169 lr=7.90e-05 tok=13.628B dt=3530 wait=109 +07:17:18 pushed ckpt_w77.pt,log_w77.txt +07:18:18 waited 61s for global r=306 +07:18:26 global r=306 load 4.318 -> 4.286 +07:19:30 step 103820 loss=3.3870 lr=7.90e-05 tok=13.630B dt=3675 wait=109 +07:20:34 step 103840 loss=3.4305 lr=7.90e-05 tok=13.633B dt=3739 wait=110 +07:21:37 step 103860 loss=3.4098 lr=7.89e-05 tok=13.635B dt=3802 wait=110 +07:22:40 step 103880 loss=3.3944 lr=7.89e-05 tok=13.638B dt=3866 wait=110 +07:23:44 step 103900 loss=3.3551 lr=7.89e-05 tok=13.641B dt=3929 wait=110 +07:24:48 step 103920 loss=3.4985 lr=7.89e-05 tok=13.643B dt=3993 wait=111 +07:25:51 step 103940 loss=3.3473 lr=7.89e-05 tok=13.646B dt=4057 wait=111 +07:26:35 pushed ckpt_w77.pt,log_w77.txt +07:27:36 waited 60s for global r=307 +07:27:45 global r=307 load 4.354 -> 4.288 +07:28:18 step 103960 loss=3.4498 lr=7.88e-05 tok=13.649B dt=4203 wait=111 +07:29:21 step 103980 loss=3.3627 lr=7.88e-05 tok=13.651B dt=4266 wait=112 +07:30:25 step 104000 loss=3.3403 lr=7.88e-05 tok=13.654B dt=4330 wait=112 +07:31:29 step 104020 loss=3.3951 lr=7.88e-05 tok=13.656B dt=4394 wait=112 +07:32:32 step 104040 loss=3.3353 lr=7.87e-05 tok=13.659B dt=4457 wait=113 +07:33:36 step 104060 loss=3.4384 lr=7.87e-05 tok=13.662B dt=4521 wait=113 +07:34:40 step 104080 loss=3.4107 lr=7.87e-05 tok=13.664B dt=4585 wait=113 +07:35:44 step 104100 loss=3.3651 lr=7.87e-05 tok=13.667B dt=4649 wait=114 +07:36:01 pushed ckpt_w77.pt,log_w77.txt +07:37:01 waited 60s for global r=308 +07:37:08 global r=308 load 4.323 -> 4.280 +07:38:13 step 104120 loss=3.3104 lr=7.87e-05 tok=13.669B dt=4798 wait=114 +07:39:17 step 104140 loss=3.3486 lr=7.86e-05 tok=13.672B dt=4862 wait=114 +07:40:21 step 104160 loss=3.3801 lr=7.86e-05 tok=13.675B dt=4926 wait=115 +07:41:24 step 104180 loss=3.3444 lr=7.86e-05 tok=13.677B dt=4989 wait=115 +07:42:27 step 104200 loss=3.4046 lr=7.86e-05 tok=13.680B dt=5053 wait=115 +07:43:31 step 104220 loss=3.4545 lr=7.86e-05 tok=13.683B dt=5116 wait=116 +07:44:34 step 104240 loss=3.3479 lr=7.85e-05 tok=13.685B dt=5179 wait=116 +07:45:21 pushed ckpt_w77.pt,log_w77.txt +07:46:21 waited 60s for global r=309 +07:46:28 global r=309 load 4.305 -> 4.285 +07:47:01 step 104260 loss=3.4450 lr=7.85e-05 tok=13.688B dt=5326 wait=116 +07:48:05 step 104280 loss=3.3616 lr=7.85e-05 tok=13.690B dt=5390 wait=117 +07:49:08 step 104300 loss=3.3251 lr=7.85e-05 tok=13.693B dt=5453 wait=117 +07:50:12 step 104320 loss=3.4384 lr=7.85e-05 tok=13.696B dt=5517 wait=117 +07:51:16 step 104340 loss=3.3490 lr=7.84e-05 tok=13.698B dt=5581 wait=118 +07:52:19 step 104360 loss=3.4359 lr=7.84e-05 tok=13.701B dt=5644 wait=118 +07:53:23 step 104380 loss=3.3842 lr=7.84e-05 tok=13.704B dt=5708 wait=118 +07:54:27 step 104400 loss=3.3717 lr=7.84e-05 tok=13.706B dt=5772 wait=119 +07:54:39 pushed ckpt_w77.pt,log_w77.txt +07:55:40 waited 61s for global r=310 +07:55:45 global r=310 load 4.285 -> 4.292 +07:56:49 step 104420 loss=3.3345 lr=7.83e-05 tok=13.709B dt=5915 wait=119 +07:57:53 step 104440 loss=3.3873 lr=7.83e-05 tok=13.711B dt=5978 wait=119 +07:58:56 step 104460 loss=3.4053 lr=7.83e-05 tok=13.714B dt=6042 wait=120 +08:00:00 step 104480 loss=3.3739 lr=7.83e-05 tok=13.717B dt=6105 wait=120 +08:01:04 step 104500 loss=3.3750 lr=7.83e-05 tok=13.719B dt=6169 wait=120 +08:02:07 step 104520 loss=3.4858 lr=7.82e-05 tok=13.722B dt=6232 wait=121 +08:03:11 step 104540 loss=3.3086 lr=7.82e-05 tok=13.725B dt=6296 wait=121 +08:03:56 pushed ckpt_w77.pt,log_w77.txt +08:04:56 waited 61s for global r=311 +08:05:04 global r=311 load 4.288 -> 4.290 +08:05:36 step 104560 loss=3.4477 lr=7.82e-05 tok=13.727B dt=6442 wait=121 +08:06:40 step 104580 loss=3.3116 lr=7.82e-05 tok=13.730B dt=6505 wait=122 +08:07:44 step 104600 loss=3.3884 lr=7.82e-05 tok=13.732B dt=6569 wait=122 +08:08:47 step 104620 loss=3.4352 lr=7.81e-05 tok=13.735B dt=6632 wait=122 +08:09:51 step 104640 loss=3.2821 lr=7.81e-05 tok=13.738B dt=6696 wait=123 +08:10:55 step 104660 loss=3.4083 lr=7.81e-05 tok=13.740B dt=6760 wait=123 +08:11:59 step 104680 loss=3.3132 lr=7.81e-05 tok=13.743B dt=6824 wait=124 +08:13:03 step 104700 loss=3.4201 lr=7.81e-05 tok=13.746B dt=6888 wait=124 +08:13:16 pushed ckpt_w77.pt,log_w77.txt +08:14:17 waited 61s for global r=312 +08:14:22 global r=312 load 4.310 -> 4.287 +08:15:26 step 104720 loss=3.3476 lr=7.80e-05 tok=13.748B dt=7032 wait=124 +08:16:30 step 104740 loss=3.4161 lr=7.80e-05 tok=13.751B dt=7095 wait=125 +08:17:34 step 104760 loss=3.4264 lr=7.80e-05 tok=13.753B dt=7159 wait=125 +08:18:37 step 104780 loss=3.3553 lr=7.80e-05 tok=13.756B dt=7222 wait=125 +08:19:41 step 104800 loss=3.4017 lr=7.79e-05 tok=13.759B dt=7286 wait=126 +08:20:45 step 104820 loss=3.4892 lr=7.79e-05 tok=13.761B dt=7350 wait=126 +08:21:48 step 104840 loss=3.3365 lr=7.79e-05 tok=13.764B dt=7414 wait=126 +08:22:33 pushed ckpt_w77.pt,log_w77.txt +08:23:33 waited 60s for global r=313 +08:23:38 global r=313 load 4.307 -> 4.303 +08:24:11 step 104860 loss=3.4647 lr=7.79e-05 tok=13.766B dt=7556 wait=127 +08:25:14 step 104880 loss=3.3302 lr=7.79e-05 tok=13.769B dt=7619 wait=127 +08:26:18 step 104900 loss=3.3580 lr=7.78e-05 tok=13.772B dt=7683 wait=127 +08:27:21 step 104920 loss=3.4296 lr=7.78e-05 tok=13.774B dt=7746 wait=128 +08:28:25 step 104940 loss=3.3302 lr=7.78e-05 tok=13.777B dt=7810 wait=128 +08:29:29 step 104960 loss=3.3490 lr=7.78e-05 tok=13.780B dt=7874 wait=129 +08:30:32 step 104980 loss=3.3768 lr=7.78e-05 tok=13.782B dt=7937 wait=129 +08:31:36 step 105000 loss=3.4463 lr=7.77e-05 tok=13.785B dt=8001 wait=129 +08:31:49 pushed ckpt_w77.pt,log_w77.txt +08:32:49 waited 61s for global r=314 +08:32:55 global r=314 load 4.310 -> 4.287 +08:34:00 step 105020 loss=3.4072 lr=7.77e-05 tok=13.787B dt=8145 wait=130 +08:35:04 step 105040 loss=3.4015 lr=7.77e-05 tok=13.790B dt=8209 wait=130 +08:36:08 step 105060 loss=3.3876 lr=7.77e-05 tok=13.793B dt=8273 wait=130 +08:37:12 step 105080 loss=3.3746 lr=7.76e-05 tok=13.795B dt=8337 wait=131 +08:38:15 step 105100 loss=3.3970 lr=7.76e-05 tok=13.798B dt=8400 wait=131 +08:39:19 step 105120 loss=3.4441 lr=7.76e-05 tok=13.801B dt=8464 wait=131 +08:40:23 step 105140 loss=3.3477 lr=7.76e-05 tok=13.803B dt=8528 wait=132 +08:41:10 pushed ckpt_w77.pt,log_w77.txt +08:42:11 waited 60s for global r=315 +08:42:17 global r=315 load 4.313 -> 4.284 +08:42:50 step 105160 loss=3.4374 lr=7.76e-05 tok=13.806B dt=8675 wait=132 +08:43:53 step 105180 loss=3.2977 lr=7.75e-05 tok=13.808B dt=8738 wait=132 +08:44:57 step 105200 loss=3.3260 lr=7.75e-05 tok=13.811B dt=8802 wait=133 +08:46:01 step 105220 loss=3.3992 lr=7.75e-05 tok=13.814B dt=8866 wait=133 +08:47:04 step 105240 loss=3.3240 lr=7.75e-05 tok=13.816B dt=8929 wait=133 +08:48:07 step 105260 loss=3.4239 lr=7.75e-05 tok=13.819B dt=8992 wait=134 +08:49:11 step 105280 loss=3.3503 lr=7.74e-05 tok=13.822B dt=9056 wait=134 +08:50:14 step 105300 loss=3.4025 lr=7.74e-05 tok=13.824B dt=9119 wait=135 +08:50:28 pushed ckpt_w77.pt,log_w77.txt +08:51:28 waited 60s for global r=316 +08:51:33 global r=316 load 4.310 -> 4.281 +08:52:38 step 105320 loss=3.3511 lr=7.74e-05 tok=13.827B dt=9263 wait=135 +08:53:41 step 105340 loss=3.4362 lr=7.74e-05 tok=13.829B dt=9326 wait=135 +08:54:45 step 105360 loss=3.3654 lr=7.74e-05 tok=13.832B dt=9390 wait=136 +08:55:48 step 105380 loss=3.3370 lr=7.73e-05 tok=13.835B dt=9453 wait=136 +08:56:52 step 105400 loss=3.3650 lr=7.73e-05 tok=13.837B dt=9517 wait=136 +08:57:56 step 105420 loss=3.4986 lr=7.73e-05 tok=13.840B dt=9581 wait=137 +08:58:59 step 105440 loss=3.3486 lr=7.73e-05 tok=13.843B dt=9645 wait=137 +08:59:45 pushed ckpt_w77.pt,log_w77.txt +09:00:46 waited 61s for global r=317 +09:00:50 global r=317 load 4.304 -> 4.298 +09:01:23 step 105460 loss=3.4530 lr=7.72e-05 tok=13.845B dt=9788 wait=137 +09:02:27 step 105480 loss=3.2696 lr=7.72e-05 tok=13.848B dt=9853 wait=138 +09:03:31 step 105500 loss=3.3350 lr=7.72e-05 tok=13.850B dt=9916 wait=138 +09:04:35 step 105520 loss=3.3657 lr=7.72e-05 tok=13.853B dt=9980 wait=138 +09:05:38 step 105540 loss=3.3371 lr=7.72e-05 tok=13.856B dt=10043 wait=139 +09:06:42 step 105560 loss=3.4092 lr=7.71e-05 tok=13.858B dt=10107 wait=139 +09:07:45 step 105580 loss=3.3519 lr=7.71e-05 tok=13.861B dt=10170 wait=139 +09:08:49 step 105600 loss=3.3801 lr=7.71e-05 tok=13.863B dt=10234 wait=140 diff --git a/log_w78.txt b/log_w78.txt new file mode 100644 index 0000000000000000000000000000000000000000..8e29832f27944855a8571f27416ec2842088f7be --- /dev/null +++ b/log_w78.txt @@ -0,0 +1,175 @@ +launch +12:00:38 name=w78 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 25/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:00:44 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:00:44 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:00:53 params 1.08B +12:00:55 resumed step=106350 seen=13.96B round=319 +12:00:55 stream salt=106350 +12:03:00 blocks compiled +12:03:00 start: accum=4, 131072 tokens per optimiser step +12:06:01 step 106351 loss=3.1093 lr=7.63e-05 tok=13.961B dt=181 wait=144 +12:06:04 step 106352 loss=3.1787 lr=7.63e-05 tok=13.961B dt=184 wait=145 +12:06:32 step 106360 loss=3.5038 lr=7.63e-05 tok=13.962B dt=212 wait=145 +12:07:39 step 106380 loss=3.3567 lr=7.63e-05 tok=13.964B dt=279 wait=145 +12:08:43 step 106400 loss=3.3920 lr=7.63e-05 tok=13.967B dt=343 wait=146 +12:09:47 step 106420 loss=3.3290 lr=7.62e-05 tok=13.970B dt=406 wait=146 +12:10:50 step 106440 loss=3.4233 lr=7.62e-05 tok=13.972B dt=470 wait=146 +12:11:54 step 106460 loss=3.5316 lr=7.62e-05 tok=13.975B dt=533 wait=147 +12:12:57 step 106480 loss=3.4617 lr=7.62e-05 tok=13.978B dt=597 wait=147 +12:14:00 step 106500 loss=3.4598 lr=7.61e-05 tok=13.980B dt=660 wait=147 +12:14:13 pushed ckpt_w78.pt,log_w78.txt +12:15:35 waited 81s for global r=320 +12:16:03 global r=320 load 4.368 -> 4.302 +12:17:10 step 106520 loss=3.3415 lr=7.61e-05 tok=13.983B dt=849 wait=148 +12:18:13 step 106540 loss=3.4246 lr=7.61e-05 tok=13.985B dt=913 wait=148 +12:19:16 step 106560 loss=3.4009 lr=7.61e-05 tok=13.988B dt=976 wait=148 +12:20:19 step 106580 loss=3.2792 lr=7.61e-05 tok=13.991B dt=1039 wait=149 +12:21:22 step 106600 loss=3.3545 lr=7.60e-05 tok=13.993B dt=1102 wait=149 +12:22:26 step 106620 loss=3.3653 lr=7.60e-05 tok=13.996B dt=1166 wait=149 +12:23:29 step 106640 loss=3.3265 lr=7.60e-05 tok=13.998B dt=1229 wait=149 +12:24:14 pushed ckpt_w78.pt,log_w78.txt +12:27:50 step 106660 loss=3.4257 lr=7.60e-05 tok=14.001B dt=1490 wait=150 +12:27:59 global r=321 load 4.288 -> 4.304 +12:28:30 step 106660 loss=3.4552 lr=7.60e-05 tok=14.002B dt=1530 wait=150 +12:29:33 step 106680 loss=3.3084 lr=7.60e-05 tok=14.005B dt=1593 wait=150 +12:30:36 step 106700 loss=3.3193 lr=7.59e-05 tok=14.008B dt=1656 wait=151 +12:31:39 step 106720 loss=3.4399 lr=7.59e-05 tok=14.010B dt=1719 wait=151 +12:32:42 step 106740 loss=3.3677 lr=7.59e-05 tok=14.013B dt=1782 wait=151 +12:33:45 step 106760 loss=3.4242 lr=7.59e-05 tok=14.016B dt=1845 wait=152 +12:34:48 step 106780 loss=3.3495 lr=7.59e-05 tok=14.018B dt=1908 wait=152 +12:35:52 step 106800 loss=3.4267 lr=7.58e-05 tok=14.021B dt=1971 wait=152 +12:36:04 pushed ckpt_w78.pt,log_w78.txt +12:39:05 waited 181s for global r=322 +12:39:14 global r=322 load 4.321 -> 4.303 +12:40:19 step 106820 loss=3.3712 lr=7.58e-05 tok=14.023B dt=2238 wait=153 +12:41:22 step 106840 loss=3.3735 lr=7.58e-05 tok=14.026B dt=2302 wait=153 +12:42:25 step 106860 loss=3.4222 lr=7.58e-05 tok=14.029B dt=2365 wait=153 +12:43:28 step 106880 loss=3.3241 lr=7.57e-05 tok=14.031B dt=2428 wait=154 +12:44:31 step 106900 loss=3.3376 lr=7.57e-05 tok=14.034B dt=2491 wait=154 +12:45:35 step 106920 loss=3.4026 lr=7.57e-05 tok=14.037B dt=2554 wait=154 +12:46:38 step 106940 loss=3.3046 lr=7.57e-05 tok=14.039B dt=2618 wait=155 +12:47:24 pushed ckpt_w78.pt,log_w78.txt +12:50:59 step 106960 loss=3.4736 lr=7.57e-05 tok=14.042B dt=2879 wait=155 +12:51:08 global r=323 load 4.324 -> 4.300 +12:51:40 step 106960 loss=3.4162 lr=7.57e-05 tok=14.043B dt=2920 wait=155 +12:52:43 step 106980 loss=3.3222 lr=7.56e-05 tok=14.046B dt=2983 wait=156 +12:53:46 step 107000 loss=3.3323 lr=7.56e-05 tok=14.048B dt=3046 wait=156 +12:54:49 step 107020 loss=3.4301 lr=7.56e-05 tok=14.051B dt=3109 wait=156 +12:55:52 step 107040 loss=3.3313 lr=7.56e-05 tok=14.054B dt=3172 wait=157 +12:56:56 step 107060 loss=3.4519 lr=7.56e-05 tok=14.056B dt=3236 wait=157 +12:57:59 step 107080 loss=3.3739 lr=7.55e-05 tok=14.059B dt=3299 wait=157 +12:59:02 step 107100 loss=3.4681 lr=7.55e-05 tok=14.061B dt=3362 wait=158 +12:59:15 pushed ckpt_w78.pt,log_w78.txt +13:03:02 global r=324 load 4.297 -> 4.306 +13:04:05 step 107120 loss=3.3931 lr=7.55e-05 tok=14.065B dt=3665 wait=158 +13:05:08 step 107140 loss=3.3826 lr=7.55e-05 tok=14.068B dt=3728 wait=158 +13:06:11 step 107160 loss=3.3730 lr=7.55e-05 tok=14.071B dt=3791 wait=159 +13:07:15 step 107180 loss=3.3996 lr=7.54e-05 tok=14.073B dt=3855 wait=159 +13:08:18 step 107200 loss=3.3375 lr=7.54e-05 tok=14.076B dt=3918 wait=159 +13:09:21 step 107220 loss=3.3965 lr=7.54e-05 tok=14.078B dt=3981 wait=160 +13:10:24 step 107240 loss=3.3427 lr=7.54e-05 tok=14.081B dt=4044 wait=160 +13:11:08 pushed ckpt_w78.pt,log_w78.txt +13:14:47 step 107260 loss=3.4709 lr=7.53e-05 tok=14.084B dt=4307 wait=160 +13:14:57 global r=325 load 4.328 -> 4.291 +13:15:29 step 107260 loss=3.4534 lr=7.53e-05 tok=14.085B dt=4348 wait=160 +13:16:32 step 107280 loss=3.3044 lr=7.53e-05 tok=14.088B dt=4412 wait=161 +13:17:35 step 107300 loss=3.3550 lr=7.53e-05 tok=14.090B dt=4475 wait=161 +13:18:38 step 107320 loss=3.4374 lr=7.53e-05 tok=14.093B dt=4538 wait=161 +13:19:41 step 107340 loss=3.3387 lr=7.53e-05 tok=14.095B dt=4601 wait=162 +13:20:45 step 107360 loss=3.4548 lr=7.52e-05 tok=14.098B dt=4665 wait=162 +13:21:48 step 107380 loss=3.2979 lr=7.52e-05 tok=14.101B dt=4728 wait=162 +13:22:51 step 107400 loss=3.4751 lr=7.52e-05 tok=14.103B dt=4791 wait=163 +13:23:04 pushed ckpt_w78.pt,log_w78.txt +13:24:05 waited 61s for global r=326 +13:24:13 global r=326 load 4.302 -> 4.288 +13:25:17 step 107420 loss=3.3810 lr=7.52e-05 tok=14.106B dt=4937 wait=163 +13:26:20 step 107440 loss=3.4250 lr=7.52e-05 tok=14.109B dt=5000 wait=163 +13:27:24 step 107460 loss=3.3832 lr=7.51e-05 tok=14.111B dt=5064 wait=164 +13:28:27 step 107480 loss=3.3177 lr=7.51e-05 tok=14.114B dt=5127 wait=164 +13:29:30 step 107500 loss=3.3459 lr=7.51e-05 tok=14.116B dt=5190 wait=164 +13:30:33 step 107520 loss=3.4768 lr=7.51e-05 tok=14.119B dt=5253 wait=165 +13:31:36 step 107540 loss=3.2982 lr=7.51e-05 tok=14.122B dt=5316 wait=165 +13:32:22 pushed ckpt_w78.pt,log_w78.txt +13:33:23 waited 61s for global r=327 +13:33:36 global r=327 load 4.282 -> 4.291 +13:34:09 step 107560 loss=3.4227 lr=7.50e-05 tok=14.124B dt=5469 wait=165 +13:35:12 step 107580 loss=3.3379 lr=7.50e-05 tok=14.127B dt=5532 wait=166 +13:36:15 step 107600 loss=3.3535 lr=7.50e-05 tok=14.130B dt=5595 wait=166 +13:37:19 step 107620 loss=3.3654 lr=7.50e-05 tok=14.132B dt=5659 wait=166 +13:38:22 step 107640 loss=3.3152 lr=7.49e-05 tok=14.135B dt=5722 wait=167 +13:39:26 step 107660 loss=3.3448 lr=7.49e-05 tok=14.137B dt=5785 wait=167 +13:40:29 step 107680 loss=3.3618 lr=7.49e-05 tok=14.140B dt=5849 wait=167 +13:41:32 step 107700 loss=3.4092 lr=7.49e-05 tok=14.143B dt=5912 wait=168 +13:41:48 pushed ckpt_w78.pt,log_w78.txt +13:42:48 waited 61s for global r=328 +13:42:57 global r=328 load 4.343 -> 4.315 +13:44:02 step 107720 loss=3.3165 lr=7.49e-05 tok=14.145B dt=6062 wait=168 +13:45:05 step 107740 loss=3.3963 lr=7.48e-05 tok=14.148B dt=6125 wait=168 +13:46:08 step 107760 loss=3.4369 lr=7.48e-05 tok=14.151B dt=6188 wait=169 +13:47:12 step 107780 loss=3.3456 lr=7.48e-05 tok=14.153B dt=6252 wait=169 +13:48:16 step 107800 loss=3.3398 lr=7.48e-05 tok=14.156B dt=6316 wait=169 +13:49:20 step 107820 loss=3.3303 lr=7.48e-05 tok=14.158B dt=6380 wait=170 +13:50:25 step 107840 loss=2.9449 lr=7.47e-05 tok=14.161B dt=6445 wait=170 +13:51:09 pushed ckpt_w78.pt,log_w78.txt +13:52:11 waited 62s for global r=329 +13:52:23 global r=329 load 4.341 -> 4.297 +13:52:56 step 107860 loss=3.4674 lr=7.47e-05 tok=14.164B dt=6596 wait=170 +13:53:59 step 107880 loss=3.3178 lr=7.47e-05 tok=14.166B dt=6659 wait=170 +13:55:02 step 107900 loss=3.3418 lr=7.47e-05 tok=14.169B dt=6722 wait=171 +13:56:06 step 107920 loss=3.4305 lr=7.47e-05 tok=14.172B dt=6786 wait=171 +13:57:09 step 107940 loss=3.3196 lr=7.46e-05 tok=14.174B dt=6849 wait=171 +13:58:12 step 107960 loss=3.4161 lr=7.46e-05 tok=14.177B dt=6912 wait=172 +13:59:15 step 107980 loss=3.3695 lr=7.46e-05 tok=14.179B dt=6975 wait=172 +14:00:18 step 108000 loss=3.3734 lr=7.46e-05 tok=14.182B dt=7038 wait=172 +14:00:30 pushed ckpt_w78.pt,log_w78.txt +14:01:46 waited 76s for global r=330 +14:02:01 global r=330 load 4.298 -> 4.286 +14:03:06 step 108020 loss=3.3064 lr=7.45e-05 tok=14.185B dt=7206 wait=173 +14:04:09 step 108040 loss=3.3243 lr=7.45e-05 tok=14.187B dt=7269 wait=173 +14:05:12 step 108060 loss=3.4032 lr=7.45e-05 tok=14.190B dt=7332 wait=173 +14:06:16 step 108080 loss=3.3487 lr=7.45e-05 tok=14.192B dt=7396 wait=174 +14:07:19 step 108100 loss=3.3802 lr=7.45e-05 tok=14.195B dt=7459 wait=174 +14:08:22 step 108120 loss=3.3994 lr=7.44e-05 tok=14.198B dt=7522 wait=174 +14:09:26 step 108140 loss=3.3397 lr=7.44e-05 tok=14.200B dt=7585 wait=175 +14:10:10 pushed ckpt_w78.pt,log_w78.txt +14:11:10 waited 61s for global r=331 +14:11:19 global r=331 load 4.339 -> 4.301 +14:11:53 step 108160 loss=3.4366 lr=7.44e-05 tok=14.203B dt=7733 wait=175 +14:12:56 step 108180 loss=3.3356 lr=7.44e-05 tok=14.206B dt=7796 wait=175 +14:13:59 step 108200 loss=3.3624 lr=7.44e-05 tok=14.208B dt=7859 wait=176 +14:15:02 step 108220 loss=3.4688 lr=7.43e-05 tok=14.211B dt=7922 wait=176 +14:16:05 step 108240 loss=3.3133 lr=7.43e-05 tok=14.213B dt=7985 wait=176 +14:17:09 step 108260 loss=3.4233 lr=7.43e-05 tok=14.216B dt=8048 wait=177 +14:18:11 step 108280 loss=3.3664 lr=7.43e-05 tok=14.219B dt=8111 wait=177 +14:19:15 step 108300 loss=3.3261 lr=7.43e-05 tok=14.221B dt=8175 wait=177 +14:19:29 pushed ckpt_w78.pt,log_w78.txt +14:20:45 waited 76s for global r=332 +14:20:53 global r=332 load 4.331 -> 4.292 +14:21:58 step 108320 loss=3.3773 lr=7.42e-05 tok=14.224B dt=8338 wait=178 +14:23:02 step 108340 loss=3.2622 lr=7.42e-05 tok=14.227B dt=8402 wait=178 +14:24:05 step 108360 loss=3.3754 lr=7.42e-05 tok=14.229B dt=8465 wait=178 +14:25:08 step 108380 loss=3.3258 lr=7.42e-05 tok=14.232B dt=8528 wait=179 +14:26:11 step 108400 loss=3.3909 lr=7.41e-05 tok=14.234B dt=8591 wait=179 +14:27:15 step 108420 loss=3.4847 lr=7.41e-05 tok=14.237B dt=8655 wait=179 +14:28:18 step 108440 loss=3.2496 lr=7.41e-05 tok=14.240B dt=8718 wait=180 +14:29:01 pushed ckpt_w78.pt,log_w78.txt +14:30:17 waited 76s for global r=333 +14:30:28 global r=333 load 4.305 -> 4.294 +14:31:01 step 108460 loss=3.4100 lr=7.41e-05 tok=14.242B dt=8881 wait=180 +14:32:04 step 108480 loss=3.3414 lr=7.41e-05 tok=14.245B dt=8944 wait=180 +14:33:07 step 108500 loss=3.3387 lr=7.40e-05 tok=14.248B dt=9007 wait=181 +14:34:10 step 108520 loss=3.4498 lr=7.40e-05 tok=14.250B dt=9070 wait=181 +14:35:14 step 108540 loss=3.2718 lr=7.40e-05 tok=14.253B dt=9133 wait=181 +14:36:17 step 108560 loss=3.4168 lr=7.40e-05 tok=14.255B dt=9196 wait=182 +14:37:20 step 108580 loss=3.3377 lr=7.40e-05 tok=14.258B dt=9260 wait=182 +14:38:24 step 108600 loss=3.4191 lr=7.39e-05 tok=14.261B dt=9324 wait=182 +14:38:36 pushed ckpt_w78.pt,log_w78.txt +14:39:53 waited 76s for global r=334 +14:40:02 global r=334 load 4.303 -> 4.295 +14:41:07 step 108620 loss=3.3564 lr=7.39e-05 tok=14.263B dt=9487 wait=183 +14:42:10 step 108640 loss=3.3961 lr=7.39e-05 tok=14.266B dt=9550 wait=183 +14:43:13 step 108660 loss=3.3584 lr=7.39e-05 tok=14.268B dt=9613 wait=183 +14:44:17 step 108680 loss=3.3324 lr=7.39e-05 tok=14.271B dt=9677 wait=183 +14:45:20 step 108700 loss=3.3305 lr=7.38e-05 tok=14.274B dt=9740 wait=184 +14:46:23 step 108720 loss=3.4394 lr=7.38e-05 tok=14.276B dt=9803 wait=184 +14:47:26 step 108740 loss=3.3003 lr=7.38e-05 tok=14.279B dt=9866 wait=184 diff --git a/log_w79.txt b/log_w79.txt new file mode 100644 index 0000000000000000000000000000000000000000..5608ba2ef99b710181c2f6cd6bcf009a47f790f1 --- /dev/null +++ b/log_w79.txt @@ -0,0 +1,374 @@ +launch +12:00:41 name=w79 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 26/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:00:47 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:00:47 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:00:57 params 1.08B +12:00:58 resumed step=106350 seen=13.96B round=319 +12:00:58 stream salt=106350 +12:03:04 blocks compiled +12:03:04 start: accum=4, 131072 tokens per optimiser step +12:06:34 step 106351 loss=3.1077 lr=7.63e-05 tok=13.961B dt=210 wait=169 +12:06:37 step 106352 loss=3.1207 lr=7.63e-05 tok=13.961B dt=214 wait=169 +12:07:04 step 106360 loss=3.5326 lr=7.63e-05 tok=13.962B dt=241 wait=170 +12:08:12 step 106380 loss=3.3790 lr=7.63e-05 tok=13.964B dt=308 wait=170 +12:09:17 step 106400 loss=3.4101 lr=7.63e-05 tok=13.967B dt=373 wait=170 +12:10:20 step 106420 loss=3.4133 lr=7.62e-05 tok=13.970B dt=437 wait=171 +12:11:24 step 106440 loss=3.4476 lr=7.62e-05 tok=13.972B dt=500 wait=171 +12:12:27 step 106460 loss=3.5924 lr=7.62e-05 tok=13.975B dt=564 wait=172 +12:13:31 step 106480 loss=3.3583 lr=7.62e-05 tok=13.978B dt=627 wait=172 +12:14:34 step 106500 loss=3.4801 lr=7.61e-05 tok=13.980B dt=691 wait=172 +12:14:50 pushed ckpt_w79.pt,log_w79.txt +12:15:36 waited 45s for global r=320 +12:15:56 global r=320 load 4.330 -> 4.302 +12:17:03 step 106520 loss=3.2758 lr=7.61e-05 tok=13.983B dt=839 wait=172 +12:18:06 step 106540 loss=3.4388 lr=7.61e-05 tok=13.985B dt=902 wait=173 +12:19:09 step 106560 loss=3.4045 lr=7.61e-05 tok=13.988B dt=966 wait=173 +12:20:13 step 106580 loss=3.3598 lr=7.61e-05 tok=13.991B dt=1029 wait=174 +12:21:15 step 106600 loss=3.4054 lr=7.60e-05 tok=13.993B dt=1092 wait=174 +12:22:19 step 106620 loss=3.4563 lr=7.60e-05 tok=13.996B dt=1155 wait=174 +12:23:22 step 106640 loss=3.3552 lr=7.60e-05 tok=13.998B dt=1218 wait=174 +12:24:07 pushed ckpt_w79.pt,log_w79.txt +12:27:43 step 106660 loss=3.4814 lr=7.60e-05 tok=14.001B dt=1479 wait=175 +12:27:49 global r=321 load 4.316 -> 4.304 +12:28:21 step 106660 loss=3.4371 lr=7.60e-05 tok=14.002B dt=1518 wait=175 +12:29:24 step 106680 loss=3.3285 lr=7.60e-05 tok=14.005B dt=1581 wait=175 +12:30:27 step 106700 loss=3.3483 lr=7.59e-05 tok=14.008B dt=1644 wait=175 +12:31:31 step 106720 loss=3.4234 lr=7.59e-05 tok=14.010B dt=1707 wait=176 +12:32:34 step 106740 loss=3.3650 lr=7.59e-05 tok=14.013B dt=1770 wait=176 +12:33:38 step 106760 loss=3.4549 lr=7.59e-05 tok=14.016B dt=1834 wait=176 +12:34:41 step 106780 loss=3.3727 lr=7.59e-05 tok=14.018B dt=1897 wait=177 +12:35:44 step 106800 loss=3.3850 lr=7.58e-05 tok=14.021B dt=1961 wait=177 +12:35:56 pushed ckpt_w79.pt,log_w79.txt +12:39:39 global r=322 load 4.313 -> 4.303 +12:40:42 step 106820 loss=3.3845 lr=7.58e-05 tok=14.025B dt=2258 wait=177 +12:41:45 step 106840 loss=3.4466 lr=7.58e-05 tok=14.027B dt=2322 wait=178 +12:42:48 step 106860 loss=3.4169 lr=7.58e-05 tok=14.030B dt=2385 wait=178 +12:43:52 step 106880 loss=3.3677 lr=7.57e-05 tok=14.033B dt=2448 wait=178 +12:44:55 step 106900 loss=3.3655 lr=7.57e-05 tok=14.035B dt=2512 wait=179 +12:45:59 step 106920 loss=3.4917 lr=7.57e-05 tok=14.038B dt=2575 wait=179 +12:47:02 step 106940 loss=3.3288 lr=7.57e-05 tok=14.040B dt=2638 wait=179 +12:47:48 pushed ckpt_w79.pt,log_w79.txt +12:50:49 waited 181s for global r=323 +12:50:57 global r=323 load 4.289 -> 4.300 +12:51:30 step 106960 loss=3.4465 lr=7.57e-05 tok=14.043B dt=2907 wait=180 +12:52:34 step 106980 loss=3.3251 lr=7.56e-05 tok=14.046B dt=2970 wait=180 +12:53:38 step 107000 loss=3.3748 lr=7.56e-05 tok=14.048B dt=3034 wait=180 +12:54:41 step 107020 loss=3.4529 lr=7.56e-05 tok=14.051B dt=3097 wait=181 +12:55:44 step 107040 loss=3.3098 lr=7.56e-05 tok=14.054B dt=3161 wait=181 +12:56:48 step 107060 loss=3.4891 lr=7.56e-05 tok=14.056B dt=3224 wait=181 +12:57:56 step 107080 loss=3.3651 lr=7.55e-05 tok=14.059B dt=3292 wait=181 +12:58:59 step 107100 loss=3.4066 lr=7.55e-05 tok=14.061B dt=3356 wait=182 +12:59:13 pushed ckpt_w79.pt,log_w79.txt +13:03:00 global r=324 load 4.316 -> 4.306 +13:04:03 step 107120 loss=3.3372 lr=7.55e-05 tok=14.065B dt=3659 wait=182 +13:05:06 step 107140 loss=3.3690 lr=7.55e-05 tok=14.068B dt=3723 wait=183 +13:06:11 step 107160 loss=3.2991 lr=7.55e-05 tok=14.071B dt=3787 wait=183 +13:07:14 step 107180 loss=3.4778 lr=7.54e-05 tok=14.073B dt=3851 wait=183 +13:08:17 step 107200 loss=3.3923 lr=7.54e-05 tok=14.076B dt=3914 wait=183 +13:09:21 step 107220 loss=3.4204 lr=7.54e-05 tok=14.078B dt=3977 wait=184 +13:10:24 step 107240 loss=3.3382 lr=7.54e-05 tok=14.081B dt=4041 wait=184 +13:11:08 pushed ckpt_w79.pt,log_w79.txt +13:14:41 step 107260 loss=3.5308 lr=7.53e-05 tok=14.084B dt=4297 wait=184 +13:14:49 global r=325 load 4.323 -> 4.291 +13:15:20 step 107260 loss=3.4889 lr=7.53e-05 tok=14.085B dt=4337 wait=185 +13:16:23 step 107280 loss=3.3270 lr=7.53e-05 tok=14.088B dt=4400 wait=185 +13:17:27 step 107300 loss=3.3552 lr=7.53e-05 tok=14.090B dt=4463 wait=185 +13:18:30 step 107320 loss=3.4242 lr=7.53e-05 tok=14.093B dt=4526 wait=186 +13:19:33 step 107340 loss=3.3218 lr=7.53e-05 tok=14.095B dt=4590 wait=186 +13:20:37 step 107360 loss=3.4234 lr=7.52e-05 tok=14.098B dt=4653 wait=186 +13:21:40 step 107380 loss=3.3662 lr=7.52e-05 tok=14.101B dt=4716 wait=187 +13:22:43 step 107400 loss=3.4098 lr=7.52e-05 tok=14.103B dt=4780 wait=187 +13:22:57 pushed ckpt_w79.pt,log_w79.txt +13:23:58 waited 61s for global r=326 +13:24:05 global r=326 load 4.304 -> 4.288 +13:25:09 step 107420 loss=3.3661 lr=7.52e-05 tok=14.106B dt=4926 wait=187 +13:26:13 step 107440 loss=3.4092 lr=7.52e-05 tok=14.109B dt=4989 wait=188 +13:27:17 step 107460 loss=3.3606 lr=7.51e-05 tok=14.111B dt=5053 wait=188 +13:28:20 step 107480 loss=3.3409 lr=7.51e-05 tok=14.114B dt=5117 wait=188 +13:29:24 step 107500 loss=3.3538 lr=7.51e-05 tok=14.116B dt=5180 wait=189 +13:30:27 step 107520 loss=3.4606 lr=7.51e-05 tok=14.119B dt=5244 wait=189 +13:31:30 step 107540 loss=3.3419 lr=7.51e-05 tok=14.122B dt=5307 wait=189 +13:32:16 pushed ckpt_w79.pt,log_w79.txt +13:33:17 waited 60s for global r=327 +13:33:25 global r=327 load 4.296 -> 4.291 +13:33:57 step 107560 loss=3.4264 lr=7.50e-05 tok=14.124B dt=5453 wait=190 +13:35:00 step 107580 loss=3.3215 lr=7.50e-05 tok=14.127B dt=5516 wait=190 +13:36:03 step 107600 loss=3.3401 lr=7.50e-05 tok=14.130B dt=5579 wait=190 +13:37:06 step 107620 loss=3.4161 lr=7.50e-05 tok=14.132B dt=5643 wait=191 +13:38:10 step 107640 loss=3.3463 lr=7.49e-05 tok=14.135B dt=5706 wait=191 +13:39:13 step 107660 loss=3.4297 lr=7.49e-05 tok=14.137B dt=5769 wait=191 +13:40:16 step 107680 loss=3.3212 lr=7.49e-05 tok=14.140B dt=5832 wait=192 +13:41:19 step 107700 loss=3.3559 lr=7.49e-05 tok=14.143B dt=5896 wait=192 +13:41:34 pushed ckpt_w79.pt,log_w79.txt +13:42:49 waited 76s for global r=328 +13:42:57 global r=328 load 4.332 -> 4.315 +13:44:00 step 107720 loss=3.2821 lr=7.49e-05 tok=14.145B dt=6057 wait=192 +13:45:04 step 107740 loss=3.4053 lr=7.48e-05 tok=14.148B dt=6120 wait=193 +13:46:08 step 107760 loss=3.4218 lr=7.48e-05 tok=14.151B dt=6184 wait=193 +13:47:11 step 107780 loss=3.3789 lr=7.48e-05 tok=14.153B dt=6248 wait=193 +13:48:14 step 107800 loss=3.3703 lr=7.48e-05 tok=14.156B dt=6311 wait=194 +13:49:18 step 107820 loss=3.4647 lr=7.48e-05 tok=14.158B dt=6374 wait=194 +13:50:21 step 107840 loss=3.3670 lr=7.47e-05 tok=14.161B dt=6438 wait=194 +13:51:05 pushed ckpt_w79.pt,log_w79.txt +13:52:05 waited 61s for global r=329 +13:52:19 global r=329 load 4.313 -> 4.297 +13:52:52 step 107860 loss=3.4437 lr=7.47e-05 tok=14.164B dt=6588 wait=194 +13:53:55 step 107880 loss=3.3362 lr=7.47e-05 tok=14.166B dt=6651 wait=195 +13:54:58 step 107900 loss=3.3012 lr=7.47e-05 tok=14.169B dt=6715 wait=195 +13:56:02 step 107920 loss=3.4804 lr=7.47e-05 tok=14.172B dt=6778 wait=195 +13:57:05 step 107940 loss=3.3143 lr=7.46e-05 tok=14.174B dt=6841 wait=196 +13:58:08 step 107960 loss=3.4607 lr=7.46e-05 tok=14.177B dt=6905 wait=196 +13:59:12 step 107980 loss=3.3642 lr=7.46e-05 tok=14.179B dt=6968 wait=196 +14:00:15 step 108000 loss=3.3707 lr=7.46e-05 tok=14.182B dt=7032 wait=197 +14:00:29 pushed ckpt_w79.pt,log_w79.txt +14:01:45 waited 76s for global r=330 +14:01:56 global r=330 load 4.320 -> 4.286 +14:03:01 step 108020 loss=3.3614 lr=7.45e-05 tok=14.185B dt=7197 wait=197 +14:04:04 step 108040 loss=3.3964 lr=7.45e-05 tok=14.187B dt=7260 wait=197 +14:05:08 step 108060 loss=3.3661 lr=7.45e-05 tok=14.190B dt=7324 wait=198 +14:06:11 step 108080 loss=3.3509 lr=7.45e-05 tok=14.192B dt=7387 wait=198 +14:07:14 step 108100 loss=3.3690 lr=7.45e-05 tok=14.195B dt=7450 wait=198 +14:08:17 step 108120 loss=3.4928 lr=7.44e-05 tok=14.198B dt=7514 wait=199 +14:09:20 step 108140 loss=3.3621 lr=7.44e-05 tok=14.200B dt=7577 wait=199 +14:10:06 pushed ckpt_w79.pt,log_w79.txt +14:11:22 waited 76s for global r=331 +14:11:32 global r=331 load 4.310 -> 4.301 +14:12:04 step 108160 loss=3.4088 lr=7.44e-05 tok=14.203B dt=7741 wait=199 +14:13:08 step 108180 loss=3.3281 lr=7.44e-05 tok=14.206B dt=7804 wait=199 +14:14:11 step 108200 loss=3.3026 lr=7.44e-05 tok=14.208B dt=7867 wait=200 +14:15:14 step 108220 loss=3.4121 lr=7.43e-05 tok=14.211B dt=7931 wait=200 +14:16:18 step 108240 loss=3.2889 lr=7.43e-05 tok=14.213B dt=7994 wait=200 +14:17:21 step 108260 loss=3.4250 lr=7.43e-05 tok=14.216B dt=8057 wait=201 +14:18:24 step 108280 loss=3.3780 lr=7.43e-05 tok=14.219B dt=8121 wait=201 +14:19:28 step 108300 loss=3.4243 lr=7.43e-05 tok=14.221B dt=8184 wait=201 +14:19:41 pushed ckpt_w79.pt,log_w79.txt +14:20:42 waited 60s for global r=332 +14:20:53 global r=332 load 4.274 -> 4.292 +14:21:58 step 108320 loss=3.3209 lr=7.42e-05 tok=14.224B dt=8334 wait=202 +14:23:01 step 108340 loss=3.3346 lr=7.42e-05 tok=14.227B dt=8397 wait=202 +14:24:04 step 108360 loss=3.3701 lr=7.42e-05 tok=14.229B dt=8460 wait=202 +14:25:08 step 108380 loss=3.3042 lr=7.42e-05 tok=14.232B dt=8524 wait=203 +14:26:11 step 108400 loss=3.3139 lr=7.41e-05 tok=14.234B dt=8587 wait=203 +14:27:14 step 108420 loss=3.4936 lr=7.41e-05 tok=14.237B dt=8651 wait=203 +14:28:18 step 108440 loss=3.3037 lr=7.41e-05 tok=14.240B dt=8714 wait=204 +14:29:06 pushed ckpt_w79.pt,log_w79.txt +14:30:21 waited 76s for global r=333 +14:30:33 global r=333 load 4.294 -> 4.294 +14:31:05 step 108460 loss=3.4386 lr=7.41e-05 tok=14.242B dt=8882 wait=204 +14:32:09 step 108480 loss=3.3526 lr=7.41e-05 tok=14.245B dt=8945 wait=204 +14:33:12 step 108500 loss=3.2744 lr=7.40e-05 tok=14.248B dt=9008 wait=205 +14:34:15 step 108520 loss=3.4560 lr=7.40e-05 tok=14.250B dt=9072 wait=205 +14:35:19 step 108540 loss=3.2985 lr=7.40e-05 tok=14.253B dt=9135 wait=205 +14:36:22 step 108560 loss=3.4681 lr=7.40e-05 tok=14.255B dt=9199 wait=206 +14:37:26 step 108580 loss=3.4097 lr=7.40e-05 tok=14.258B dt=9262 wait=206 +14:38:29 step 108600 loss=3.3813 lr=7.39e-05 tok=14.261B dt=9325 wait=206 +14:38:44 pushed ckpt_w79.pt,log_w79.txt +14:39:44 waited 61s for global r=334 +14:39:53 global r=334 load 4.312 -> 4.295 +14:40:58 step 108620 loss=3.3806 lr=7.39e-05 tok=14.263B dt=9474 wait=207 +14:42:01 step 108640 loss=3.4043 lr=7.39e-05 tok=14.266B dt=9537 wait=207 +14:43:04 step 108660 loss=3.3753 lr=7.39e-05 tok=14.268B dt=9600 wait=207 +14:44:07 step 108680 loss=3.3121 lr=7.39e-05 tok=14.271B dt=9664 wait=208 +14:45:11 step 108700 loss=3.3163 lr=7.38e-05 tok=14.274B dt=9727 wait=208 +14:46:15 step 108720 loss=3.4834 lr=7.38e-05 tok=14.276B dt=9791 wait=208 +14:47:18 step 108740 loss=3.3392 lr=7.38e-05 tok=14.279B dt=9854 wait=208 +14:48:05 pushed ckpt_w79.pt,log_w79.txt +14:51:39 step 108760 loss=3.4658 lr=7.38e-05 tok=14.282B dt=10116 wait=209 +14:51:46 global r=335 load 4.344 -> 4.310 +14:52:18 step 108760 loss=3.4369 lr=7.38e-05 tok=14.283B dt=10154 wait=209 +14:53:21 step 108780 loss=3.3230 lr=7.37e-05 tok=14.286B dt=10218 wait=209 +14:54:25 step 108800 loss=3.3343 lr=7.37e-05 tok=14.288B dt=10281 wait=209 +14:55:28 step 108820 loss=3.4692 lr=7.37e-05 tok=14.291B dt=10344 wait=210 +14:56:31 step 108840 loss=3.3547 lr=7.37e-05 tok=14.293B dt=10408 wait=210 +14:57:34 step 108860 loss=3.4305 lr=7.37e-05 tok=14.296B dt=10471 wait=210 +14:58:38 step 108880 loss=3.3229 lr=7.36e-05 tok=14.299B dt=10534 wait=211 +14:59:43 step 108900 loss=3.4090 lr=7.36e-05 tok=14.301B dt=10599 wait=211 +14:59:57 pushed ckpt_w79.pt,log_w79.txt +15:04:03 step 108920 loss=3.3886 lr=7.36e-05 tok=14.304B dt=10859 wait=212 +15:04:44 global r=336 load 4.308 -> 4.312 +15:05:46 step 108920 loss=3.3272 lr=7.36e-05 tok=14.308B dt=10963 wait=212 +15:06:50 step 108940 loss=3.3852 lr=7.36e-05 tok=14.310B dt=11026 wait=212 +15:07:53 step 108960 loss=3.3844 lr=7.36e-05 tok=14.313B dt=11089 wait=213 +15:08:57 step 108980 loss=3.3572 lr=7.35e-05 tok=14.316B dt=11153 wait=213 +15:10:00 step 109000 loss=3.3628 lr=7.35e-05 tok=14.318B dt=11216 wait=214 +15:11:03 step 109020 loss=3.4704 lr=7.35e-05 tok=14.321B dt=11280 wait=214 +15:12:07 step 109040 loss=3.3120 lr=7.35e-05 tok=14.324B dt=11343 wait=214 +15:12:54 pushed ckpt_w79.pt,log_w79.txt +15:16:28 step 109060 loss=3.4557 lr=7.35e-05 tok=14.326B dt=11604 wait=215 +15:17:18 global r=337 load 4.349 -> 4.308 +15:17:50 step 109060 loss=3.3744 lr=7.35e-05 tok=14.329B dt=11686 wait=215 +15:18:53 step 109080 loss=3.3263 lr=7.34e-05 tok=14.331B dt=11750 wait=215 +15:19:56 step 109100 loss=3.3678 lr=7.34e-05 tok=14.334B dt=11813 wait=216 +15:21:00 step 109120 loss=3.4584 lr=7.34e-05 tok=14.337B dt=11876 wait=216 +15:22:03 step 109140 loss=3.3543 lr=7.34e-05 tok=14.339B dt=11939 wait=216 +15:23:07 step 109160 loss=3.4226 lr=7.33e-05 tok=14.342B dt=12003 wait=217 +15:24:10 step 109180 loss=3.3360 lr=7.33e-05 tok=14.345B dt=12067 wait=217 +15:25:14 step 109200 loss=3.3702 lr=7.33e-05 tok=14.347B dt=12130 wait=217 +15:25:29 pushed ckpt_w79.pt,log_w79.txt +15:28:39 waited 191s for global r=338 +15:28:47 global r=338 load 4.297 -> 4.298 +15:29:52 step 109220 loss=3.3243 lr=7.33e-05 tok=14.350B dt=12408 wait=218 +15:30:55 step 109240 loss=3.4206 lr=7.33e-05 tok=14.352B dt=12471 wait=218 +15:31:58 step 109260 loss=3.3902 lr=7.32e-05 tok=14.355B dt=12534 wait=218 +15:33:01 step 109280 loss=3.3518 lr=7.32e-05 tok=14.358B dt=12598 wait=219 +15:34:05 step 109300 loss=3.3649 lr=7.32e-05 tok=14.360B dt=12661 wait=219 +15:35:08 step 109320 loss=3.4274 lr=7.32e-05 tok=14.363B dt=12725 wait=219 +15:36:11 step 109340 loss=3.3425 lr=7.32e-05 tok=14.365B dt=12788 wait=220 +15:36:56 pushed ckpt_w79.pt,log_w79.txt +15:40:29 step 109360 loss=3.4912 lr=7.31e-05 tok=14.368B dt=13046 wait=220 +15:40:38 global r=339 load 4.306 -> 4.285 +15:41:10 step 109360 loss=3.4920 lr=7.31e-05 tok=14.369B dt=13086 wait=220 +15:42:13 step 109380 loss=3.3105 lr=7.31e-05 tok=14.372B dt=13150 wait=220 +15:43:17 step 109400 loss=3.3770 lr=7.31e-05 tok=14.375B dt=13213 wait=221 +15:44:20 step 109420 loss=3.3683 lr=7.31e-05 tok=14.377B dt=13276 wait=221 +15:45:23 step 109440 loss=3.2827 lr=7.31e-05 tok=14.380B dt=13339 wait=221 +15:46:26 step 109460 loss=3.3854 lr=7.30e-05 tok=14.383B dt=13403 wait=222 +15:47:30 step 109480 loss=3.2945 lr=7.30e-05 tok=14.385B dt=13466 wait=222 +15:48:39 step 109500 loss=3.4336 lr=7.30e-05 tok=14.388B dt=13535 wait=222 +15:48:58 pushed ckpt_w79.pt,log_w79.txt +15:52:00 waited 181s for global r=340 +15:52:10 global r=340 load 4.312 -> 4.283 +15:53:14 step 109520 loss=3.3705 lr=7.30e-05 tok=14.390B dt=13811 wait=223 +15:54:18 step 109540 loss=3.3925 lr=7.29e-05 tok=14.393B dt=13874 wait=223 +15:55:21 step 109560 loss=3.3848 lr=7.29e-05 tok=14.396B dt=13937 wait=223 +15:56:24 step 109580 loss=3.3784 lr=7.29e-05 tok=14.398B dt=14001 wait=224 +15:57:27 step 109600 loss=3.3699 lr=7.29e-05 tok=14.401B dt=14064 wait=224 +15:58:31 step 109620 loss=3.4505 lr=7.29e-05 tok=14.404B dt=14127 wait=224 +15:59:34 step 109640 loss=3.2948 lr=7.28e-05 tok=14.406B dt=14190 wait=225 +16:00:19 pushed ckpt_w79.pt,log_w79.txt +16:03:54 step 109660 loss=3.4482 lr=7.28e-05 tok=14.409B dt=14450 wait=225 +16:04:02 global r=341 load 4.329 -> 4.289 +16:04:33 step 109660 loss=3.4685 lr=7.28e-05 tok=14.410B dt=14490 wait=225 +16:05:37 step 109680 loss=3.3557 lr=7.28e-05 tok=14.413B dt=14553 wait=226 +16:06:40 step 109700 loss=3.3165 lr=7.28e-05 tok=14.415B dt=14616 wait=226 +16:07:43 step 109720 loss=3.3752 lr=7.28e-05 tok=14.418B dt=14680 wait=226 +16:08:47 step 109740 loss=3.3229 lr=7.27e-05 tok=14.421B dt=14743 wait=227 +16:09:50 step 109760 loss=3.4392 lr=7.27e-05 tok=14.423B dt=14807 wait=227 +16:10:54 step 109780 loss=3.3656 lr=7.27e-05 tok=14.426B dt=14870 wait=227 +16:11:57 step 109800 loss=3.4318 lr=7.27e-05 tok=14.428B dt=14933 wait=228 +16:12:14 pushed ckpt_w79.pt,log_w79.txt +16:13:30 waited 76s for global r=342 +16:13:38 global r=342 load 4.277 -> 4.273 +16:14:43 step 109820 loss=3.3210 lr=7.27e-05 tok=14.431B dt=15099 wait=228 +16:15:46 step 109840 loss=3.4035 lr=7.26e-05 tok=14.434B dt=15162 wait=228 +16:16:49 step 109860 loss=3.4030 lr=7.26e-05 tok=14.436B dt=15225 wait=229 +16:17:52 step 109880 loss=3.3516 lr=7.26e-05 tok=14.439B dt=15289 wait=229 +16:18:55 step 109900 loss=3.3872 lr=7.26e-05 tok=14.442B dt=15352 wait=229 +16:19:58 step 109920 loss=3.4659 lr=7.25e-05 tok=14.444B dt=15415 wait=230 +16:21:01 step 109940 loss=3.3460 lr=7.25e-05 tok=14.447B dt=15478 wait=230 +16:21:47 pushed ckpt_w79.pt,log_w79.txt +16:23:18 waited 91s for global r=343 +16:23:34 global r=343 load 4.303 -> 4.281 +16:24:06 step 109960 loss=3.4668 lr=7.25e-05 tok=14.449B dt=15663 wait=230 +16:25:10 step 109980 loss=3.3399 lr=7.25e-05 tok=14.452B dt=15726 wait=231 +16:26:13 step 110000 loss=3.3868 lr=7.25e-05 tok=14.455B dt=15790 wait=231 +16:27:17 step 110020 loss=3.4701 lr=7.24e-05 tok=14.457B dt=15854 wait=231 +16:28:20 step 110040 loss=3.3446 lr=7.24e-05 tok=14.460B dt=15917 wait=232 +16:29:24 step 110060 loss=3.4142 lr=7.24e-05 tok=14.462B dt=15980 wait=232 +16:30:27 step 110080 loss=3.3470 lr=7.24e-05 tok=14.465B dt=16043 wait=232 +16:31:31 step 110100 loss=3.4161 lr=7.24e-05 tok=14.468B dt=16107 wait=232 +16:31:42 pushed ckpt_w79.pt,log_w79.txt +16:33:14 waited 91s for global r=344 +16:33:23 global r=344 load 4.325 -> 4.295 +16:34:27 step 110120 loss=3.3625 lr=7.23e-05 tok=14.470B dt=16283 wait=233 +16:35:30 step 110140 loss=3.3994 lr=7.23e-05 tok=14.473B dt=16347 wait=233 +16:36:34 step 110160 loss=3.3894 lr=7.23e-05 tok=14.476B dt=16410 wait=233 +16:37:37 step 110180 loss=3.3378 lr=7.23e-05 tok=14.478B dt=16474 wait=234 +16:38:40 step 110200 loss=3.3264 lr=7.23e-05 tok=14.481B dt=16537 wait=234 +16:39:44 step 110220 loss=3.3999 lr=7.22e-05 tok=14.483B dt=16600 wait=234 +16:40:47 step 110240 loss=3.3100 lr=7.22e-05 tok=14.486B dt=16664 wait=235 +16:41:33 pushed ckpt_w79.pt,log_w79.txt +16:42:35 waited 62s for global r=345 +16:42:51 global r=345 load 4.305 -> 4.309 +16:43:23 step 110260 loss=3.3940 lr=7.22e-05 tok=14.489B dt=16820 wait=235 +16:44:27 step 110280 loss=3.2605 lr=7.22e-05 tok=14.491B dt=16883 wait=236 +16:45:30 step 110300 loss=3.3867 lr=7.22e-05 tok=14.494B dt=16947 wait=236 +16:46:34 step 110320 loss=3.4175 lr=7.21e-05 tok=14.497B dt=17010 wait=236 +16:47:38 step 110340 loss=3.2675 lr=7.21e-05 tok=14.499B dt=17074 wait=237 +16:48:41 step 110360 loss=3.4205 lr=7.21e-05 tok=14.502B dt=17137 wait=237 +16:49:44 step 110380 loss=3.3386 lr=7.21e-05 tok=14.504B dt=17201 wait=237 +16:50:48 step 110400 loss=3.4174 lr=7.20e-05 tok=14.507B dt=17264 wait=238 +16:51:01 pushed ckpt_w79.pt,log_w79.txt +16:52:02 waited 61s for global r=346 +16:52:11 global r=346 load 4.281 -> 4.281 +16:53:17 step 110420 loss=3.3129 lr=7.20e-05 tok=14.510B dt=17414 wait=238 +16:54:21 step 110440 loss=3.3631 lr=7.20e-05 tok=14.512B dt=17477 wait=238 +16:55:24 step 110460 loss=3.4109 lr=7.20e-05 tok=14.515B dt=17540 wait=239 +16:56:28 step 110480 loss=3.3841 lr=7.20e-05 tok=14.518B dt=17604 wait=239 +16:57:31 step 110500 loss=3.3744 lr=7.19e-05 tok=14.520B dt=17667 wait=239 +16:58:34 step 110520 loss=3.3841 lr=7.19e-05 tok=14.523B dt=17730 wait=240 +16:59:37 step 110540 loss=3.3182 lr=7.19e-05 tok=14.525B dt=17794 wait=240 +17:00:23 pushed ckpt_w79.pt,log_w79.txt +17:01:29 waited 66s for global r=347 +17:01:40 global r=347 load 4.273 -> 4.269 +17:02:13 step 110560 loss=3.5020 lr=7.19e-05 tok=14.528B dt=17949 wait=240 +17:03:16 step 110580 loss=3.3468 lr=7.19e-05 tok=14.531B dt=18013 wait=241 +17:04:20 step 110600 loss=3.3151 lr=7.18e-05 tok=14.533B dt=18076 wait=241 +17:05:23 step 110620 loss=3.4144 lr=7.18e-05 tok=14.536B dt=18139 wait=241 +17:06:26 step 110640 loss=3.3473 lr=7.18e-05 tok=14.539B dt=18203 wait=242 +17:07:30 step 110660 loss=3.4160 lr=7.18e-05 tok=14.541B dt=18266 wait=242 +17:08:33 step 110680 loss=3.3403 lr=7.18e-05 tok=14.544B dt=18330 wait=242 +17:09:37 step 110700 loss=3.3508 lr=7.17e-05 tok=14.546B dt=18393 wait=243 +17:09:49 pushed ckpt_w79.pt,log_w79.txt +17:11:35 waited 106s for global r=348 +17:11:50 global r=348 load 4.299 -> 4.297 +17:12:54 step 110720 loss=3.3587 lr=7.17e-05 tok=14.549B dt=18591 wait=243 +17:14:03 step 110740 loss=3.4105 lr=7.17e-05 tok=14.552B dt=18659 wait=243 +17:15:06 step 110760 loss=3.4282 lr=7.17e-05 tok=14.554B dt=18723 wait=244 +17:16:10 step 110780 loss=3.2948 lr=7.16e-05 tok=14.557B dt=18786 wait=244 +17:17:13 step 110800 loss=3.3233 lr=7.16e-05 tok=14.559B dt=18850 wait=244 +17:18:17 step 110820 loss=3.4620 lr=7.16e-05 tok=14.562B dt=18913 wait=245 +17:19:21 step 110840 loss=3.2503 lr=7.16e-05 tok=14.565B dt=18977 wait=245 +17:20:24 pushed ckpt_w79.pt,log_w79.txt +17:21:10 waited 46s for global r=349 +17:21:24 global r=349 load 4.307 -> 4.284 +17:21:58 step 110860 loss=3.4872 lr=7.16e-05 tok=14.567B dt=19135 wait=245 +17:23:02 step 110880 loss=3.3478 lr=7.15e-05 tok=14.570B dt=19198 wait=246 +17:24:05 step 110900 loss=3.3593 lr=7.15e-05 tok=14.573B dt=19261 wait=246 +17:25:09 step 110920 loss=3.4336 lr=7.15e-05 tok=14.575B dt=19325 wait=246 +17:26:12 step 110940 loss=3.3255 lr=7.15e-05 tok=14.578B dt=19389 wait=247 +17:27:16 step 110960 loss=3.4236 lr=7.15e-05 tok=14.580B dt=19452 wait=247 +17:28:19 step 110980 loss=3.3269 lr=7.14e-05 tok=14.583B dt=19515 wait=247 +17:29:22 step 111000 loss=3.4032 lr=7.14e-05 tok=14.586B dt=19579 wait=248 +17:29:35 pushed ckpt_w79.pt,log_w79.txt +17:30:51 waited 76s for global r=350 +17:30:59 global r=350 load 4.330 -> 4.281 +17:32:03 step 111020 loss=3.3698 lr=7.14e-05 tok=14.588B dt=19740 wait=248 +17:33:07 step 111040 loss=3.4232 lr=7.14e-05 tok=14.591B dt=19803 wait=248 +17:34:10 step 111060 loss=3.3645 lr=7.14e-05 tok=14.594B dt=19866 wait=249 +17:35:13 step 111080 loss=3.3530 lr=7.13e-05 tok=14.596B dt=19929 wait=249 +17:36:16 step 111100 loss=3.3522 lr=7.13e-05 tok=14.599B dt=19993 wait=249 +17:37:20 step 111120 loss=3.4369 lr=7.13e-05 tok=14.601B dt=20056 wait=250 +17:38:23 step 111140 loss=3.2921 lr=7.13e-05 tok=14.604B dt=20120 wait=250 +17:39:08 pushed ckpt_w79.pt,log_w79.txt +17:40:24 waited 76s for global r=351 +17:40:32 global r=351 load 4.322 -> 4.275 +17:41:05 step 111160 loss=3.4587 lr=7.13e-05 tok=14.607B dt=20281 wait=250 +17:42:08 step 111180 loss=3.3483 lr=7.12e-05 tok=14.609B dt=20345 wait=251 +17:43:12 step 111200 loss=3.3462 lr=7.12e-05 tok=14.612B dt=20408 wait=251 +17:44:16 step 111220 loss=3.4417 lr=7.12e-05 tok=14.615B dt=20472 wait=252 +17:45:19 step 111240 loss=3.3831 lr=7.12e-05 tok=14.617B dt=20536 wait=252 +17:46:23 step 111260 loss=3.4451 lr=7.11e-05 tok=14.620B dt=20599 wait=252 +17:47:26 step 111280 loss=3.3624 lr=7.11e-05 tok=14.622B dt=20662 wait=252 +17:48:29 step 111300 loss=3.3812 lr=7.11e-05 tok=14.625B dt=20726 wait=253 +17:48:43 pushed ckpt_w79.pt,log_w79.txt +17:49:59 waited 76s for global r=352 +17:50:08 global r=352 load 4.318 -> 4.294 +17:51:13 step 111320 loss=3.3755 lr=7.11e-05 tok=14.628B dt=20889 wait=253 +17:52:16 step 111340 loss=3.3722 lr=7.11e-05 tok=14.630B dt=20952 wait=253 +17:53:20 step 111360 loss=3.3564 lr=7.10e-05 tok=14.633B dt=21016 wait=254 +17:54:23 step 111380 loss=3.3521 lr=7.10e-05 tok=14.635B dt=21079 wait=254 +17:55:26 step 111400 loss=3.3551 lr=7.10e-05 tok=14.638B dt=21142 wait=254 +17:56:29 step 111420 loss=3.4564 lr=7.10e-05 tok=14.641B dt=21206 wait=255 +17:57:33 step 111440 loss=3.2784 lr=7.10e-05 tok=14.643B dt=21269 wait=255 +17:58:16 pushed ckpt_w79.pt,log_w79.txt +18:01:54 step 111460 loss=3.4852 lr=7.09e-05 tok=14.646B dt=21530 wait=255 +18:02:00 global r=353 load 4.301 -> 4.289 +18:02:32 step 111460 loss=3.4694 lr=7.09e-05 tok=14.647B dt=21568 wait=256 +18:03:35 step 111480 loss=3.3378 lr=7.09e-05 tok=14.650B dt=21631 wait=256 +18:04:39 step 111500 loss=3.3189 lr=7.09e-05 tok=14.653B dt=21695 wait=256 +18:05:43 step 111520 loss=3.2795 lr=7.09e-05 tok=14.655B dt=21760 wait=257 +18:06:47 step 111540 loss=3.3644 lr=7.09e-05 tok=14.658B dt=21823 wait=257 +18:07:50 step 111560 loss=3.4605 lr=7.08e-05 tok=14.660B dt=21887 wait=257 +18:08:54 step 111580 loss=3.3345 lr=7.08e-05 tok=14.663B dt=21950 wait=258 +18:09:57 step 111600 loss=3.4451 lr=7.08e-05 tok=14.666B dt=22013 wait=258 diff --git a/log_w8.txt b/log_w8.txt new file mode 100644 index 0000000000000000000000000000000000000000..91189551bfde58a2aebd7b1e0e5323a7ff031713 --- /dev/null +++ b/log_w8.txt @@ -0,0 +1,134 @@ +launch +03:58:12 name=w8 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 0/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +03:58:16 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +03:58:25 params 1.08B +03:58:30 resumed step=37200 seen=4.88B round=17 +03:58:30 stream salt=37200 +04:00:05 start: accum=4, 131072 tokens per optimiser step +04:02:22 step 37220 loss=3.6781 lr=2.78e-04 tok=4.878B dt=137 +04:03:40 step 37240 loss=3.8322 lr=2.78e-04 tok=4.881B dt=215 +04:04:55 step 37260 loss=3.7323 lr=2.78e-04 tok=4.884B dt=290 +04:06:11 step 37280 loss=3.8096 lr=2.78e-04 tok=4.886B dt=366 +04:07:27 step 37300 loss=3.6654 lr=2.78e-04 tok=4.889B dt=442 +04:08:43 step 37320 loss=3.7418 lr=2.78e-04 tok=4.892B dt=518 +04:09:59 step 37340 loss=3.7227 lr=2.78e-04 tok=4.894B dt=594 +04:11:14 step 37360 loss=3.7202 lr=2.78e-04 tok=4.897B dt=669 +04:12:30 step 37380 loss=3.6368 lr=2.78e-04 tok=4.899B dt=745 +04:13:46 step 37400 loss=3.7555 lr=2.78e-04 tok=4.902B dt=821 +04:15:02 step 37420 loss=3.7264 lr=2.78e-04 tok=4.905B dt=897 +04:16:17 step 37440 loss=3.7039 lr=2.78e-04 tok=4.907B dt=972 +04:17:33 step 37460 loss=3.7092 lr=2.78e-04 tok=4.910B dt=1048 +04:18:49 step 37480 loss=3.6740 lr=2.78e-04 tok=4.913B dt=1124 +04:20:04 step 37500 loss=3.7384 lr=2.78e-04 tok=4.915B dt=1199 +04:20:18 pushed ckpt_w8.pt,log_w8.txt +04:21:41 step 37520 loss=3.8401 lr=2.78e-04 tok=4.918B dt=1296 +04:22:57 step 37540 loss=3.7685 lr=2.78e-04 tok=4.920B dt=1372 +04:24:12 step 37560 loss=3.7402 lr=2.78e-04 tok=4.923B dt=1447 +04:25:28 step 37580 loss=3.8468 lr=2.78e-04 tok=4.926B dt=1523 +04:26:44 step 37600 loss=3.7673 lr=2.78e-04 tok=4.928B dt=1599 +04:28:00 step 37620 loss=3.7296 lr=2.78e-04 tok=4.931B dt=1675 +04:29:15 step 37640 loss=3.6914 lr=2.78e-04 tok=4.934B dt=1750 +04:30:31 step 37660 loss=3.6863 lr=2.78e-04 tok=4.936B dt=1826 +04:31:46 step 37680 loss=3.6386 lr=2.78e-04 tok=4.939B dt=1902 +04:33:02 step 37700 loss=3.7373 lr=2.78e-04 tok=4.941B dt=1977 +04:34:18 step 37720 loss=3.7085 lr=2.78e-04 tok=4.944B dt=2053 +04:35:33 step 37740 loss=3.6715 lr=2.78e-04 tok=4.947B dt=2128 +04:36:49 step 37760 loss=3.8088 lr=2.78e-04 tok=4.949B dt=2204 +04:38:05 step 37780 loss=3.6358 lr=2.78e-04 tok=4.952B dt=2280 +04:39:20 step 37800 loss=3.7326 lr=2.77e-04 tok=4.955B dt=2355 +04:39:32 pushed ckpt_w8.pt,log_w8.txt +04:40:55 step 37820 loss=3.7481 lr=2.77e-04 tok=4.957B dt=2450 +04:42:10 step 37840 loss=3.6841 lr=2.77e-04 tok=4.960B dt=2525 +04:43:26 step 37860 loss=3.5992 lr=2.77e-04 tok=4.962B dt=2601 +04:44:42 step 37880 loss=3.6416 lr=2.77e-04 tok=4.965B dt=2677 +04:45:57 step 37900 loss=3.6786 lr=2.77e-04 tok=4.968B dt=2752 +04:47:13 step 37920 loss=3.7956 lr=2.77e-04 tok=4.970B dt=2828 +04:48:29 step 37940 loss=3.9123 lr=2.77e-04 tok=4.973B dt=2904 +04:49:45 step 37960 loss=3.8019 lr=2.77e-04 tok=4.975B dt=2980 +04:51:00 step 37980 loss=3.7170 lr=2.77e-04 tok=4.978B dt=3055 +04:52:16 step 38000 loss=3.6737 lr=2.77e-04 tok=4.981B dt=3131 +04:53:32 step 38020 loss=3.7238 lr=2.77e-04 tok=4.983B dt=3207 +04:54:48 step 38040 loss=3.7343 lr=2.77e-04 tok=4.986B dt=3283 +04:56:03 step 38060 loss=3.7043 lr=2.77e-04 tok=4.989B dt=3358 +04:57:19 step 38080 loss=3.7778 lr=2.77e-04 tok=4.991B dt=3434 +04:58:35 step 38100 loss=3.6533 lr=2.77e-04 tok=4.994B dt=3510 +04:58:49 pushed ckpt_w8.pt,log_w8.txt +05:00:08 step 38120 loss=3.7678 lr=2.77e-04 tok=4.996B dt=3603 +05:01:24 step 38140 loss=3.7074 lr=2.77e-04 tok=4.999B dt=3679 +05:02:39 step 38160 loss=3.7215 lr=2.77e-04 tok=5.002B dt=3755 +05:03:55 step 38180 loss=3.7969 lr=2.77e-04 tok=5.004B dt=3830 +05:05:11 step 38200 loss=3.7084 lr=2.77e-04 tok=5.007B dt=3906 +05:06:27 step 38220 loss=3.7655 lr=2.77e-04 tok=5.010B dt=3982 +05:07:43 step 38240 loss=3.8530 lr=2.77e-04 tok=5.012B dt=4058 +05:08:58 step 38260 loss=3.7917 lr=2.77e-04 tok=5.015B dt=4133 +05:10:14 step 38280 loss=3.7128 lr=2.77e-04 tok=5.017B dt=4209 +05:11:30 step 38300 loss=3.7340 lr=2.77e-04 tok=5.020B dt=4285 +05:12:45 step 38320 loss=3.6795 lr=2.77e-04 tok=5.023B dt=4360 +05:14:01 step 38340 loss=3.7626 lr=2.77e-04 tok=5.025B dt=4436 +05:15:17 step 38360 loss=3.6922 lr=2.77e-04 tok=5.028B dt=4512 +05:16:32 step 38380 loss=3.6998 lr=2.77e-04 tok=5.031B dt=4587 +05:17:48 step 38400 loss=3.8202 lr=2.77e-04 tok=5.033B dt=4663 +05:18:00 pushed ckpt_w8.pt,log_w8.txt +05:19:23 step 38420 loss=3.7313 lr=2.77e-04 tok=5.036B dt=4758 +05:20:39 step 38440 loss=3.7427 lr=2.77e-04 tok=5.038B dt=4834 +05:21:54 step 38460 loss=3.6900 lr=2.77e-04 tok=5.041B dt=4909 +05:23:10 step 38480 loss=3.7159 lr=2.77e-04 tok=5.044B dt=4985 +05:24:26 step 38500 loss=3.6335 lr=2.77e-04 tok=5.046B dt=5061 +05:25:42 step 38520 loss=3.8048 lr=2.77e-04 tok=5.049B dt=5137 +05:26:57 step 38540 loss=3.7465 lr=2.77e-04 tok=5.052B dt=5212 +05:28:13 step 38560 loss=3.6001 lr=2.77e-04 tok=5.054B dt=5288 +05:29:28 step 38580 loss=3.7585 lr=2.77e-04 tok=5.057B dt=5363 +05:30:44 step 38600 loss=3.8088 lr=2.77e-04 tok=5.059B dt=5439 +05:32:00 step 38620 loss=3.7943 lr=2.77e-04 tok=5.062B dt=5515 +05:33:16 step 38640 loss=3.7890 lr=2.76e-04 tok=5.065B dt=5591 +05:34:31 step 38660 loss=3.6564 lr=2.76e-04 tok=5.067B dt=5666 +05:35:47 step 38680 loss=3.7088 lr=2.76e-04 tok=5.070B dt=5742 +05:37:03 step 38700 loss=3.7732 lr=2.76e-04 tok=5.072B dt=5818 +05:37:16 pushed ckpt_w8.pt,log_w8.txt +05:38:38 step 38720 loss=3.7236 lr=2.76e-04 tok=5.075B dt=5914 +05:39:54 step 38740 loss=3.6925 lr=2.76e-04 tok=5.078B dt=5989 +05:41:10 step 38760 loss=3.8259 lr=2.76e-04 tok=5.080B dt=6065 +05:42:26 step 38780 loss=3.7908 lr=2.76e-04 tok=5.083B dt=6141 +05:43:42 step 38800 loss=3.7442 lr=2.76e-04 tok=5.086B dt=6217 +05:44:57 step 38820 loss=3.7902 lr=2.76e-04 tok=5.088B dt=6292 +05:46:13 step 38840 loss=3.7045 lr=2.76e-04 tok=5.091B dt=6368 +05:47:28 step 38860 loss=3.7896 lr=2.76e-04 tok=5.093B dt=6443 +05:48:44 step 38880 loss=3.6338 lr=2.76e-04 tok=5.096B dt=6519 +05:50:00 step 38900 loss=3.6489 lr=2.76e-04 tok=5.099B dt=6595 +05:51:16 step 38920 loss=3.7055 lr=2.76e-04 tok=5.101B dt=6671 +05:52:31 step 38940 loss=3.7803 lr=2.76e-04 tok=5.104B dt=6747 +05:53:47 step 38960 loss=3.7231 lr=2.76e-04 tok=5.107B dt=6822 +05:55:03 step 38980 loss=3.7495 lr=2.76e-04 tok=5.109B dt=6898 +05:56:19 step 39000 loss=3.6696 lr=2.76e-04 tok=5.112B dt=6974 +05:56:31 pushed ckpt_w8.pt,log_w8.txt +05:57:52 step 39020 loss=3.8338 lr=2.76e-04 tok=5.114B dt=7068 +05:59:08 step 39040 loss=3.7511 lr=2.76e-04 tok=5.117B dt=7143 +06:00:24 step 39060 loss=3.7523 lr=2.76e-04 tok=5.120B dt=7219 +06:01:40 step 39080 loss=3.6439 lr=2.76e-04 tok=5.122B dt=7295 +06:02:55 step 39100 loss=3.7317 lr=2.76e-04 tok=5.125B dt=7370 +06:04:11 step 39120 loss=3.6970 lr=2.76e-04 tok=5.128B dt=7446 +06:05:27 step 39140 loss=3.7344 lr=2.76e-04 tok=5.130B dt=7522 +06:06:42 step 39160 loss=3.8036 lr=2.76e-04 tok=5.133B dt=7597 +06:07:58 step 39180 loss=3.8780 lr=2.76e-04 tok=5.135B dt=7673 +06:09:14 step 39200 loss=3.7339 lr=2.76e-04 tok=5.138B dt=7749 +06:10:30 step 39220 loss=3.7584 lr=2.76e-04 tok=5.141B dt=7825 +06:11:45 step 39240 loss=3.7666 lr=2.76e-04 tok=5.143B dt=7901 +06:13:01 step 39260 loss=3.6913 lr=2.76e-04 tok=5.146B dt=7976 +06:14:17 step 39280 loss=3.6456 lr=2.76e-04 tok=5.149B dt=8052 +06:15:33 step 39300 loss=3.7126 lr=2.76e-04 tok=5.151B dt=8128 +06:15:45 pushed ckpt_w8.pt,log_w8.txt +06:17:09 step 39320 loss=3.7551 lr=2.76e-04 tok=5.154B dt=8225 +06:18:25 step 39340 loss=3.7043 lr=2.76e-04 tok=5.156B dt=8300 +06:19:41 step 39360 loss=3.7441 lr=2.76e-04 tok=5.159B dt=8376 +06:20:56 step 39380 loss=3.7459 lr=2.76e-04 tok=5.162B dt=8451 +06:22:12 step 39400 loss=3.7008 lr=2.76e-04 tok=5.164B dt=8527 +06:23:28 step 39420 loss=3.7566 lr=2.76e-04 tok=5.167B dt=8603 +06:24:44 step 39440 loss=3.6983 lr=2.76e-04 tok=5.169B dt=8679 +06:25:59 step 39460 loss=3.8046 lr=2.76e-04 tok=5.172B dt=8754 +06:27:15 step 39480 loss=3.7462 lr=2.75e-04 tok=5.175B dt=8830 +06:28:31 step 39500 loss=3.7099 lr=2.75e-04 tok=5.177B dt=8906 +06:29:46 step 39520 loss=3.7780 lr=2.75e-04 tok=5.180B dt=8981 +06:31:02 step 39540 loss=3.7520 lr=2.75e-04 tok=5.183B dt=9057 +06:32:18 step 39560 loss=3.7370 lr=2.75e-04 tok=5.185B dt=9133 +06:33:33 step 39580 loss=3.7790 lr=2.75e-04 tok=5.188B dt=9208 +06:34:49 step 39600 loss=3.6450 lr=2.75e-04 tok=5.190B dt=9284 diff --git a/log_w80.txt b/log_w80.txt new file mode 100644 index 0000000000000000000000000000000000000000..42d803bb06f66eba2ba09de7b483f5fe1972bbb3 --- /dev/null +++ b/log_w80.txt @@ -0,0 +1,176 @@ +launch +12:00:37 name=w80 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 27/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:00:43 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:00:43 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:00:53 params 1.08B +12:00:54 resumed step=106350 seen=13.96B round=319 +12:00:54 stream salt=106350 +12:02:57 blocks compiled +12:02:57 start: accum=4, 131072 tokens per optimiser step +12:06:27 step 106351 loss=3.1648 lr=7.63e-05 tok=13.961B dt=211 wait=173 +12:06:31 step 106352 loss=3.2097 lr=7.63e-05 tok=13.961B dt=214 wait=173 +12:06:58 step 106360 loss=3.4485 lr=7.63e-05 tok=13.962B dt=241 wait=173 +12:08:06 step 106380 loss=3.3952 lr=7.63e-05 tok=13.964B dt=309 wait=174 +12:09:11 step 106400 loss=3.4407 lr=7.63e-05 tok=13.967B dt=374 wait=174 +12:10:14 step 106420 loss=3.3902 lr=7.62e-05 tok=13.970B dt=437 wait=175 +12:11:17 step 106440 loss=3.4465 lr=7.62e-05 tok=13.972B dt=501 wait=175 +12:12:21 step 106460 loss=3.5830 lr=7.62e-05 tok=13.975B dt=564 wait=175 +12:13:24 step 106480 loss=3.3940 lr=7.62e-05 tok=13.978B dt=627 wait=175 +12:14:27 step 106500 loss=3.4072 lr=7.61e-05 tok=13.980B dt=690 wait=176 +12:14:41 pushed ckpt_w80.pt,log_w80.txt +12:15:42 waited 60s for global r=320 +12:16:00 global r=320 load 4.322 -> 4.302 +12:17:15 step 106520 loss=3.3692 lr=7.61e-05 tok=13.983B dt=859 wait=176 +12:18:23 step 106540 loss=3.3723 lr=7.61e-05 tok=13.985B dt=927 wait=176 +12:19:26 step 106560 loss=3.4131 lr=7.61e-05 tok=13.988B dt=990 wait=177 +12:20:29 step 106580 loss=3.3298 lr=7.61e-05 tok=13.991B dt=1053 wait=177 +12:21:32 step 106600 loss=3.3857 lr=7.60e-05 tok=13.993B dt=1116 wait=177 +12:22:35 step 106620 loss=3.4441 lr=7.60e-05 tok=13.996B dt=1179 wait=178 +12:23:38 step 106640 loss=3.2474 lr=7.60e-05 tok=13.998B dt=1242 wait=178 +12:24:23 pushed ckpt_w80.pt,log_w80.txt +12:28:02 step 106660 loss=3.4687 lr=7.60e-05 tok=14.001B dt=1505 wait=178 +12:28:15 global r=321 load 4.305 -> 4.304 +12:28:56 step 106660 loss=3.4560 lr=7.60e-05 tok=14.002B dt=1559 wait=179 +12:29:59 step 106680 loss=3.3277 lr=7.60e-05 tok=14.005B dt=1622 wait=179 +12:31:02 step 106700 loss=3.3507 lr=7.59e-05 tok=14.008B dt=1686 wait=179 +12:32:05 step 106720 loss=3.3832 lr=7.59e-05 tok=14.010B dt=1749 wait=180 +12:33:08 step 106740 loss=3.3335 lr=7.59e-05 tok=14.013B dt=1812 wait=180 +12:34:12 step 106760 loss=3.3909 lr=7.59e-05 tok=14.016B dt=1875 wait=180 +12:35:15 step 106780 loss=3.3723 lr=7.59e-05 tok=14.018B dt=1938 wait=180 +12:36:18 step 106800 loss=3.4020 lr=7.58e-05 tok=14.021B dt=2001 wait=181 +12:36:32 pushed ckpt_w80.pt,log_w80.txt +12:39:03 waited 151s for global r=322 +12:39:10 global r=322 load 4.310 -> 4.303 +12:40:15 step 106820 loss=3.3107 lr=7.58e-05 tok=14.023B dt=2239 wait=181 +12:41:22 step 106840 loss=3.4070 lr=7.58e-05 tok=14.026B dt=2305 wait=181 +12:42:36 step 106860 loss=3.4830 lr=7.58e-05 tok=14.029B dt=2380 wait=182 +12:43:40 step 106880 loss=3.3471 lr=7.57e-05 tok=14.031B dt=2443 wait=182 +12:44:43 step 106900 loss=3.3539 lr=7.57e-05 tok=14.034B dt=2506 wait=182 +12:45:46 step 106920 loss=3.4070 lr=7.57e-05 tok=14.037B dt=2570 wait=183 +12:46:49 step 106940 loss=3.2572 lr=7.57e-05 tok=14.039B dt=2633 wait=183 +12:47:33 pushed ckpt_w80.pt,log_w80.txt +12:51:07 step 106960 loss=3.4091 lr=7.57e-05 tok=14.042B dt=2891 wait=183 +12:51:16 global r=323 load 4.295 -> 4.300 +12:51:48 step 106960 loss=3.3943 lr=7.57e-05 tok=14.043B dt=2931 wait=183 +12:52:56 step 106980 loss=3.3133 lr=7.56e-05 tok=14.046B dt=2999 wait=184 +12:54:08 step 107000 loss=3.2801 lr=7.56e-05 tok=14.048B dt=3071 wait=184 +12:55:11 step 107020 loss=3.4597 lr=7.56e-05 tok=14.051B dt=3134 wait=184 +12:56:14 step 107040 loss=3.2166 lr=7.56e-05 tok=14.054B dt=3198 wait=185 +12:57:17 step 107060 loss=3.4348 lr=7.56e-05 tok=14.056B dt=3261 wait=185 +12:58:21 step 107080 loss=3.3781 lr=7.55e-05 tok=14.059B dt=3324 wait=185 +12:59:23 step 107100 loss=3.4118 lr=7.55e-05 tok=14.061B dt=3387 wait=186 +12:59:35 pushed ckpt_w80.pt,log_w80.txt +13:02:37 waited 181s for global r=324 +13:02:45 global r=324 load 4.320 -> 4.306 +13:03:49 step 107120 loss=3.3696 lr=7.55e-05 tok=14.064B dt=3653 wait=186 +13:05:02 step 107140 loss=3.4120 lr=7.55e-05 tok=14.067B dt=3725 wait=186 +13:06:10 step 107160 loss=3.3160 lr=7.55e-05 tok=14.069B dt=3793 wait=187 +13:07:13 step 107180 loss=3.3630 lr=7.54e-05 tok=14.072B dt=3857 wait=187 +13:08:16 step 107200 loss=3.3682 lr=7.54e-05 tok=14.075B dt=3920 wait=187 +13:09:20 step 107220 loss=3.4494 lr=7.54e-05 tok=14.077B dt=3983 wait=188 +13:10:23 step 107240 loss=3.2761 lr=7.54e-05 tok=14.080B dt=4046 wait=188 +13:11:06 pushed ckpt_w80.pt,log_w80.txt +13:14:40 step 107260 loss=3.4471 lr=7.53e-05 tok=14.082B dt=4304 wait=188 +13:14:50 global r=325 load 4.290 -> 4.291 +13:15:22 step 107260 loss=3.4223 lr=7.53e-05 tok=14.084B dt=4345 wait=188 +13:16:36 step 107280 loss=3.3445 lr=7.53e-05 tok=14.086B dt=4420 wait=189 +13:17:43 step 107300 loss=3.2908 lr=7.53e-05 tok=14.089B dt=4486 wait=189 +13:18:46 step 107320 loss=3.4188 lr=7.53e-05 tok=14.092B dt=4549 wait=189 +13:19:48 step 107340 loss=3.3190 lr=7.53e-05 tok=14.094B dt=4612 wait=190 +13:20:52 step 107360 loss=3.3598 lr=7.52e-05 tok=14.097B dt=4675 wait=190 +13:21:54 step 107380 loss=3.3677 lr=7.52e-05 tok=14.099B dt=4738 wait=190 +13:22:57 step 107400 loss=3.4055 lr=7.52e-05 tok=14.102B dt=4801 wait=191 +13:23:12 pushed ckpt_w80.pt,log_w80.txt +13:23:58 waited 46s for global r=326 +13:24:05 global r=326 load 4.299 -> 4.288 +13:25:13 step 107420 loss=3.3688 lr=7.52e-05 tok=14.105B dt=4937 wait=191 +13:26:27 step 107440 loss=3.3660 lr=7.52e-05 tok=14.107B dt=5011 wait=191 +13:27:30 step 107460 loss=3.3569 lr=7.51e-05 tok=14.110B dt=5074 wait=192 +13:28:33 step 107480 loss=3.3333 lr=7.51e-05 tok=14.113B dt=5137 wait=192 +13:29:37 step 107500 loss=3.4034 lr=7.51e-05 tok=14.115B dt=5200 wait=192 +13:30:39 step 107520 loss=3.4190 lr=7.51e-05 tok=14.118B dt=5263 wait=193 +13:31:42 step 107540 loss=3.3284 lr=7.51e-05 tok=14.120B dt=5326 wait=193 +13:32:26 pushed ckpt_w80.pt,log_w80.txt +13:33:27 waited 60s for global r=327 +13:33:34 global r=327 load 4.310 -> 4.291 +13:34:07 step 107560 loss=3.4149 lr=7.50e-05 tok=14.123B dt=5470 wait=193 +13:35:20 step 107580 loss=3.2873 lr=7.50e-05 tok=14.126B dt=5543 wait=194 +13:36:28 step 107600 loss=3.3311 lr=7.50e-05 tok=14.128B dt=5611 wait=194 +13:37:31 step 107620 loss=3.3973 lr=7.50e-05 tok=14.131B dt=5675 wait=194 +13:38:35 step 107640 loss=3.3367 lr=7.49e-05 tok=14.133B dt=5738 wait=195 +13:39:38 step 107660 loss=3.4411 lr=7.49e-05 tok=14.136B dt=5801 wait=195 +13:40:40 step 107680 loss=3.3995 lr=7.49e-05 tok=14.139B dt=5864 wait=195 +13:41:44 step 107700 loss=3.3836 lr=7.49e-05 tok=14.141B dt=5927 wait=196 +13:41:56 pushed ckpt_w80.pt,log_w80.txt +13:42:41 waited 45s for global r=328 +13:42:50 global r=328 load 4.322 -> 4.315 +13:43:55 step 107720 loss=3.3270 lr=7.49e-05 tok=14.144B dt=6058 wait=196 +13:45:13 step 107740 loss=3.4068 lr=7.48e-05 tok=14.147B dt=6136 wait=196 +13:46:16 step 107760 loss=3.3769 lr=7.48e-05 tok=14.149B dt=6199 wait=196 +13:47:19 step 107780 loss=3.3607 lr=7.48e-05 tok=14.152B dt=6263 wait=197 +13:48:22 step 107800 loss=3.3637 lr=7.48e-05 tok=14.154B dt=6326 wait=197 +13:49:25 step 107820 loss=3.4689 lr=7.48e-05 tok=14.157B dt=6389 wait=197 +13:50:29 step 107840 loss=3.3338 lr=7.47e-05 tok=14.160B dt=6452 wait=198 +13:51:17 pushed ckpt_w80.pt,log_w80.txt +13:52:02 waited 45s for global r=329 +13:52:13 global r=329 load 4.298 -> 4.297 +13:52:46 step 107860 loss=3.4205 lr=7.47e-05 tok=14.162B dt=6590 wait=198 +13:53:55 step 107880 loss=3.2916 lr=7.47e-05 tok=14.165B dt=6658 wait=198 +13:55:08 step 107900 loss=3.3389 lr=7.47e-05 tok=14.168B dt=6731 wait=199 +13:56:11 step 107920 loss=3.3926 lr=7.47e-05 tok=14.170B dt=6795 wait=199 +13:57:14 step 107940 loss=3.3154 lr=7.46e-05 tok=14.173B dt=6857 wait=199 +13:58:17 step 107960 loss=3.3807 lr=7.46e-05 tok=14.175B dt=6921 wait=200 +13:59:20 step 107980 loss=3.3360 lr=7.46e-05 tok=14.178B dt=6983 wait=200 +14:00:23 step 108000 loss=3.4084 lr=7.46e-05 tok=14.181B dt=7046 wait=200 +14:00:37 pushed ckpt_w80.pt,log_w80.txt +14:01:52 waited 76s for global r=330 +14:02:02 global r=330 load 4.295 -> 4.286 +14:03:11 step 108020 loss=3.3884 lr=7.45e-05 tok=14.183B dt=7214 wait=201 +14:04:24 step 108040 loss=3.3662 lr=7.45e-05 tok=14.186B dt=7288 wait=201 +14:05:27 step 108060 loss=3.3699 lr=7.45e-05 tok=14.189B dt=7351 wait=201 +14:06:30 step 108080 loss=3.3696 lr=7.45e-05 tok=14.191B dt=7414 wait=202 +14:07:33 step 108100 loss=3.3958 lr=7.45e-05 tok=14.194B dt=7476 wait=202 +14:08:36 step 108120 loss=3.4199 lr=7.44e-05 tok=14.196B dt=7539 wait=202 +14:09:39 step 108140 loss=3.3961 lr=7.44e-05 tok=14.199B dt=7602 wait=202 +14:10:23 pushed ckpt_w80.pt,log_w80.txt +14:11:08 waited 45s for global r=331 +14:11:15 global r=331 load 4.315 -> 4.301 +14:11:48 step 108160 loss=3.4543 lr=7.44e-05 tok=14.202B dt=7731 wait=203 +14:12:57 step 108180 loss=3.3404 lr=7.44e-05 tok=14.204B dt=7800 wait=203 +14:14:09 step 108200 loss=3.3760 lr=7.44e-05 tok=14.207B dt=7872 wait=203 +14:15:12 step 108220 loss=3.4230 lr=7.43e-05 tok=14.210B dt=7936 wait=204 +14:16:15 step 108240 loss=3.3397 lr=7.43e-05 tok=14.212B dt=7999 wait=204 +14:17:18 step 108260 loss=3.4173 lr=7.43e-05 tok=14.215B dt=8062 wait=204 +14:18:21 step 108280 loss=3.2626 lr=7.43e-05 tok=14.217B dt=8125 wait=205 +14:19:25 step 108300 loss=3.3058 lr=7.43e-05 tok=14.220B dt=8188 wait=205 +14:19:37 pushed ckpt_w80.pt,log_w80.txt +14:20:37 waited 61s for global r=332 +14:20:49 global r=332 load 4.326 -> 4.292 +14:21:57 step 108320 loss=3.3325 lr=7.42e-05 tok=14.223B dt=8340 wait=205 +14:23:11 step 108340 loss=3.3815 lr=7.42e-05 tok=14.225B dt=8415 wait=205 +14:24:20 step 108360 loss=3.3546 lr=7.42e-05 tok=14.228B dt=8483 wait=206 +14:25:23 step 108380 loss=3.3699 lr=7.42e-05 tok=14.230B dt=8546 wait=206 +14:26:26 step 108400 loss=3.2707 lr=7.41e-05 tok=14.233B dt=8609 wait=206 +14:27:29 step 108420 loss=3.4524 lr=7.41e-05 tok=14.236B dt=8673 wait=207 +14:28:32 step 108440 loss=3.3356 lr=7.41e-05 tok=14.238B dt=8736 wait=207 +14:29:17 pushed ckpt_w80.pt,log_w80.txt +14:30:18 waited 60s for global r=333 +14:30:30 global r=333 load 4.322 -> 4.294 +14:31:04 step 108460 loss=3.4003 lr=7.41e-05 tok=14.241B dt=8887 wait=207 +14:32:18 step 108480 loss=3.2959 lr=7.41e-05 tok=14.244B dt=8962 wait=208 +14:33:24 step 108500 loss=3.3383 lr=7.40e-05 tok=14.246B dt=9027 wait=208 +14:34:27 step 108520 loss=3.3938 lr=7.40e-05 tok=14.249B dt=9091 wait=208 +14:35:30 step 108540 loss=3.3172 lr=7.40e-05 tok=14.251B dt=9154 wait=209 +14:36:34 step 108560 loss=3.4435 lr=7.40e-05 tok=14.254B dt=9217 wait=209 +14:37:37 step 108580 loss=3.3510 lr=7.40e-05 tok=14.257B dt=9280 wait=209 +14:38:40 step 108600 loss=3.3996 lr=7.39e-05 tok=14.259B dt=9343 wait=210 +14:38:52 pushed ckpt_w80.pt,log_w80.txt +14:39:53 waited 60s for global r=334 +14:40:03 global r=334 load 4.302 -> 4.295 +14:41:15 step 108620 loss=3.3108 lr=7.39e-05 tok=14.262B dt=9499 wait=210 +14:42:31 step 108640 loss=3.3869 lr=7.39e-05 tok=14.265B dt=9574 wait=210 +14:43:34 step 108660 loss=3.3346 lr=7.39e-05 tok=14.267B dt=9637 wait=211 +14:44:37 step 108680 loss=3.3193 lr=7.39e-05 tok=14.270B dt=9701 wait=211 +14:45:40 step 108700 loss=3.3215 lr=7.38e-05 tok=14.272B dt=9764 wait=211 +14:46:43 step 108720 loss=3.4209 lr=7.38e-05 tok=14.275B dt=9827 wait=212 +14:47:46 step 108740 loss=3.2814 lr=7.38e-05 tok=14.278B dt=9890 wait=212 diff --git a/log_w81.txt b/log_w81.txt new file mode 100644 index 0000000000000000000000000000000000000000..ecdf18f19c1c0dbc3288adc2bdf54612b69ac040 --- /dev/null +++ b/log_w81.txt @@ -0,0 +1,362 @@ +launch +12:00:39 name=w81 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 28/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:00:45 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:00:45 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:00:55 params 1.08B +12:00:57 resumed step=106350 seen=13.96B round=319 +12:00:57 stream salt=106350 +12:03:02 blocks compiled +12:03:02 start: accum=4, 131072 tokens per optimiser step +12:06:27 step 106351 loss=3.2931 lr=7.63e-05 tok=13.961B dt=205 wait=165 +12:06:31 step 106352 loss=3.2721 lr=7.63e-05 tok=13.961B dt=208 wait=165 +12:06:59 step 106360 loss=3.5023 lr=7.63e-05 tok=13.962B dt=237 wait=165 +12:08:08 step 106380 loss=3.3984 lr=7.63e-05 tok=13.964B dt=306 wait=166 +12:09:12 step 106400 loss=3.4623 lr=7.63e-05 tok=13.967B dt=370 wait=166 +12:10:16 step 106420 loss=3.3567 lr=7.62e-05 tok=13.970B dt=433 wait=167 +12:11:19 step 106440 loss=3.4482 lr=7.62e-05 tok=13.972B dt=497 wait=167 +12:12:23 step 106460 loss=3.6162 lr=7.62e-05 tok=13.975B dt=561 wait=167 +12:13:27 step 106480 loss=3.4421 lr=7.62e-05 tok=13.978B dt=625 wait=168 +12:14:31 step 106500 loss=3.4618 lr=7.61e-05 tok=13.980B dt=689 wait=168 +12:14:47 pushed ckpt_w81.pt,log_w81.txt +12:15:33 waited 45s for global r=320 +12:15:51 global r=320 load 4.299 -> 4.302 +12:16:58 step 106520 loss=3.3558 lr=7.61e-05 tok=13.983B dt=835 wait=168 +12:18:02 step 106540 loss=3.4470 lr=7.61e-05 tok=13.985B dt=899 wait=169 +12:19:05 step 106560 loss=3.4411 lr=7.61e-05 tok=13.988B dt=963 wait=169 +12:20:09 step 106580 loss=3.3739 lr=7.61e-05 tok=13.991B dt=1026 wait=169 +12:21:12 step 106600 loss=3.3915 lr=7.60e-05 tok=13.993B dt=1090 wait=170 +12:22:16 step 106620 loss=3.4909 lr=7.60e-05 tok=13.996B dt=1154 wait=170 +12:23:20 step 106640 loss=3.3412 lr=7.60e-05 tok=13.998B dt=1218 wait=170 +12:24:10 pushed ckpt_w81.pt,log_w81.txt +12:27:46 step 106660 loss=3.4624 lr=7.60e-05 tok=14.001B dt=1484 wait=171 +12:27:55 global r=321 load 4.312 -> 4.304 +12:28:26 step 106660 loss=3.4604 lr=7.60e-05 tok=14.002B dt=1524 wait=171 +12:29:30 step 106680 loss=3.3909 lr=7.60e-05 tok=14.005B dt=1587 wait=171 +12:30:34 step 106700 loss=3.3598 lr=7.59e-05 tok=14.008B dt=1651 wait=171 +12:31:38 step 106720 loss=3.4616 lr=7.59e-05 tok=14.010B dt=1715 wait=172 +12:32:42 step 106740 loss=3.3839 lr=7.59e-05 tok=14.013B dt=1779 wait=172 +12:33:45 step 106760 loss=3.4371 lr=7.59e-05 tok=14.016B dt=1843 wait=172 +12:34:49 step 106780 loss=3.3844 lr=7.59e-05 tok=14.018B dt=1906 wait=173 +12:35:54 step 106800 loss=3.4683 lr=7.58e-05 tok=14.021B dt=1971 wait=173 +12:36:06 pushed ckpt_w81.pt,log_w81.txt +12:39:07 waited 181s for global r=322 +12:39:14 global r=322 load 4.329 -> 4.303 +12:40:20 step 106820 loss=3.3340 lr=7.58e-05 tok=14.023B dt=2237 wait=173 +12:41:24 step 106840 loss=3.4094 lr=7.58e-05 tok=14.026B dt=2301 wait=174 +12:42:28 step 106860 loss=3.4017 lr=7.58e-05 tok=14.029B dt=2365 wait=174 +12:43:32 step 106880 loss=3.3885 lr=7.57e-05 tok=14.031B dt=2429 wait=174 +12:44:35 step 106900 loss=3.3947 lr=7.57e-05 tok=14.034B dt=2493 wait=175 +12:45:40 step 106920 loss=3.4867 lr=7.57e-05 tok=14.037B dt=2557 wait=175 +12:46:43 step 106940 loss=3.3341 lr=7.57e-05 tok=14.039B dt=2621 wait=175 +12:47:29 pushed ckpt_w81.pt,log_w81.txt +12:51:04 step 106960 loss=3.5160 lr=7.57e-05 tok=14.042B dt=2881 wait=175 +12:51:13 global r=323 load 4.349 -> 4.300 +12:51:44 step 106960 loss=3.4588 lr=7.57e-05 tok=14.043B dt=2922 wait=176 +12:52:48 step 106980 loss=3.3347 lr=7.56e-05 tok=14.046B dt=2986 wait=176 +12:53:52 step 107000 loss=3.4543 lr=7.56e-05 tok=14.048B dt=3050 wait=176 +12:54:56 step 107020 loss=3.4741 lr=7.56e-05 tok=14.051B dt=3113 wait=177 +12:56:00 step 107040 loss=3.3436 lr=7.56e-05 tok=14.054B dt=3177 wait=177 +12:57:03 step 107060 loss=3.4621 lr=7.56e-05 tok=14.056B dt=3241 wait=177 +12:58:07 step 107080 loss=3.3977 lr=7.55e-05 tok=14.059B dt=3305 wait=178 +12:59:12 step 107100 loss=3.4338 lr=7.55e-05 tok=14.061B dt=3369 wait=178 +12:59:24 pushed ckpt_w81.pt,log_w81.txt +13:03:05 global r=324 load 4.352 -> 4.306 +13:04:09 step 107120 loss=3.3641 lr=7.55e-05 tok=14.065B dt=3666 wait=178 +13:05:13 step 107140 loss=3.3440 lr=7.55e-05 tok=14.068B dt=3730 wait=179 +13:06:17 step 107160 loss=3.3800 lr=7.55e-05 tok=14.071B dt=3794 wait=179 +13:07:20 step 107180 loss=3.3424 lr=7.54e-05 tok=14.073B dt=3858 wait=179 +13:08:24 step 107200 loss=3.4048 lr=7.54e-05 tok=14.076B dt=3921 wait=180 +13:09:28 step 107220 loss=3.4921 lr=7.54e-05 tok=14.078B dt=3985 wait=180 +13:10:31 step 107240 loss=3.3608 lr=7.54e-05 tok=14.081B dt=4049 wait=180 +13:11:15 pushed ckpt_w81.pt,log_w81.txt +13:14:49 step 107260 loss=3.4723 lr=7.53e-05 tok=14.084B dt=4307 wait=180 +13:14:59 global r=325 load 4.284 -> 4.291 +13:15:31 step 107260 loss=3.4827 lr=7.53e-05 tok=14.085B dt=4348 wait=181 +13:16:34 step 107280 loss=3.3539 lr=7.53e-05 tok=14.088B dt=4412 wait=181 +13:17:38 step 107300 loss=3.3794 lr=7.53e-05 tok=14.090B dt=4475 wait=181 +13:18:42 step 107320 loss=3.4770 lr=7.53e-05 tok=14.093B dt=4539 wait=182 +13:19:46 step 107340 loss=3.3985 lr=7.53e-05 tok=14.095B dt=4603 wait=182 +13:20:49 step 107360 loss=3.4320 lr=7.52e-05 tok=14.098B dt=4667 wait=182 +13:21:53 step 107380 loss=3.4224 lr=7.52e-05 tok=14.101B dt=4731 wait=182 +13:22:57 step 107400 loss=3.4594 lr=7.52e-05 tok=14.103B dt=4795 wait=183 +13:23:10 pushed ckpt_w81.pt,log_w81.txt +13:24:10 waited 60s for global r=326 +13:24:17 global r=326 load 4.312 -> 4.288 +13:25:21 step 107420 loss=3.3910 lr=7.52e-05 tok=14.106B dt=4939 wait=183 +13:26:25 step 107440 loss=3.4254 lr=7.52e-05 tok=14.109B dt=5003 wait=183 +13:27:29 step 107460 loss=3.4237 lr=7.51e-05 tok=14.111B dt=5066 wait=184 +13:28:33 step 107480 loss=3.3930 lr=7.51e-05 tok=14.114B dt=5130 wait=184 +13:29:36 step 107500 loss=3.3747 lr=7.51e-05 tok=14.116B dt=5194 wait=184 +13:30:40 step 107520 loss=3.4903 lr=7.51e-05 tok=14.119B dt=5258 wait=185 +13:31:44 step 107540 loss=3.2966 lr=7.51e-05 tok=14.122B dt=5322 wait=185 +13:32:30 pushed ckpt_w81.pt,log_w81.txt +13:33:15 waited 45s for global r=327 +13:33:25 global r=327 load 4.343 -> 4.291 +13:33:58 step 107560 loss=3.4325 lr=7.50e-05 tok=14.124B dt=5455 wait=185 +13:35:01 step 107580 loss=3.3253 lr=7.50e-05 tok=14.127B dt=5519 wait=186 +13:36:05 step 107600 loss=3.3762 lr=7.50e-05 tok=14.130B dt=5583 wait=186 +13:37:09 step 107620 loss=3.4515 lr=7.50e-05 tok=14.132B dt=5647 wait=186 +13:38:13 step 107640 loss=3.3821 lr=7.49e-05 tok=14.135B dt=5711 wait=187 +13:39:17 step 107660 loss=3.4452 lr=7.49e-05 tok=14.137B dt=5774 wait=187 +13:40:21 step 107680 loss=3.3878 lr=7.49e-05 tok=14.140B dt=5838 wait=187 +13:41:25 step 107700 loss=3.4243 lr=7.49e-05 tok=14.143B dt=5902 wait=188 +13:41:41 pushed ckpt_w81.pt,log_w81.txt +13:42:41 waited 61s for global r=328 +13:42:49 global r=328 load 4.311 -> 4.315 +13:43:53 step 107720 loss=3.3928 lr=7.49e-05 tok=14.145B dt=6051 wait=188 +13:44:58 step 107740 loss=3.3930 lr=7.48e-05 tok=14.148B dt=6115 wait=188 +13:46:01 step 107760 loss=3.4202 lr=7.48e-05 tok=14.151B dt=6179 wait=189 +13:47:05 step 107780 loss=3.3378 lr=7.48e-05 tok=14.153B dt=6242 wait=189 +13:48:08 step 107800 loss=3.3720 lr=7.48e-05 tok=14.156B dt=6306 wait=189 +13:49:12 step 107820 loss=3.4739 lr=7.48e-05 tok=14.158B dt=6370 wait=189 +13:50:16 step 107840 loss=3.3813 lr=7.47e-05 tok=14.161B dt=6433 wait=190 +13:50:59 pushed ckpt_w81.pt,log_w81.txt +13:52:15 waited 76s for global r=329 +13:52:22 global r=329 load 4.290 -> 4.297 +13:52:55 step 107860 loss=3.4549 lr=7.47e-05 tok=14.164B dt=6593 wait=190 +13:53:59 step 107880 loss=3.3175 lr=7.47e-05 tok=14.166B dt=6656 wait=190 +13:55:03 step 107900 loss=3.3303 lr=7.47e-05 tok=14.169B dt=6720 wait=191 +13:56:06 step 107920 loss=3.4762 lr=7.47e-05 tok=14.172B dt=6784 wait=191 +13:57:10 step 107940 loss=3.2998 lr=7.46e-05 tok=14.174B dt=6848 wait=191 +13:58:14 step 107960 loss=3.4565 lr=7.46e-05 tok=14.177B dt=6912 wait=192 +13:59:18 step 107980 loss=3.3752 lr=7.46e-05 tok=14.179B dt=6975 wait=192 +14:00:22 step 108000 loss=3.4701 lr=7.46e-05 tok=14.182B dt=7039 wait=192 +14:00:37 pushed ckpt_w81.pt,log_w81.txt +14:01:52 waited 76s for global r=330 +14:02:03 global r=330 load 4.282 -> 4.286 +14:03:08 step 108020 loss=3.3824 lr=7.45e-05 tok=14.185B dt=7205 wait=192 +14:04:11 step 108040 loss=3.4120 lr=7.45e-05 tok=14.187B dt=7269 wait=193 +14:05:15 step 108060 loss=3.4119 lr=7.45e-05 tok=14.190B dt=7333 wait=193 +14:06:19 step 108080 loss=3.3445 lr=7.45e-05 tok=14.192B dt=7397 wait=193 +14:07:22 step 108100 loss=3.3859 lr=7.45e-05 tok=14.195B dt=7460 wait=194 +14:08:26 step 108120 loss=3.4488 lr=7.44e-05 tok=14.198B dt=7524 wait=194 +14:09:29 step 108140 loss=3.3183 lr=7.44e-05 tok=14.200B dt=7587 wait=194 +14:10:13 pushed ckpt_w81.pt,log_w81.txt +14:11:14 waited 61s for global r=331 +14:11:25 global r=331 load 4.284 -> 4.301 +14:11:58 step 108160 loss=3.4411 lr=7.44e-05 tok=14.203B dt=7735 wait=195 +14:13:01 step 108180 loss=3.3369 lr=7.44e-05 tok=14.206B dt=7799 wait=195 +14:14:05 step 108200 loss=3.3425 lr=7.44e-05 tok=14.208B dt=7863 wait=195 +14:15:09 step 108220 loss=3.4555 lr=7.43e-05 tok=14.211B dt=7927 wait=196 +14:16:13 step 108240 loss=3.3145 lr=7.43e-05 tok=14.213B dt=7991 wait=196 +14:17:17 step 108260 loss=3.3752 lr=7.43e-05 tok=14.216B dt=8054 wait=196 +14:18:20 step 108280 loss=3.3811 lr=7.43e-05 tok=14.219B dt=8118 wait=197 +14:19:24 step 108300 loss=3.4024 lr=7.43e-05 tok=14.221B dt=8182 wait=197 +14:19:36 pushed ckpt_w81.pt,log_w81.txt +14:20:36 waited 61s for global r=332 +14:20:50 global r=332 load 4.298 -> 4.292 +14:21:55 step 108320 loss=3.3506 lr=7.42e-05 tok=14.224B dt=8333 wait=197 +14:22:59 step 108340 loss=3.4231 lr=7.42e-05 tok=14.227B dt=8397 wait=198 +14:24:03 step 108360 loss=3.3855 lr=7.42e-05 tok=14.229B dt=8461 wait=198 +14:25:07 step 108380 loss=3.3540 lr=7.42e-05 tok=14.232B dt=8524 wait=198 +14:26:10 step 108400 loss=3.3608 lr=7.41e-05 tok=14.234B dt=8588 wait=199 +14:27:14 step 108420 loss=3.4372 lr=7.41e-05 tok=14.237B dt=8652 wait=199 +14:28:18 step 108440 loss=3.3596 lr=7.41e-05 tok=14.240B dt=8716 wait=199 +14:29:02 pushed ckpt_w81.pt,log_w81.txt +14:30:17 waited 76s for global r=333 +14:30:33 global r=333 load 4.310 -> 4.294 +14:31:06 step 108460 loss=3.5079 lr=7.41e-05 tok=14.242B dt=8884 wait=199 +14:32:10 step 108480 loss=3.3002 lr=7.41e-05 tok=14.245B dt=8947 wait=200 +14:33:14 step 108500 loss=3.3467 lr=7.40e-05 tok=14.248B dt=9012 wait=200 +14:34:18 step 108520 loss=3.4894 lr=7.40e-05 tok=14.250B dt=9075 wait=200 +14:35:22 step 108540 loss=3.3491 lr=7.40e-05 tok=14.253B dt=9139 wait=201 +14:36:26 step 108560 loss=3.5318 lr=7.40e-05 tok=14.255B dt=9203 wait=201 +14:37:29 step 108580 loss=3.3755 lr=7.40e-05 tok=14.258B dt=9267 wait=201 +14:38:34 step 108600 loss=2.9629 lr=7.39e-05 tok=14.261B dt=9332 wait=202 +14:38:48 pushed ckpt_w81.pt,log_w81.txt +14:39:49 waited 61s for global r=334 +14:39:59 global r=334 load 4.314 -> 4.295 +14:41:31 step 108620 loss=3.3646 lr=7.39e-05 tok=14.263B dt=9508 wait=202 +14:43:02 step 108640 loss=3.3775 lr=7.39e-05 tok=14.266B dt=9599 wait=202 +14:44:34 step 108660 loss=3.4049 lr=7.39e-05 tok=14.268B dt=9692 wait=203 +14:46:05 step 108680 loss=3.3622 lr=7.39e-05 tok=14.271B dt=9783 wait=203 +14:47:34 step 108700 loss=3.3587 lr=7.38e-05 tok=14.274B dt=9872 wait=203 +14:49:07 step 108720 loss=3.4595 lr=7.38e-05 tok=14.276B dt=9965 wait=203 +14:50:40 step 108740 loss=3.3160 lr=7.38e-05 tok=14.279B dt=10057 wait=204 +14:51:38 pushed ckpt_w81.pt,log_w81.txt +14:51:38 waited 0s for global r=335 +14:51:45 global r=335 load 4.318 -> 4.310 +14:52:32 step 108760 loss=3.4594 lr=7.38e-05 tok=14.282B dt=10170 wait=204 +14:54:00 step 108780 loss=3.3448 lr=7.37e-05 tok=14.284B dt=10257 wait=204 +14:55:33 step 108800 loss=3.3425 lr=7.37e-05 tok=14.287B dt=10351 wait=205 +14:57:06 step 108820 loss=3.4540 lr=7.37e-05 tok=14.289B dt=10444 wait=205 +14:58:39 step 108840 loss=3.3348 lr=7.37e-05 tok=14.292B dt=10537 wait=205 +15:00:13 step 108860 loss=3.4908 lr=7.37e-05 tok=14.295B dt=10631 wait=206 +15:01:41 step 108880 loss=3.3801 lr=7.36e-05 tok=14.297B dt=10719 wait=206 +15:03:15 step 108900 loss=3.4469 lr=7.36e-05 tok=14.300B dt=10813 wait=206 +15:03:27 pushed ckpt_w81.pt,log_w81.txt +15:04:13 waited 45s for global r=336 +15:04:21 global r=336 load 4.298 -> 4.312 +15:05:56 step 108920 loss=3.3822 lr=7.36e-05 tok=14.303B dt=10973 wait=207 +15:07:29 step 108940 loss=3.3902 lr=7.36e-05 tok=14.305B dt=11067 wait=207 +15:08:58 step 108960 loss=3.4634 lr=7.36e-05 tok=14.308B dt=11156 wait=207 +15:10:32 step 108980 loss=3.3722 lr=7.35e-05 tok=14.310B dt=11249 wait=207 +15:12:05 step 109000 loss=3.4236 lr=7.35e-05 tok=14.313B dt=11343 wait=208 +15:13:40 step 109020 loss=3.4661 lr=7.35e-05 tok=14.316B dt=11437 wait=208 +15:15:14 step 109040 loss=3.3312 lr=7.35e-05 tok=14.318B dt=11532 wait=208 +15:16:13 pushed ckpt_w81.pt,log_w81.txt +15:16:59 waited 45s for global r=337 +15:17:06 global r=337 load 4.293 -> 4.308 +15:17:39 step 109060 loss=3.4459 lr=7.35e-05 tok=14.321B dt=11676 wait=209 +15:18:43 step 109080 loss=3.3228 lr=7.34e-05 tok=14.324B dt=11741 wait=209 +15:19:46 step 109100 loss=3.3493 lr=7.34e-05 tok=14.326B dt=11804 wait=209 +15:20:51 step 109120 loss=3.4635 lr=7.34e-05 tok=14.329B dt=11868 wait=210 +15:21:54 step 109140 loss=3.3286 lr=7.34e-05 tok=14.331B dt=11932 wait=210 +15:22:58 step 109160 loss=3.4450 lr=7.33e-05 tok=14.334B dt=11996 wait=210 +15:24:02 step 109180 loss=3.3634 lr=7.33e-05 tok=14.337B dt=12059 wait=210 +15:25:06 step 109200 loss=3.4372 lr=7.33e-05 tok=14.339B dt=12123 wait=211 +15:25:18 pushed ckpt_w81.pt,log_w81.txt +15:29:03 global r=338 load 4.325 -> 4.298 +15:30:06 step 109220 loss=3.3832 lr=7.33e-05 tok=14.343B dt=12424 wait=211 +15:31:10 step 109240 loss=3.4328 lr=7.33e-05 tok=14.346B dt=12487 wait=212 +15:32:14 step 109260 loss=3.3675 lr=7.32e-05 tok=14.348B dt=12552 wait=212 +15:33:18 step 109280 loss=3.3385 lr=7.32e-05 tok=14.351B dt=12615 wait=212 +15:34:21 step 109300 loss=3.3674 lr=7.32e-05 tok=14.354B dt=12679 wait=212 +15:35:25 step 109320 loss=3.4576 lr=7.32e-05 tok=14.356B dt=12743 wait=213 +15:36:29 step 109340 loss=3.3425 lr=7.32e-05 tok=14.359B dt=12807 wait=213 +15:37:15 pushed ckpt_w81.pt,log_w81.txt +15:40:16 waited 181s for global r=339 +15:40:25 global r=339 load 4.302 -> 4.285 +15:40:58 step 109360 loss=3.4718 lr=7.31e-05 tok=14.362B dt=13076 wait=213 +15:42:02 step 109380 loss=3.3732 lr=7.31e-05 tok=14.364B dt=13140 wait=214 +15:43:06 step 109400 loss=3.3487 lr=7.31e-05 tok=14.367B dt=13203 wait=214 +15:44:10 step 109420 loss=3.4950 lr=7.31e-05 tok=14.369B dt=13267 wait=214 +15:45:14 step 109440 loss=3.3648 lr=7.31e-05 tok=14.372B dt=13331 wait=215 +15:46:17 step 109460 loss=3.4442 lr=7.30e-05 tok=14.375B dt=13395 wait=215 +15:47:21 step 109480 loss=3.3214 lr=7.30e-05 tok=14.377B dt=13459 wait=215 +15:48:25 step 109500 loss=3.4279 lr=7.30e-05 tok=14.380B dt=13523 wait=216 +15:48:37 pushed ckpt_w81.pt,log_w81.txt +15:52:20 global r=340 load 4.303 -> 4.283 +15:53:23 step 109520 loss=3.3781 lr=7.30e-05 tok=14.384B dt=13821 wait=216 +15:54:27 step 109540 loss=3.4271 lr=7.29e-05 tok=14.386B dt=13885 wait=216 +15:55:31 step 109560 loss=3.4155 lr=7.29e-05 tok=14.389B dt=13949 wait=217 +15:56:35 step 109580 loss=3.3061 lr=7.29e-05 tok=14.392B dt=14013 wait=217 +15:57:39 step 109600 loss=3.3106 lr=7.29e-05 tok=14.394B dt=14076 wait=217 +15:58:43 step 109620 loss=3.4362 lr=7.29e-05 tok=14.397B dt=14140 wait=218 +15:59:46 step 109640 loss=3.3752 lr=7.28e-05 tok=14.400B dt=14204 wait=218 +16:00:35 pushed ckpt_w81.pt,log_w81.txt +16:03:36 waited 181s for global r=341 +16:03:47 global r=341 load 4.301 -> 4.289 +16:04:21 step 109660 loss=3.4735 lr=7.28e-05 tok=14.402B dt=14479 wait=218 +16:05:25 step 109680 loss=3.3749 lr=7.28e-05 tok=14.405B dt=14543 wait=219 +16:06:29 step 109700 loss=3.3559 lr=7.28e-05 tok=14.407B dt=14607 wait=219 +16:07:33 step 109720 loss=3.4437 lr=7.28e-05 tok=14.410B dt=14670 wait=219 +16:08:37 step 109740 loss=3.3838 lr=7.27e-05 tok=14.413B dt=14734 wait=219 +16:09:40 step 109760 loss=3.4486 lr=7.27e-05 tok=14.415B dt=14798 wait=220 +16:10:44 step 109780 loss=3.3506 lr=7.27e-05 tok=14.418B dt=14861 wait=220 +16:11:47 step 109800 loss=3.3410 lr=7.27e-05 tok=14.421B dt=14925 wait=220 +16:11:59 pushed ckpt_w81.pt,log_w81.txt +16:13:30 waited 91s for global r=342 +16:13:38 global r=342 load 4.310 -> 4.273 +16:14:44 step 109820 loss=3.2061 lr=7.27e-05 tok=14.423B dt=15102 wait=221 +16:15:48 step 109840 loss=3.4199 lr=7.26e-05 tok=14.426B dt=15166 wait=221 +16:16:52 step 109860 loss=3.3802 lr=7.26e-05 tok=14.428B dt=15229 wait=221 +16:17:56 step 109880 loss=3.3010 lr=7.26e-05 tok=14.431B dt=15293 wait=222 +16:18:59 step 109900 loss=3.3422 lr=7.26e-05 tok=14.434B dt=15357 wait=222 +16:20:03 step 109920 loss=3.4765 lr=7.25e-05 tok=14.436B dt=15421 wait=222 +16:21:07 step 109940 loss=3.3426 lr=7.25e-05 tok=14.439B dt=15485 wait=223 +16:21:52 pushed ckpt_w81.pt,log_w81.txt +16:23:23 waited 91s for global r=343 +16:23:34 global r=343 load 4.307 -> 4.281 +16:24:07 step 109960 loss=3.4394 lr=7.25e-05 tok=14.442B dt=15665 wait=223 +16:25:11 step 109980 loss=3.3698 lr=7.25e-05 tok=14.444B dt=15728 wait=223 +16:26:15 step 110000 loss=3.2666 lr=7.25e-05 tok=14.447B dt=15792 wait=224 +16:27:19 step 110020 loss=3.3947 lr=7.24e-05 tok=14.449B dt=15856 wait=224 +16:28:23 step 110040 loss=3.3790 lr=7.24e-05 tok=14.452B dt=15920 wait=224 +16:29:26 step 110060 loss=3.4408 lr=7.24e-05 tok=14.455B dt=15984 wait=224 +16:30:30 step 110080 loss=3.3188 lr=7.24e-05 tok=14.457B dt=16048 wait=225 +16:31:34 step 110100 loss=3.4071 lr=7.24e-05 tok=14.460B dt=16111 wait=225 +16:31:46 pushed ckpt_w81.pt,log_w81.txt +16:33:16 waited 91s for global r=344 +16:33:23 global r=344 load 4.297 -> 4.295 +16:34:29 step 110120 loss=3.3619 lr=7.23e-05 tok=14.462B dt=16286 wait=225 +16:35:33 step 110140 loss=3.3339 lr=7.23e-05 tok=14.465B dt=16350 wait=226 +16:36:37 step 110160 loss=3.4012 lr=7.23e-05 tok=14.468B dt=16414 wait=226 +16:37:40 step 110180 loss=3.3210 lr=7.23e-05 tok=14.470B dt=16478 wait=226 +16:38:44 step 110200 loss=3.3937 lr=7.23e-05 tok=14.473B dt=16542 wait=227 +16:39:48 step 110220 loss=3.4355 lr=7.22e-05 tok=14.476B dt=16606 wait=227 +16:40:52 step 110240 loss=3.3471 lr=7.22e-05 tok=14.478B dt=16669 wait=227 +16:41:36 pushed ckpt_w81.pt,log_w81.txt +16:42:36 waited 60s for global r=345 +16:42:42 global r=345 load 4.309 -> 4.309 +16:43:15 step 110260 loss=3.4782 lr=7.22e-05 tok=14.481B dt=16813 wait=228 +16:44:19 step 110280 loss=3.3787 lr=7.22e-05 tok=14.483B dt=16877 wait=228 +16:45:23 step 110300 loss=3.3504 lr=7.22e-05 tok=14.486B dt=16941 wait=228 +16:46:27 step 110320 loss=3.4225 lr=7.21e-05 tok=14.489B dt=17005 wait=229 +16:47:31 step 110340 loss=3.3034 lr=7.21e-05 tok=14.491B dt=17068 wait=229 +16:48:34 step 110360 loss=3.4090 lr=7.21e-05 tok=14.494B dt=17132 wait=229 +16:49:38 step 110380 loss=3.3400 lr=7.21e-05 tok=14.497B dt=17196 wait=230 +16:50:42 step 110400 loss=3.3864 lr=7.20e-05 tok=14.499B dt=17259 wait=230 +16:50:57 pushed ckpt_w81.pt,log_w81.txt +16:52:12 waited 76s for global r=346 +16:52:19 global r=346 load 4.295 -> 4.281 +16:53:24 step 110420 loss=3.3231 lr=7.20e-05 tok=14.502B dt=17422 wait=230 +16:54:28 step 110440 loss=3.4557 lr=7.20e-05 tok=14.504B dt=17486 wait=230 +16:55:32 step 110460 loss=3.3533 lr=7.20e-05 tok=14.507B dt=17550 wait=231 +16:56:36 step 110480 loss=3.3986 lr=7.20e-05 tok=14.510B dt=17614 wait=231 +16:57:40 step 110500 loss=3.3786 lr=7.19e-05 tok=14.512B dt=17677 wait=231 +16:58:44 step 110520 loss=3.4509 lr=7.19e-05 tok=14.515B dt=17741 wait=232 +16:59:47 step 110540 loss=3.3103 lr=7.19e-05 tok=14.518B dt=17805 wait=232 +17:00:32 pushed ckpt_w81.pt,log_w81.txt +17:01:32 waited 60s for global r=347 +17:01:39 global r=347 load 4.277 -> 4.269 +17:02:12 step 110560 loss=3.4299 lr=7.19e-05 tok=14.520B dt=17949 wait=232 +17:03:16 step 110580 loss=3.2987 lr=7.19e-05 tok=14.523B dt=18013 wait=233 +17:04:20 step 110600 loss=3.3179 lr=7.18e-05 tok=14.525B dt=18077 wait=233 +17:05:23 step 110620 loss=3.4706 lr=7.18e-05 tok=14.528B dt=18141 wait=233 +17:06:27 step 110640 loss=3.3227 lr=7.18e-05 tok=14.531B dt=18204 wait=234 +17:07:31 step 110660 loss=3.4575 lr=7.18e-05 tok=14.533B dt=18269 wait=234 +17:08:34 step 110680 loss=3.3413 lr=7.18e-05 tok=14.536B dt=18332 wait=234 +17:09:38 step 110700 loss=3.4093 lr=7.17e-05 tok=14.539B dt=18396 wait=235 +17:09:50 pushed ckpt_w81.pt,log_w81.txt +17:11:36 waited 106s for global r=348 +17:11:41 global r=348 load 4.294 -> 4.297 +17:12:46 step 110720 loss=3.3419 lr=7.17e-05 tok=14.541B dt=18583 wait=235 +17:13:50 step 110740 loss=3.3788 lr=7.17e-05 tok=14.544B dt=18647 wait=235 +17:14:53 step 110760 loss=3.3675 lr=7.17e-05 tok=14.546B dt=18711 wait=235 +17:15:57 step 110780 loss=3.3214 lr=7.16e-05 tok=14.549B dt=18775 wait=236 +17:17:01 step 110800 loss=3.3818 lr=7.16e-05 tok=14.552B dt=18839 wait=236 +17:18:05 step 110820 loss=3.4237 lr=7.16e-05 tok=14.554B dt=18903 wait=236 +17:19:08 step 110840 loss=3.3139 lr=7.16e-05 tok=14.557B dt=18966 wait=237 +17:19:55 pushed ckpt_w81.pt,log_w81.txt +17:21:10 waited 76s for global r=349 +17:21:20 global r=349 load 4.296 -> 4.284 +17:21:53 step 110860 loss=3.4532 lr=7.16e-05 tok=14.559B dt=19131 wait=237 +17:22:57 step 110880 loss=3.3440 lr=7.15e-05 tok=14.562B dt=19195 wait=237 +17:24:01 step 110900 loss=3.3771 lr=7.15e-05 tok=14.565B dt=19259 wait=238 +17:25:05 step 110920 loss=3.4171 lr=7.15e-05 tok=14.567B dt=19323 wait=238 +17:26:09 step 110940 loss=3.2754 lr=7.15e-05 tok=14.570B dt=19387 wait=238 +17:27:13 step 110960 loss=3.3792 lr=7.15e-05 tok=14.573B dt=19451 wait=239 +17:28:17 step 110980 loss=3.3547 lr=7.14e-05 tok=14.575B dt=19514 wait=239 +17:29:20 step 111000 loss=3.3683 lr=7.14e-05 tok=14.578B dt=19578 wait=239 +17:29:32 pushed ckpt_w81.pt,log_w81.txt +17:30:48 waited 76s for global r=350 +17:30:53 global r=350 load 4.292 -> 4.281 +17:31:58 step 111020 loss=3.3589 lr=7.14e-05 tok=14.580B dt=19736 wait=240 +17:33:02 step 111040 loss=3.3474 lr=7.14e-05 tok=14.583B dt=19800 wait=240 +17:34:06 step 111060 loss=3.3856 lr=7.14e-05 tok=14.586B dt=19864 wait=240 +17:35:10 step 111080 loss=3.3152 lr=7.13e-05 tok=14.588B dt=19928 wait=241 +17:36:14 step 111100 loss=3.3278 lr=7.13e-05 tok=14.591B dt=19991 wait=241 +17:37:18 step 111120 loss=3.5026 lr=7.13e-05 tok=14.594B dt=20055 wait=241 +17:38:22 step 111140 loss=3.3643 lr=7.13e-05 tok=14.596B dt=20119 wait=242 +17:39:05 pushed ckpt_w81.pt,log_w81.txt +17:40:21 waited 76s for global r=351 +17:40:26 global r=351 load 4.287 -> 4.275 +17:40:59 step 111160 loss=3.3903 lr=7.13e-05 tok=14.599B dt=20277 wait=242 +17:42:03 step 111180 loss=3.3506 lr=7.12e-05 tok=14.601B dt=20340 wait=242 +17:43:07 step 111200 loss=3.3505 lr=7.12e-05 tok=14.604B dt=20404 wait=242 +17:44:10 step 111220 loss=3.4570 lr=7.12e-05 tok=14.607B dt=20468 wait=243 +17:45:14 step 111240 loss=3.3766 lr=7.12e-05 tok=14.609B dt=20532 wait=243 +17:46:18 step 111260 loss=3.4757 lr=7.11e-05 tok=14.612B dt=20595 wait=243 +17:47:22 step 111280 loss=3.3774 lr=7.11e-05 tok=14.615B dt=20659 wait=244 +17:48:26 step 111300 loss=3.4160 lr=7.11e-05 tok=14.617B dt=20724 wait=244 +17:48:40 pushed ckpt_w81.pt,log_w81.txt +17:49:55 waited 76s for global r=352 +17:50:04 global r=352 load 4.302 -> 4.294 +17:51:08 step 111320 loss=3.3801 lr=7.11e-05 tok=14.620B dt=20886 wait=244 +17:52:12 step 111340 loss=3.3696 lr=7.11e-05 tok=14.622B dt=20950 wait=245 +17:53:17 step 111360 loss=3.4091 lr=7.10e-05 tok=14.625B dt=21014 wait=245 +17:54:21 step 111380 loss=3.3333 lr=7.10e-05 tok=14.628B dt=21078 wait=245 +17:55:24 step 111400 loss=3.3598 lr=7.10e-05 tok=14.630B dt=21142 wait=246 +17:56:28 step 111420 loss=3.4295 lr=7.10e-05 tok=14.633B dt=21206 wait=246 +17:57:31 step 111440 loss=3.3143 lr=7.10e-05 tok=14.635B dt=21269 wait=246 diff --git a/log_w82.txt b/log_w82.txt new file mode 100644 index 0000000000000000000000000000000000000000..48e48dce3ad9862a0606bde46ffd27cd2c396e57 --- /dev/null +++ b/log_w82.txt @@ -0,0 +1,19 @@ +launch +12:00:37 name=w82 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 29/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:00:43 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:00:43 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:00:53 params 1.08B +12:00:54 resumed step=106350 seen=13.96B round=319 +12:00:54 stream salt=106350 +12:03:02 blocks compiled +12:03:02 start: accum=4, 131072 tokens per optimiser step +12:06:27 step 106351 loss=3.1940 lr=7.63e-05 tok=13.961B dt=205 wait=168 +12:06:31 step 106352 loss=3.2232 lr=7.63e-05 tok=13.961B dt=209 wait=168 +12:06:59 step 106360 loss=3.5255 lr=7.63e-05 tok=13.962B dt=236 wait=168 +12:08:07 step 106380 loss=3.4425 lr=7.63e-05 tok=13.964B dt=305 wait=169 +12:09:12 step 106400 loss=3.5005 lr=7.63e-05 tok=13.967B dt=370 wait=169 +12:10:16 step 106420 loss=3.4538 lr=7.62e-05 tok=13.970B dt=434 wait=169 +12:11:20 step 106440 loss=3.5292 lr=7.62e-05 tok=13.972B dt=498 wait=169 +12:12:24 step 106460 loss=3.6148 lr=7.62e-05 tok=13.975B dt=562 wait=170 +12:13:29 step 106480 loss=3.4501 lr=7.62e-05 tok=13.978B dt=626 wait=170 +12:14:33 step 106500 loss=3.5412 lr=7.61e-05 tok=13.980B dt=690 wait=170 diff --git a/log_w84.txt b/log_w84.txt new file mode 100644 index 0000000000000000000000000000000000000000..5fcdffcc3109ff7d832a3d0d8a4eab4990f6cda3 --- /dev/null +++ b/log_w84.txt @@ -0,0 +1,20 @@ +launch +12:46:35 name=w84 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 31/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +12:46:41 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +12:46:41 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +12:46:51 params 1.08B +12:46:52 resumed step=106800 seen=14.02B round=322 +12:46:52 stream salt=106800 +12:48:58 blocks compiled +12:48:58 start: accum=4, 131072 tokens per optimiser step +12:52:27 step 106801 loss=3.1229 lr=7.58e-05 tok=14.021B dt=209 wait=172 +12:52:31 step 106802 loss=3.0524 lr=7.58e-05 tok=14.021B dt=213 wait=172 +12:53:16 global r=323 load 4.312 -> 4.300 +12:53:52 step 106960 loss=3.4850 lr=7.57e-05 tok=14.023B dt=294 wait=173 +12:54:57 step 106980 loss=3.4360 lr=7.56e-05 tok=14.026B dt=359 wait=173 +12:56:01 step 107000 loss=3.3987 lr=7.56e-05 tok=14.029B dt=423 wait=173 +12:57:04 step 107020 loss=3.5006 lr=7.56e-05 tok=14.031B dt=486 wait=174 +12:58:08 step 107040 loss=3.3339 lr=7.56e-05 tok=14.034B dt=550 wait=174 +12:59:11 step 107060 loss=3.4315 lr=7.56e-05 tok=14.037B dt=614 wait=174 +13:00:15 step 107080 loss=3.4151 lr=7.55e-05 tok=14.039B dt=677 wait=174 +13:01:18 step 107100 loss=3.4042 lr=7.55e-05 tok=14.042B dt=741 wait=175 diff --git a/log_w86.txt b/log_w86.txt new file mode 100644 index 0000000000000000000000000000000000000000..65d2b215d136b9353c6bb6152e609d233db6cc56 --- /dev/null +++ b/log_w86.txt @@ -0,0 +1,179 @@ +launch +15:07:34 name=w86 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 33/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:07:45 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:07:45 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:07:54 params 1.08B +15:07:56 resumed step=108900 seen=14.30B round=336 +15:07:56 stream salt=108900 +15:10:32 blocks compiled +15:10:32 start: accum=4, 131072 tokens per optimiser step +15:14:29 step 108901 loss=3.2626 lr=7.36e-05 tok=14.301B dt=237 wait=197 +15:14:33 step 108902 loss=3.2011 lr=7.36e-05 tok=14.302B dt=241 wait=197 +15:15:36 step 108920 loss=3.5056 lr=7.36e-05 tok=14.304B dt=304 wait=197 +15:16:41 step 108940 loss=3.4386 lr=7.36e-05 tok=14.307B dt=370 wait=198 +15:17:28 global r=337 load 4.314 -> 4.308 +15:17:59 step 109060 loss=3.4639 lr=7.35e-05 tok=14.309B dt=448 wait=198 +15:19:11 step 109080 loss=3.4094 lr=7.34e-05 tok=14.312B dt=519 wait=198 +15:20:28 step 109100 loss=3.4539 lr=7.34e-05 tok=14.314B dt=596 wait=199 +15:21:33 step 109120 loss=3.4832 lr=7.34e-05 tok=14.317B dt=661 wait=199 +15:22:37 step 109140 loss=3.3838 lr=7.34e-05 tok=14.320B dt=725 wait=199 +15:23:41 step 109160 loss=3.5221 lr=7.33e-05 tok=14.322B dt=789 wait=200 +15:24:45 step 109180 loss=3.4008 lr=7.33e-05 tok=14.325B dt=853 wait=200 +15:25:50 step 109200 loss=3.4120 lr=7.33e-05 tok=14.327B dt=918 wait=200 +15:26:08 pushed ckpt_w86.pt,log_w86.txt +15:28:25 waited 136s for global r=338 +15:28:34 global r=338 load 4.311 -> 4.298 +15:29:40 step 109220 loss=3.3077 lr=7.33e-05 tok=14.330B dt=1148 wait=201 +15:30:57 step 109240 loss=3.4721 lr=7.33e-05 tok=14.333B dt=1225 wait=201 +15:32:12 step 109260 loss=3.4022 lr=7.32e-05 tok=14.335B dt=1301 wait=201 +15:33:17 step 109280 loss=3.4015 lr=7.32e-05 tok=14.338B dt=1365 wait=202 +15:34:21 step 109300 loss=3.4106 lr=7.32e-05 tok=14.341B dt=1429 wait=202 +15:35:25 step 109320 loss=3.4770 lr=7.32e-05 tok=14.343B dt=1493 wait=202 +15:36:29 step 109340 loss=3.3823 lr=7.32e-05 tok=14.346B dt=1557 wait=203 +15:37:14 pushed ckpt_w86.pt,log_w86.txt +15:40:15 waited 181s for global r=339 +15:40:23 global r=339 load 4.281 -> 4.285 +15:40:57 step 109360 loss=3.5053 lr=7.31e-05 tok=14.348B dt=1825 wait=203 +15:42:01 step 109380 loss=3.3682 lr=7.31e-05 tok=14.351B dt=1889 wait=203 +15:43:25 step 109400 loss=3.3603 lr=7.31e-05 tok=14.354B dt=1973 wait=204 +15:44:28 step 109420 loss=3.3989 lr=7.31e-05 tok=14.356B dt=2036 wait=204 +15:45:33 step 109440 loss=3.3919 lr=7.31e-05 tok=14.359B dt=2101 wait=204 +15:46:37 step 109460 loss=3.4459 lr=7.30e-05 tok=14.362B dt=2165 wait=205 +15:47:41 step 109480 loss=3.3826 lr=7.30e-05 tok=14.364B dt=2229 wait=205 +15:48:45 step 109500 loss=3.3938 lr=7.30e-05 tok=14.367B dt=2293 wait=205 +15:48:58 pushed ckpt_w86.pt,log_w86.txt +15:51:59 waited 181s for global r=340 +15:52:08 global r=340 load 4.278 -> 4.283 +15:53:13 step 109520 loss=3.3647 lr=7.30e-05 tok=14.369B dt=2561 wait=206 +15:54:25 step 109540 loss=3.4248 lr=7.29e-05 tok=14.372B dt=2633 wait=206 +15:55:41 step 109560 loss=3.3725 lr=7.29e-05 tok=14.375B dt=2709 wait=207 +15:56:46 step 109580 loss=3.3455 lr=7.29e-05 tok=14.377B dt=2774 wait=207 +15:57:50 step 109600 loss=3.4048 lr=7.29e-05 tok=14.380B dt=2838 wait=207 +15:58:54 step 109620 loss=3.4440 lr=7.29e-05 tok=14.383B dt=2902 wait=208 +15:59:58 step 109640 loss=3.4146 lr=7.28e-05 tok=14.385B dt=2966 wait=208 +16:00:43 pushed ckpt_w86.pt,log_w86.txt +16:03:44 waited 182s for global r=341 +16:03:52 global r=341 load 4.275 -> 4.289 +16:04:25 step 109660 loss=3.4461 lr=7.28e-05 tok=14.388B dt=3233 wait=208 +16:05:35 step 109680 loss=3.3870 lr=7.28e-05 tok=14.390B dt=3303 wait=209 +16:06:53 step 109700 loss=3.4101 lr=7.28e-05 tok=14.393B dt=3381 wait=209 +16:07:58 step 109720 loss=3.4481 lr=7.28e-05 tok=14.396B dt=3446 wait=209 +16:09:02 step 109740 loss=3.3879 lr=7.27e-05 tok=14.398B dt=3510 wait=210 +16:10:07 step 109760 loss=3.4796 lr=7.27e-05 tok=14.401B dt=3575 wait=210 +16:11:11 step 109780 loss=3.4147 lr=7.27e-05 tok=14.404B dt=3639 wait=210 +16:12:17 step 109800 loss=3.4574 lr=7.27e-05 tok=14.406B dt=3705 wait=211 +16:12:29 pushed ckpt_w86.pt,log_w86.txt +16:13:29 waited 60s for global r=342 +16:13:38 global r=342 load 4.298 -> 4.273 +16:14:43 step 109820 loss=3.4056 lr=7.27e-05 tok=14.409B dt=3851 wait=211 +16:16:01 step 109840 loss=3.4332 lr=7.26e-05 tok=14.411B dt=3929 wait=212 +16:17:11 step 109860 loss=3.4473 lr=7.26e-05 tok=14.414B dt=3999 wait=212 +16:18:15 step 109880 loss=3.4637 lr=7.26e-05 tok=14.417B dt=4064 wait=212 +16:19:19 step 109900 loss=3.4291 lr=7.26e-05 tok=14.419B dt=4128 wait=213 +16:20:24 step 109920 loss=3.4793 lr=7.25e-05 tok=14.422B dt=4192 wait=213 +16:21:28 step 109940 loss=3.3732 lr=7.25e-05 tok=14.424B dt=4256 wait=213 +16:22:12 pushed ckpt_w86.pt,log_w86.txt +16:23:13 waited 61s for global r=343 +16:23:28 global r=343 load 4.285 -> 4.281 +16:24:02 step 109960 loss=3.4649 lr=7.25e-05 tok=14.427B dt=4410 wait=214 +16:25:13 step 109980 loss=3.3464 lr=7.25e-05 tok=14.430B dt=4481 wait=214 +16:26:29 step 110000 loss=3.3615 lr=7.25e-05 tok=14.432B dt=4557 wait=214 +16:27:33 step 110020 loss=3.4756 lr=7.24e-05 tok=14.435B dt=4621 wait=215 +16:28:37 step 110040 loss=3.3814 lr=7.24e-05 tok=14.438B dt=4685 wait=215 +16:29:41 step 110060 loss=3.4473 lr=7.24e-05 tok=14.440B dt=4749 wait=215 +16:30:45 step 110080 loss=3.3712 lr=7.24e-05 tok=14.443B dt=4813 wait=216 +16:31:49 step 110100 loss=3.4277 lr=7.24e-05 tok=14.445B dt=4877 wait=216 +16:32:01 pushed ckpt_w86.pt,log_w86.txt +16:33:16 waited 76s for global r=344 +16:33:22 global r=344 load 4.319 -> 4.295 +16:34:29 step 110120 loss=3.3777 lr=7.23e-05 tok=14.448B dt=5037 wait=216 +16:35:50 step 110140 loss=3.4160 lr=7.23e-05 tok=14.451B dt=5118 wait=217 +16:36:54 step 110160 loss=3.3715 lr=7.23e-05 tok=14.453B dt=5182 wait=217 +16:37:57 step 110180 loss=3.2938 lr=7.23e-05 tok=14.456B dt=5245 wait=217 +16:39:01 step 110200 loss=3.3597 lr=7.23e-05 tok=14.459B dt=5309 wait=218 +16:40:05 step 110220 loss=3.4429 lr=7.22e-05 tok=14.461B dt=5374 wait=218 +16:41:09 step 110240 loss=3.2702 lr=7.22e-05 tok=14.464B dt=5437 wait=218 +16:41:56 pushed ckpt_w86.pt,log_w86.txt +16:42:42 waited 45s for global r=345 +16:42:48 global r=345 load 4.363 -> 4.309 +16:43:21 step 110260 loss=3.4184 lr=7.22e-05 tok=14.466B dt=5569 wait=219 +16:44:32 step 110280 loss=3.3158 lr=7.22e-05 tok=14.469B dt=5640 wait=219 +16:45:47 step 110300 loss=3.3138 lr=7.22e-05 tok=14.472B dt=5715 wait=219 +16:46:51 step 110320 loss=3.4098 lr=7.21e-05 tok=14.474B dt=5779 wait=220 +16:47:54 step 110340 loss=3.3204 lr=7.21e-05 tok=14.477B dt=5842 wait=220 +16:48:58 step 110360 loss=3.4085 lr=7.21e-05 tok=14.480B dt=5906 wait=221 +16:50:01 step 110380 loss=3.3459 lr=7.21e-05 tok=14.482B dt=5969 wait=221 +16:51:05 step 110400 loss=3.4215 lr=7.20e-05 tok=14.485B dt=6033 wait=221 +16:51:17 pushed ckpt_w86.pt,log_w86.txt +16:52:03 waited 46s for global r=346 +16:52:10 global r=346 load 4.280 -> 4.281 +16:53:16 step 110420 loss=3.2956 lr=7.20e-05 tok=14.487B dt=6164 wait=222 +16:54:36 step 110440 loss=3.3881 lr=7.20e-05 tok=14.490B dt=6244 wait=222 +16:55:42 step 110460 loss=3.3432 lr=7.20e-05 tok=14.493B dt=6310 wait=222 +16:56:46 step 110480 loss=3.3517 lr=7.20e-05 tok=14.495B dt=6374 wait=223 +16:57:49 step 110500 loss=3.3357 lr=7.19e-05 tok=14.498B dt=6437 wait=223 +16:58:53 step 110520 loss=3.3589 lr=7.19e-05 tok=14.500B dt=6501 wait=223 +16:59:57 step 110540 loss=3.2954 lr=7.19e-05 tok=14.503B dt=6565 wait=224 +17:00:44 pushed ckpt_w86.pt,log_w86.txt +17:01:30 waited 45s for global r=347 +17:01:38 global r=347 load 4.291 -> 4.269 +17:02:11 step 110560 loss=3.4302 lr=7.19e-05 tok=14.506B dt=6699 wait=224 +17:03:24 step 110580 loss=2.8304 lr=7.19e-05 tok=14.508B dt=6772 wait=224 +17:04:41 step 110600 loss=3.3256 lr=7.18e-05 tok=14.511B dt=6849 wait=225 +17:05:45 step 110620 loss=3.2953 lr=7.18e-05 tok=14.514B dt=6913 wait=225 +17:07:00 step 110640 loss=3.1844 lr=7.18e-05 tok=14.516B dt=6988 wait=225 +17:08:22 step 110660 loss=3.2517 lr=7.18e-05 tok=14.519B dt=7070 wait=226 +17:09:37 step 110680 loss=3.3570 lr=7.18e-05 tok=14.521B dt=7145 wait=226 +17:10:41 step 110700 loss=3.3468 lr=7.17e-05 tok=14.524B dt=7209 wait=227 +17:10:54 pushed ckpt_w86.pt,log_w86.txt +17:11:39 waited 45s for global r=348 +17:11:46 global r=348 load 4.291 -> 4.297 +17:13:10 step 110720 loss=3.3364 lr=7.17e-05 tok=14.527B dt=7358 wait=227 +17:14:27 step 110740 loss=3.3766 lr=7.17e-05 tok=14.529B dt=7435 wait=227 +17:15:31 step 110760 loss=3.3708 lr=7.17e-05 tok=14.532B dt=7499 wait=228 +17:16:35 step 110780 loss=3.3435 lr=7.16e-05 tok=14.535B dt=7563 wait=228 +17:17:38 step 110800 loss=3.3507 lr=7.16e-05 tok=14.537B dt=7627 wait=228 +17:18:42 step 110820 loss=3.3865 lr=7.16e-05 tok=14.540B dt=7690 wait=229 +17:19:46 step 110840 loss=3.2693 lr=7.16e-05 tok=14.542B dt=7754 wait=229 +17:20:34 pushed ckpt_w86.pt,log_w86.txt +17:21:19 waited 45s for global r=349 +17:21:26 global r=349 load 4.311 -> 4.284 +17:22:10 step 110860 loss=3.4383 lr=7.16e-05 tok=14.545B dt=7898 wait=229 +17:23:27 step 110880 loss=3.3710 lr=7.15e-05 tok=14.548B dt=7975 wait=230 +17:24:30 step 110900 loss=3.3622 lr=7.15e-05 tok=14.550B dt=8038 wait=230 +17:25:34 step 110920 loss=3.3995 lr=7.15e-05 tok=14.553B dt=8102 wait=230 +17:26:37 step 110940 loss=3.3504 lr=7.15e-05 tok=14.556B dt=8166 wait=231 +17:27:41 step 110960 loss=3.4085 lr=7.15e-05 tok=14.558B dt=8229 wait=231 +17:28:45 step 110980 loss=3.3565 lr=7.14e-05 tok=14.561B dt=8293 wait=231 +17:29:48 step 111000 loss=3.4312 lr=7.14e-05 tok=14.563B dt=8356 wait=232 +17:30:03 pushed ckpt_w86.pt,log_w86.txt +17:30:48 waited 45s for global r=350 +17:30:55 global r=350 load 4.268 -> 4.281 +17:32:10 step 111020 loss=3.3546 lr=7.14e-05 tok=14.566B dt=8498 wait=232 +17:33:31 step 111040 loss=3.3435 lr=7.14e-05 tok=14.569B dt=8579 wait=232 +17:34:47 step 111060 loss=3.3746 lr=7.14e-05 tok=14.571B dt=8655 wait=233 +17:35:51 step 111080 loss=3.3324 lr=7.13e-05 tok=14.574B dt=8719 wait=233 +17:36:54 step 111100 loss=3.3538 lr=7.13e-05 tok=14.577B dt=8782 wait=233 +17:37:58 step 111120 loss=3.4212 lr=7.13e-05 tok=14.579B dt=8846 wait=234 +17:39:01 step 111140 loss=3.3324 lr=7.13e-05 tok=14.582B dt=8909 wait=234 +17:39:46 pushed ckpt_w86.pt,log_w86.txt +17:40:16 waited 30s for global r=351 +17:40:23 global r=351 load 4.278 -> 4.275 +17:41:02 step 111160 loss=3.4382 lr=7.13e-05 tok=14.584B dt=9030 wait=234 +17:42:21 step 111180 loss=3.3265 lr=7.12e-05 tok=14.587B dt=9109 wait=235 +17:43:44 step 111200 loss=3.3453 lr=7.12e-05 tok=14.590B dt=9192 wait=235 +17:44:48 step 111220 loss=3.3829 lr=7.12e-05 tok=14.592B dt=9256 wait=235 +17:45:52 step 111240 loss=3.3104 lr=7.12e-05 tok=14.595B dt=9320 wait=236 +17:46:56 step 111260 loss=3.3882 lr=7.11e-05 tok=14.597B dt=9384 wait=236 +17:47:59 step 111280 loss=3.3412 lr=7.11e-05 tok=14.600B dt=9447 wait=236 +17:49:03 step 111300 loss=3.3249 lr=7.11e-05 tok=14.603B dt=9511 wait=237 +17:49:15 pushed ckpt_w86.pt,log_w86.txt +17:50:01 waited 46s for global r=352 +17:50:11 global r=352 load 4.304 -> 4.294 +17:51:31 step 111320 loss=3.3001 lr=7.11e-05 tok=14.605B dt=9659 wait=237 +17:52:55 step 111340 loss=3.3911 lr=7.11e-05 tok=14.608B dt=9743 wait=237 +17:53:59 step 111360 loss=3.2209 lr=7.10e-05 tok=14.611B dt=9807 wait=238 +17:55:02 step 111380 loss=3.3026 lr=7.10e-05 tok=14.613B dt=9870 wait=238 +17:56:06 step 111400 loss=3.3531 lr=7.10e-05 tok=14.616B dt=9934 wait=238 +17:57:10 step 111420 loss=3.4466 lr=7.10e-05 tok=14.618B dt=9998 wait=239 +17:58:13 step 111440 loss=3.3126 lr=7.10e-05 tok=14.621B dt=10061 wait=239 diff --git a/log_w87.txt b/log_w87.txt new file mode 100644 index 0000000000000000000000000000000000000000..a433c07c31f050ea8cb5627cadb09f68252c6323 --- /dev/null +++ b/log_w87.txt @@ -0,0 +1,204 @@ +launch +15:07:29 name=w87 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 34/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:07:35 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:07:35 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:07:45 params 1.08B +15:07:46 resumed step=108900 seen=14.30B round=336 +15:07:46 stream salt=108900 +launch +15:09:57 name=w87 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 34/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:10:02 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:10:02 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:10:11 params 1.08B +15:10:13 resumed step=108900 seen=14.30B round=336 +15:10:13 stream salt=108900 +15:12:52 blocks compiled +15:12:52 start: accum=4, 131072 tokens per optimiser step +15:16:06 step 108901 loss=3.1011 lr=7.36e-05 tok=14.301B dt=194 wait=153 +15:16:10 step 108902 loss=3.2455 lr=7.36e-05 tok=14.302B dt=198 wait=153 +15:17:28 step 108920 loss=3.4383 lr=7.36e-05 tok=14.304B dt=276 wait=153 +15:17:41 global r=337 load 4.289 -> 4.308 +15:18:13 step 109060 loss=3.4164 lr=7.35e-05 tok=14.305B dt=321 wait=153 +15:19:18 step 109080 loss=3.4024 lr=7.34e-05 tok=14.308B dt=386 wait=154 +15:20:23 step 109100 loss=3.3763 lr=7.34e-05 tok=14.310B dt=451 wait=154 +15:21:28 step 109120 loss=3.4762 lr=7.34e-05 tok=14.313B dt=516 wait=154 +15:22:33 step 109140 loss=3.3924 lr=7.34e-05 tok=14.316B dt=581 wait=155 +15:23:38 step 109160 loss=3.4326 lr=7.33e-05 tok=14.318B dt=646 wait=155 +15:24:43 step 109180 loss=3.4034 lr=7.33e-05 tok=14.321B dt=711 wait=155 +15:25:49 step 109200 loss=3.4137 lr=7.33e-05 tok=14.324B dt=777 wait=156 +15:26:04 pushed ckpt_w87.pt,log_w87.txt +15:28:35 waited 151s for global r=338 +15:28:47 global r=338 load 4.327 -> 4.298 +15:29:54 step 109220 loss=3.3208 lr=7.33e-05 tok=14.326B dt=1023 wait=156 +15:31:00 step 109240 loss=3.4322 lr=7.33e-05 tok=14.329B dt=1088 wait=156 +15:32:05 step 109260 loss=3.3843 lr=7.32e-05 tok=14.331B dt=1153 wait=157 +15:33:09 step 109280 loss=3.3237 lr=7.32e-05 tok=14.334B dt=1217 wait=157 +15:34:14 step 109300 loss=3.3926 lr=7.32e-05 tok=14.337B dt=1282 wait=157 +15:35:19 step 109320 loss=3.4923 lr=7.32e-05 tok=14.339B dt=1347 wait=158 +15:36:24 step 109340 loss=3.4093 lr=7.32e-05 tok=14.342B dt=1413 wait=158 +15:37:11 pushed ckpt_w87.pt,log_w87.txt +15:40:47 step 109360 loss=3.5191 lr=7.31e-05 tok=14.345B dt=1675 wait=158 +15:40:58 global r=339 load 4.311 -> 4.285 +15:41:31 step 109360 loss=3.4375 lr=7.31e-05 tok=14.346B dt=1719 wait=159 +15:42:36 step 109380 loss=3.3349 lr=7.31e-05 tok=14.348B dt=1784 wait=159 +15:43:41 step 109400 loss=3.3542 lr=7.31e-05 tok=14.351B dt=1849 wait=159 +15:44:46 step 109420 loss=3.4407 lr=7.31e-05 tok=14.354B dt=1914 wait=160 +15:45:51 step 109440 loss=3.2524 lr=7.31e-05 tok=14.356B dt=1979 wait=160 +15:46:56 step 109460 loss=3.4758 lr=7.30e-05 tok=14.359B dt=2044 wait=160 +15:48:01 step 109480 loss=3.3884 lr=7.30e-05 tok=14.362B dt=2109 wait=161 +15:49:06 step 109500 loss=3.3767 lr=7.30e-05 tok=14.364B dt=2174 wait=161 +15:49:18 pushed ckpt_w87.pt,log_w87.txt +15:51:49 waited 151s for global r=340 +15:51:59 global r=340 load 4.280 -> 4.283 +15:53:06 step 109520 loss=3.3718 lr=7.30e-05 tok=14.367B dt=2414 wait=161 +15:54:11 step 109540 loss=3.3822 lr=7.29e-05 tok=14.369B dt=2479 wait=162 +15:55:16 step 109560 loss=3.3498 lr=7.29e-05 tok=14.372B dt=2544 wait=162 +15:56:21 step 109580 loss=3.3880 lr=7.29e-05 tok=14.375B dt=2609 wait=162 +15:57:26 step 109600 loss=3.3725 lr=7.29e-05 tok=14.377B dt=2674 wait=163 +15:58:31 step 109620 loss=3.4870 lr=7.29e-05 tok=14.380B dt=2739 wait=163 +15:59:36 step 109640 loss=3.3551 lr=7.28e-05 tok=14.383B dt=2804 wait=163 +16:00:24 pushed ckpt_w87.pt,log_w87.txt +16:04:00 step 109660 loss=3.4786 lr=7.28e-05 tok=14.385B dt=3068 wait=164 +16:04:08 global r=341 load 4.307 -> 4.289 +16:04:40 step 109660 loss=3.4669 lr=7.28e-05 tok=14.386B dt=3108 wait=164 +16:05:45 step 109680 loss=3.3165 lr=7.28e-05 tok=14.389B dt=3173 wait=164 +16:06:50 step 109700 loss=3.3531 lr=7.28e-05 tok=14.392B dt=3238 wait=165 +16:07:55 step 109720 loss=3.4480 lr=7.28e-05 tok=14.394B dt=3303 wait=165 +16:09:00 step 109740 loss=3.3479 lr=7.27e-05 tok=14.397B dt=3368 wait=165 +16:10:05 step 109760 loss=3.4121 lr=7.27e-05 tok=14.400B dt=3433 wait=166 +16:11:10 step 109780 loss=3.3001 lr=7.27e-05 tok=14.402B dt=3498 wait=166 +16:12:15 step 109800 loss=3.4047 lr=7.27e-05 tok=14.405B dt=3563 wait=166 +16:12:28 pushed ckpt_w87.pt,log_w87.txt +16:13:29 waited 61s for global r=342 +16:13:38 global r=342 load 4.268 -> 4.273 +16:14:44 step 109820 loss=3.3353 lr=7.27e-05 tok=14.407B dt=3712 wait=167 +16:15:49 step 109840 loss=3.3676 lr=7.26e-05 tok=14.410B dt=3777 wait=167 +16:16:54 step 109860 loss=3.3648 lr=7.26e-05 tok=14.413B dt=3843 wait=167 +16:17:59 step 109880 loss=3.3171 lr=7.26e-05 tok=14.415B dt=3907 wait=168 +16:19:04 step 109900 loss=3.3016 lr=7.26e-05 tok=14.418B dt=3972 wait=168 +16:20:09 step 109920 loss=3.4696 lr=7.25e-05 tok=14.421B dt=4037 wait=168 +16:21:15 step 109940 loss=3.3157 lr=7.25e-05 tok=14.423B dt=4103 wait=169 +16:22:00 pushed ckpt_w87.pt,log_w87.txt +16:23:16 waited 76s for global r=343 +16:23:30 global r=343 load 4.278 -> 4.281 +16:24:03 step 109960 loss=3.4478 lr=7.25e-05 tok=14.426B dt=4271 wait=169 +16:25:08 step 109980 loss=3.3367 lr=7.25e-05 tok=14.428B dt=4336 wait=169 +16:26:13 step 110000 loss=3.3475 lr=7.25e-05 tok=14.431B dt=4401 wait=170 +16:27:18 step 110020 loss=3.4576 lr=7.24e-05 tok=14.434B dt=4466 wait=170 +16:28:23 step 110040 loss=3.3774 lr=7.24e-05 tok=14.436B dt=4532 wait=170 +16:29:28 step 110060 loss=3.4374 lr=7.24e-05 tok=14.439B dt=4597 wait=171 +16:30:33 step 110080 loss=3.3221 lr=7.24e-05 tok=14.442B dt=4661 wait=171 +16:31:38 step 110100 loss=3.3545 lr=7.24e-05 tok=14.444B dt=4726 wait=171 +16:31:51 pushed ckpt_w87.pt,log_w87.txt +16:33:06 waited 76s for global r=344 +16:33:15 global r=344 load 4.301 -> 4.295 +16:34:21 step 110120 loss=3.3536 lr=7.23e-05 tok=14.447B dt=4889 wait=172 +16:35:26 step 110140 loss=3.3947 lr=7.23e-05 tok=14.449B dt=4954 wait=172 +16:36:31 step 110160 loss=3.4431 lr=7.23e-05 tok=14.452B dt=5019 wait=172 +16:37:36 step 110180 loss=3.3195 lr=7.23e-05 tok=14.455B dt=5084 wait=173 +16:38:41 step 110200 loss=3.3559 lr=7.23e-05 tok=14.457B dt=5149 wait=173 +16:39:46 step 110220 loss=3.4874 lr=7.22e-05 tok=14.460B dt=5214 wait=173 +16:40:51 step 110240 loss=3.2973 lr=7.22e-05 tok=14.462B dt=5279 wait=174 +16:41:36 pushed ckpt_w87.pt,log_w87.txt +16:42:36 waited 61s for global r=345 +16:42:43 global r=345 load 4.299 -> 4.309 +16:43:17 step 110260 loss=3.4702 lr=7.22e-05 tok=14.465B dt=5425 wait=174 +16:44:22 step 110280 loss=3.3121 lr=7.22e-05 tok=14.468B dt=5490 wait=174 +16:45:27 step 110300 loss=3.3206 lr=7.22e-05 tok=14.470B dt=5555 wait=175 +16:46:32 step 110320 loss=3.3860 lr=7.21e-05 tok=14.473B dt=5620 wait=175 +16:47:37 step 110340 loss=3.2725 lr=7.21e-05 tok=14.476B dt=5685 wait=175 +16:48:42 step 110360 loss=3.3876 lr=7.21e-05 tok=14.478B dt=5750 wait=176 +16:49:47 step 110380 loss=3.3875 lr=7.21e-05 tok=14.481B dt=5815 wait=176 +16:50:52 step 110400 loss=3.3834 lr=7.20e-05 tok=14.483B dt=5880 wait=177 +16:51:04 pushed ckpt_w87.pt,log_w87.txt +16:52:04 waited 61s for global r=346 +16:52:13 global r=346 load 4.317 -> 4.281 +16:53:19 step 110420 loss=3.3392 lr=7.20e-05 tok=14.486B dt=6027 wait=177 +16:54:24 step 110440 loss=3.4015 lr=7.20e-05 tok=14.489B dt=6092 wait=177 +16:55:30 step 110460 loss=3.4405 lr=7.20e-05 tok=14.491B dt=6158 wait=178 +16:56:34 step 110480 loss=3.3464 lr=7.20e-05 tok=14.494B dt=6222 wait=178 +16:57:40 step 110500 loss=3.3380 lr=7.19e-05 tok=14.497B dt=6288 wait=178 +16:58:45 step 110520 loss=3.4531 lr=7.19e-05 tok=14.499B dt=6353 wait=179 +16:59:50 step 110540 loss=3.2784 lr=7.19e-05 tok=14.502B dt=6418 wait=179 +17:00:35 pushed ckpt_w87.pt,log_w87.txt +17:01:35 waited 60s for global r=347 +17:01:46 global r=347 load 4.294 -> 4.269 +17:02:20 step 110560 loss=3.4228 lr=7.19e-05 tok=14.504B dt=6568 wait=179 +17:03:25 step 110580 loss=3.2388 lr=7.19e-05 tok=14.507B dt=6634 wait=180 +17:04:30 step 110600 loss=3.2981 lr=7.18e-05 tok=14.510B dt=6698 wait=180 +17:05:36 step 110620 loss=3.3300 lr=7.18e-05 tok=14.512B dt=6764 wait=180 +17:06:41 step 110640 loss=3.2984 lr=7.18e-05 tok=14.515B dt=6829 wait=181 +17:07:46 step 110660 loss=3.4230 lr=7.18e-05 tok=14.518B dt=6894 wait=181 +17:08:51 step 110680 loss=3.4012 lr=7.18e-05 tok=14.520B dt=6959 wait=181 +17:09:56 step 110700 loss=3.3504 lr=7.17e-05 tok=14.523B dt=7024 wait=182 +17:10:10 pushed ckpt_w87.pt,log_w87.txt +17:11:26 waited 76s for global r=348 +17:11:32 global r=348 load 4.361 -> 4.297 +17:12:39 step 110720 loss=3.3431 lr=7.17e-05 tok=14.525B dt=7187 wait=182 +17:13:44 step 110740 loss=3.3695 lr=7.17e-05 tok=14.528B dt=7252 wait=182 +17:14:49 step 110760 loss=3.3735 lr=7.17e-05 tok=14.531B dt=7317 wait=183 +17:15:54 step 110780 loss=3.3118 lr=7.16e-05 tok=14.533B dt=7382 wait=183 +17:16:59 step 110800 loss=3.3765 lr=7.16e-05 tok=14.536B dt=7447 wait=183 +17:18:04 step 110820 loss=3.4205 lr=7.16e-05 tok=14.539B dt=7512 wait=184 +17:19:09 step 110840 loss=3.3114 lr=7.16e-05 tok=14.541B dt=7577 wait=184 +17:19:54 pushed ckpt_w87.pt,log_w87.txt +17:21:10 waited 76s for global r=349 +17:21:18 global r=349 load 4.278 -> 4.284 +17:21:52 step 110860 loss=3.4754 lr=7.16e-05 tok=14.544B dt=7740 wait=184 +17:22:57 step 110880 loss=3.3294 lr=7.15e-05 tok=14.546B dt=7805 wait=185 +17:24:02 step 110900 loss=3.3582 lr=7.15e-05 tok=14.549B dt=7870 wait=185 +17:25:07 step 110920 loss=3.4403 lr=7.15e-05 tok=14.552B dt=7935 wait=185 +17:26:12 step 110940 loss=3.3094 lr=7.15e-05 tok=14.554B dt=8000 wait=186 +17:27:17 step 110960 loss=3.1591 lr=7.15e-05 tok=14.557B dt=8065 wait=186 +17:28:22 step 110980 loss=3.2803 lr=7.14e-05 tok=14.559B dt=8130 wait=186 +17:29:28 step 111000 loss=3.3947 lr=7.14e-05 tok=14.562B dt=8196 wait=187 +17:29:42 pushed ckpt_w87.pt,log_w87.txt +17:30:42 waited 61s for global r=350 +17:30:50 global r=350 load 4.301 -> 4.281 +17:31:56 step 111020 loss=3.3336 lr=7.14e-05 tok=14.565B dt=8344 wait=187 +17:33:01 step 111040 loss=3.3520 lr=7.14e-05 tok=14.567B dt=8409 wait=187 +17:34:06 step 111060 loss=3.3940 lr=7.14e-05 tok=14.570B dt=8474 wait=188 +17:35:11 step 111080 loss=3.3423 lr=7.13e-05 tok=14.573B dt=8539 wait=188 +17:36:16 step 111100 loss=3.2121 lr=7.13e-05 tok=14.575B dt=8604 wait=188 +17:37:21 step 111120 loss=3.4436 lr=7.13e-05 tok=14.578B dt=8669 wait=189 +17:38:27 step 111140 loss=3.2585 lr=7.13e-05 tok=14.580B dt=8735 wait=189 +17:39:14 pushed ckpt_w87.pt,log_w87.txt +17:40:29 waited 76s for global r=351 +17:40:38 global r=351 load 4.286 -> 4.275 +17:41:11 step 111160 loss=3.4758 lr=7.13e-05 tok=14.583B dt=8900 wait=189 +17:42:17 step 111180 loss=3.3240 lr=7.12e-05 tok=14.586B dt=8965 wait=190 +17:43:22 step 111200 loss=3.2807 lr=7.12e-05 tok=14.588B dt=9030 wait=190 +17:44:27 step 111220 loss=3.4431 lr=7.12e-05 tok=14.591B dt=9095 wait=190 +17:45:33 step 111240 loss=3.3177 lr=7.12e-05 tok=14.594B dt=9161 wait=191 +17:46:38 step 111260 loss=3.3315 lr=7.11e-05 tok=14.596B dt=9226 wait=191 +17:47:43 step 111280 loss=3.3526 lr=7.11e-05 tok=14.599B dt=9291 wait=191 +17:48:47 step 111300 loss=3.4168 lr=7.11e-05 tok=14.601B dt=9355 wait=192 +17:48:59 pushed ckpt_w87.pt,log_w87.txt +17:50:00 waited 61s for global r=352 +17:50:07 global r=352 load 4.303 -> 4.294 +17:51:13 step 111320 loss=3.3424 lr=7.11e-05 tok=14.604B dt=9501 wait=192 +17:52:18 step 111340 loss=3.3598 lr=7.11e-05 tok=14.607B dt=9566 wait=192 +17:53:23 step 111360 loss=3.3463 lr=7.10e-05 tok=14.609B dt=9631 wait=193 +17:54:28 step 111380 loss=3.3391 lr=7.10e-05 tok=14.612B dt=9696 wait=193 +17:55:33 step 111400 loss=3.3556 lr=7.10e-05 tok=14.615B dt=9761 wait=193 +17:56:39 step 111420 loss=3.4824 lr=7.10e-05 tok=14.617B dt=9827 wait=194 +launch +17:57:05 name=w87 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 34/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +17:57:10 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +17:57:10 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +17:57:20 params 1.08B +17:57:24 resumed step=111300 seen=14.60B round=352 +17:57:24 stream salt=111300 +17:59:13 blocks compiled +17:59:13 start: accum=4, 131072 tokens per optimiser step +18:01:29 step 111301 loss=3.1007 lr=7.11e-05 tok=14.602B dt=136 wait=112 +18:01:32 step 111302 loss=3.0960 lr=7.11e-05 tok=14.602B dt=139 wait=112 +18:02:12 global r=353 load 4.304 -> 4.289 +18:02:47 step 111460 loss=3.4144 lr=7.09e-05 tok=14.604B dt=213 wait=112 +18:03:52 step 111480 loss=3.3424 lr=7.09e-05 tok=14.607B dt=279 wait=113 +18:04:55 step 111500 loss=3.3247 lr=7.09e-05 tok=14.609B dt=342 wait=113 +18:05:59 step 111520 loss=3.4155 lr=7.09e-05 tok=14.612B dt=405 wait=114 +18:07:02 step 111540 loss=3.3423 lr=7.09e-05 tok=14.615B dt=469 wait=114 +18:08:06 step 111560 loss=3.4524 lr=7.08e-05 tok=14.617B dt=532 wait=114 +18:09:09 step 111580 loss=3.3638 lr=7.08e-05 tok=14.620B dt=596 wait=115 +18:10:12 step 111600 loss=3.3933 lr=7.08e-05 tok=14.622B dt=659 wait=115 diff --git a/log_w88.txt b/log_w88.txt new file mode 100644 index 0000000000000000000000000000000000000000..ad154db3aecd1b648f2872008cffe1472bed768e --- /dev/null +++ b/log_w88.txt @@ -0,0 +1,23 @@ +launch +15:07:28 name=w88 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 35/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +15:07:34 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +15:07:34 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +15:07:44 params 1.08B +15:07:45 resumed step=108900 seen=14.30B round=336 +15:07:45 stream salt=108900 +15:10:17 blocks compiled +15:10:17 start: accum=4, 131072 tokens per optimiser step +15:13:20 step 108901 loss=3.1692 lr=7.36e-05 tok=14.301B dt=183 wait=144 +15:13:23 step 108902 loss=3.2276 lr=7.36e-05 tok=14.302B dt=186 wait=144 +15:14:24 step 108920 loss=3.3704 lr=7.36e-05 tok=14.304B dt=247 wait=144 +15:15:32 step 108940 loss=3.3666 lr=7.36e-05 tok=14.307B dt=316 wait=145 +15:16:36 step 108960 loss=3.3736 lr=7.36e-05 tok=14.309B dt=379 wait=145 +15:17:20 global r=337 load 4.351 -> 4.308 +15:17:52 step 109060 loss=3.4013 lr=7.35e-05 tok=14.312B dt=455 wait=146 +15:18:55 step 109080 loss=3.3045 lr=7.34e-05 tok=14.314B dt=518 wait=146 +15:19:58 step 109100 loss=3.3328 lr=7.34e-05 tok=14.317B dt=581 wait=146 +15:21:01 step 109120 loss=3.4083 lr=7.34e-05 tok=14.320B dt=644 wait=147 +15:22:04 step 109140 loss=3.3192 lr=7.34e-05 tok=14.322B dt=707 wait=147 +15:23:07 step 109160 loss=3.4028 lr=7.33e-05 tok=14.325B dt=771 wait=147 +15:24:10 step 109180 loss=3.3717 lr=7.33e-05 tok=14.327B dt=834 wait=148 +15:25:14 step 109200 loss=3.3866 lr=7.33e-05 tok=14.330B dt=897 wait=148 diff --git a/log_w9.txt b/log_w9.txt new file mode 100644 index 0000000000000000000000000000000000000000..94bade8cb218e9f6e7a03e8b4541621477bbd52e --- /dev/null +++ b/log_w9.txt @@ -0,0 +1,70 @@ +launch +22:34:14 name=w9 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 1/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:34:20 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:34:29 params 1.08B +22:34:31 resumed step=32400 seen=4.25B round=16 +22:34:31 stream salt=32400 +22:36:35 start: accum=4, 131072 tokens per optimiser step +22:39:15 step 32420 loss=3.7400 lr=2.83e-04 tok=4.249B dt=160 +22:40:39 step 32440 loss=3.8560 lr=2.83e-04 tok=4.252B dt=244 +22:41:55 step 32460 loss=3.7739 lr=2.83e-04 tok=4.255B dt=320 +22:43:11 step 32480 loss=3.8048 lr=2.83e-04 tok=4.257B dt=395 +22:44:26 step 32500 loss=3.6839 lr=2.83e-04 tok=4.260B dt=471 +22:45:42 step 32520 loss=3.7687 lr=2.83e-04 tok=4.262B dt=547 +22:46:58 step 32540 loss=3.8004 lr=2.83e-04 tok=4.265B dt=623 +22:48:13 step 32560 loss=3.7760 lr=2.83e-04 tok=4.268B dt=698 +22:49:29 step 32580 loss=3.6705 lr=2.83e-04 tok=4.270B dt=774 +22:50:45 step 32600 loss=3.8042 lr=2.83e-04 tok=4.273B dt=850 +22:52:01 step 32620 loss=3.7252 lr=2.83e-04 tok=4.276B dt=925 +22:53:16 step 32640 loss=3.7166 lr=2.83e-04 tok=4.278B dt=1001 +22:54:32 step 32660 loss=3.7252 lr=2.83e-04 tok=4.281B dt=1077 +22:55:47 step 32680 loss=3.7254 lr=2.83e-04 tok=4.283B dt=1152 +22:57:03 step 32700 loss=3.7919 lr=2.83e-04 tok=4.286B dt=1228 +22:58:19 step 32720 loss=3.8713 lr=2.83e-04 tok=4.289B dt=1304 +22:59:35 step 32740 loss=3.7874 lr=2.83e-04 tok=4.291B dt=1379 +23:00:50 step 32760 loss=3.7580 lr=2.83e-04 tok=4.294B dt=1455 +23:02:06 step 32780 loss=3.8418 lr=2.83e-04 tok=4.297B dt=1531 +23:03:22 step 32800 loss=3.7979 lr=2.83e-04 tok=4.299B dt=1607 +23:04:38 step 32820 loss=3.7511 lr=2.83e-04 tok=4.302B dt=1682 +23:05:53 step 32840 loss=3.7031 lr=2.83e-04 tok=4.304B dt=1758 +23:07:09 step 32860 loss=3.7622 lr=2.83e-04 tok=4.307B dt=1834 +23:08:24 step 32880 loss=3.6618 lr=2.83e-04 tok=4.310B dt=1909 +23:09:40 step 32900 loss=3.8084 lr=2.83e-04 tok=4.312B dt=1985 +23:10:56 step 32920 loss=3.6931 lr=2.83e-04 tok=4.315B dt=2060 +23:12:11 step 32940 loss=3.6744 lr=2.83e-04 tok=4.318B dt=2136 +23:13:27 step 32960 loss=3.7531 lr=2.83e-04 tok=4.320B dt=2212 +23:14:42 step 32980 loss=3.6644 lr=2.83e-04 tok=4.323B dt=2287 +23:15:58 step 33000 loss=3.7914 lr=2.83e-04 tok=4.325B dt=2363 +23:16:11 pushed ckpt_w9.pt +23:16:11 pushed log_w9.txt +23:17:33 step 33020 loss=3.7941 lr=2.83e-04 tok=4.328B dt=2458 +23:18:48 step 33040 loss=3.7759 lr=2.83e-04 tok=4.331B dt=2533 +23:20:04 step 33060 loss=3.7100 lr=2.83e-04 tok=4.333B dt=2609 +23:21:20 step 33080 loss=3.6933 lr=2.83e-04 tok=4.336B dt=2685 +23:22:35 step 33100 loss=3.7116 lr=2.83e-04 tok=4.338B dt=2760 +23:23:51 step 33120 loss=3.7929 lr=2.83e-04 tok=4.341B dt=2836 +23:25:07 step 33140 loss=3.8492 lr=2.83e-04 tok=4.344B dt=2912 +23:26:23 step 33160 loss=3.8142 lr=2.83e-04 tok=4.346B dt=2987 +23:27:38 step 33180 loss=3.7566 lr=2.83e-04 tok=4.349B dt=3063 +23:28:54 step 33200 loss=3.7377 lr=2.83e-04 tok=4.352B dt=3139 +23:30:10 step 33220 loss=3.7973 lr=2.83e-04 tok=4.354B dt=3214 +23:31:25 step 33240 loss=3.7561 lr=2.83e-04 tok=4.357B dt=3290 +23:32:41 step 33260 loss=3.7056 lr=2.83e-04 tok=4.359B dt=3366 +23:33:57 step 33280 loss=3.8131 lr=2.82e-04 tok=4.362B dt=3442 +23:35:12 step 33300 loss=3.6880 lr=2.82e-04 tok=4.365B dt=3517 +23:36:28 step 33320 loss=3.8327 lr=2.82e-04 tok=4.367B dt=3593 +23:37:44 step 33340 loss=3.7252 lr=2.82e-04 tok=4.370B dt=3668 +23:38:59 step 33360 loss=3.7447 lr=2.82e-04 tok=4.373B dt=3744 +23:40:15 step 33380 loss=3.8627 lr=2.82e-04 tok=4.375B dt=3820 +23:41:31 step 33400 loss=3.7453 lr=2.82e-04 tok=4.378B dt=3896 +23:42:46 step 33420 loss=3.7800 lr=2.82e-04 tok=4.380B dt=3971 +23:44:02 step 33440 loss=3.8705 lr=2.82e-04 tok=4.383B dt=4047 +23:45:18 step 33460 loss=3.7931 lr=2.82e-04 tok=4.386B dt=4123 +23:46:34 step 33480 loss=3.6783 lr=2.82e-04 tok=4.388B dt=4198 +23:47:49 step 33500 loss=3.7417 lr=2.82e-04 tok=4.391B dt=4274 +23:49:05 step 33520 loss=3.6764 lr=2.82e-04 tok=4.394B dt=4350 +23:50:21 step 33540 loss=3.7693 lr=2.82e-04 tok=4.396B dt=4426 +23:51:36 step 33560 loss=3.7341 lr=2.82e-04 tok=4.399B dt=4501 +23:52:52 step 33580 loss=3.6949 lr=2.82e-04 tok=4.401B dt=4577 +23:54:08 step 33600 loss=3.8693 lr=2.82e-04 tok=4.404B dt=4652 +23:54:19 pushed ckpt_w9.pt diff --git a/log_w90.txt b/log_w90.txt new file mode 100644 index 0000000000000000000000000000000000000000..8b2b160488a5ee3bdcd046611c3f221eeb8fecf2 --- /dev/null +++ b/log_w90.txt @@ -0,0 +1,18 @@ +launch +18:57:41 name=w90 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 37/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:57:47 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:57:47 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:57:56 params 1.08B +18:57:58 resumed step=111600 seen=14.67B round=354 +18:57:58 stream salt=111600 +18:59:49 blocks compiled +18:59:49 start: accum=4, 131072 tokens per optimiser step +19:02:28 step 111601 loss=3.4951 lr=7.08e-05 tok=14.666B dt=159 wait=116 +19:02:32 step 111602 loss=3.4317 lr=7.08e-05 tok=14.666B dt=163 wait=116 +19:03:36 step 111620 loss=3.5243 lr=7.08e-05 tok=14.668B dt=227 wait=117 +19:04:43 step 111640 loss=3.4971 lr=7.07e-05 tok=14.671B dt=293 wait=117 +19:05:48 step 111660 loss=3.4812 lr=7.07e-05 tok=14.674B dt=358 wait=118 +19:06:53 step 111680 loss=3.4763 lr=7.07e-05 tok=14.676B dt=423 wait=118 +19:07:58 step 111700 loss=3.5486 lr=7.07e-05 tok=14.679B dt=488 wait=118 +19:09:03 step 111720 loss=3.5178 lr=7.07e-05 tok=14.681B dt=553 wait=119 +19:10:07 step 111740 loss=3.4390 lr=7.06e-05 tok=14.684B dt=618 wait=119 diff --git a/log_w92.txt b/log_w92.txt new file mode 100644 index 0000000000000000000000000000000000000000..5385bb2e8d5e7323588efcb5beb1c3dcadd09895 --- /dev/null +++ b/log_w92.txt @@ -0,0 +1,184 @@ +launch +18:57:48 name=w92 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 39/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:57:54 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:57:54 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:58:03 params 1.08B +18:58:05 resumed step=111600 seen=14.67B round=354 +18:58:05 stream salt=111600 +18:59:57 blocks compiled +18:59:57 start: accum=4, 131072 tokens per optimiser step +19:03:00 step 111601 loss=3.1087 lr=7.08e-05 tok=14.666B dt=183 wait=137 +19:03:03 step 111602 loss=3.1334 lr=7.08e-05 tok=14.666B dt=186 wait=137 +19:04:08 step 111620 loss=3.3731 lr=7.08e-05 tok=14.668B dt=251 wait=138 +19:05:12 step 111640 loss=3.3860 lr=7.07e-05 tok=14.671B dt=315 wait=138 +19:06:15 step 111660 loss=3.3866 lr=7.07e-05 tok=14.674B dt=379 wait=139 +19:07:19 step 111680 loss=3.4055 lr=7.07e-05 tok=14.676B dt=442 wait=139 +19:08:23 step 111700 loss=3.4872 lr=7.07e-05 tok=14.679B dt=506 wait=139 +19:09:27 step 111720 loss=3.4570 lr=7.07e-05 tok=14.681B dt=570 wait=139 +19:10:31 step 111740 loss=3.3114 lr=7.06e-05 tok=14.684B dt=634 wait=140 +19:11:21 pushed ckpt_w92.pt,log_w92.txt +19:12:06 waited 45s for global r=355 +19:12:16 global r=355 load 4.298 -> 4.287 +19:12:50 step 111760 loss=3.4396 lr=7.06e-05 tok=14.687B dt=773 wait=140 +19:14:01 step 111780 loss=3.3514 lr=7.06e-05 tok=14.689B dt=845 wait=140 +19:15:05 step 111800 loss=3.3573 lr=7.06e-05 tok=14.692B dt=908 wait=141 +19:16:08 step 111820 loss=3.5029 lr=7.06e-05 tok=14.694B dt=971 wait=141 +19:17:12 step 111840 loss=3.3440 lr=7.05e-05 tok=14.697B dt=1035 wait=141 +19:18:16 step 111860 loss=3.4376 lr=7.05e-05 tok=14.700B dt=1099 wait=142 +19:19:19 step 111880 loss=3.3735 lr=7.05e-05 tok=14.702B dt=1163 wait=142 +19:20:23 step 111900 loss=3.3464 lr=7.05e-05 tok=14.705B dt=1227 wait=142 +19:20:38 pushed ckpt_w92.pt,log_w92.txt +19:24:46 step 111920 loss=3.4549 lr=7.05e-05 tok=14.708B dt=1489 wait=143 +19:25:49 step 111940 loss=3.3519 lr=7.04e-05 tok=14.710B dt=1552 wait=143 +19:26:52 step 111960 loss=3.3114 lr=7.04e-05 tok=14.713B dt=1615 wait=143 +19:27:55 step 111980 loss=3.3351 lr=7.04e-05 tok=14.715B dt=1679 wait=143 +19:28:59 step 112000 loss=3.2398 lr=7.04e-05 tok=14.718B dt=1742 wait=144 +19:30:02 step 112020 loss=3.4426 lr=7.04e-05 tok=14.721B dt=1806 wait=144 +19:31:06 step 112040 loss=3.3210 lr=7.03e-05 tok=14.723B dt=1869 wait=144 +19:31:51 pushed ckpt_w92.pt,log_w92.txt +19:33:06 waited 76s for global r=356 +19:33:13 global r=356 load 4.290 -> 4.281 +19:33:47 step 112060 loss=3.4847 lr=7.03e-05 tok=14.726B dt=2030 wait=145 +19:35:11 step 112080 loss=3.3200 lr=7.03e-05 tok=14.729B dt=2114 wait=145 +19:36:15 step 112100 loss=3.3446 lr=7.03e-05 tok=14.731B dt=2178 wait=145 +19:37:18 step 112120 loss=3.4351 lr=7.02e-05 tok=14.734B dt=2241 wait=146 +19:38:22 step 112140 loss=3.3425 lr=7.02e-05 tok=14.736B dt=2305 wait=146 +19:39:25 step 112160 loss=3.4515 lr=7.02e-05 tok=14.739B dt=2368 wait=146 +19:40:29 step 112180 loss=3.3673 lr=7.02e-05 tok=14.742B dt=2432 wait=147 +19:41:32 step 112200 loss=3.3813 lr=7.02e-05 tok=14.744B dt=2496 wait=147 +19:41:44 pushed ckpt_w92.pt,log_w92.txt +19:42:30 waited 46s for global r=357 +19:42:38 global r=357 load 4.281 -> 4.278 +19:43:50 step 112220 loss=3.3656 lr=7.01e-05 tok=14.747B dt=2633 wait=147 +19:45:07 step 112240 loss=3.3123 lr=7.01e-05 tok=14.750B dt=2710 wait=148 +19:46:11 step 112260 loss=3.3646 lr=7.01e-05 tok=14.752B dt=2774 wait=148 +19:47:14 step 112280 loss=3.3419 lr=7.01e-05 tok=14.755B dt=2837 wait=148 +19:48:18 step 112300 loss=3.3501 lr=7.01e-05 tok=14.757B dt=2901 wait=149 +19:49:21 step 112320 loss=3.4104 lr=7.00e-05 tok=14.760B dt=2964 wait=149 +19:50:25 step 112340 loss=3.3312 lr=7.00e-05 tok=14.763B dt=3028 wait=149 +19:51:11 pushed ckpt_w92.pt,log_w92.txt +19:52:11 waited 60s for global r=358 +19:52:17 global r=358 load 4.283 -> 4.291 +19:52:50 step 112360 loss=3.4632 lr=7.00e-05 tok=14.765B dt=3173 wait=149 +19:54:08 step 112380 loss=3.3221 lr=7.00e-05 tok=14.768B dt=3251 wait=150 +19:55:17 step 112400 loss=3.3694 lr=7.00e-05 tok=14.771B dt=3320 wait=150 +19:56:20 step 112420 loss=3.4527 lr=6.99e-05 tok=14.773B dt=3383 wait=150 +19:57:24 step 112440 loss=3.3688 lr=6.99e-05 tok=14.776B dt=3447 wait=151 +19:58:28 step 112460 loss=3.3761 lr=6.99e-05 tok=14.778B dt=3511 wait=151 +19:59:31 step 112480 loss=3.3799 lr=6.99e-05 tok=14.781B dt=3575 wait=151 +20:00:35 step 112500 loss=3.4021 lr=6.99e-05 tok=14.784B dt=3638 wait=152 +20:00:49 pushed ckpt_w92.pt,log_w92.txt +20:03:51 waited 181s for global r=359 +20:03:59 global r=359 load 4.321 -> 4.281 +20:05:11 step 112520 loss=3.3609 lr=6.98e-05 tok=14.786B dt=3914 wait=152 +20:06:26 step 112540 loss=3.3620 lr=6.98e-05 tok=14.789B dt=3990 wait=152 +20:07:30 step 112560 loss=3.3390 lr=6.98e-05 tok=14.791B dt=4053 wait=153 +20:08:34 step 112580 loss=3.3591 lr=6.98e-05 tok=14.794B dt=4117 wait=153 +20:09:38 step 112600 loss=3.3500 lr=6.97e-05 tok=14.797B dt=4181 wait=153 +20:10:42 step 112620 loss=3.4836 lr=6.97e-05 tok=14.799B dt=4245 wait=154 +20:11:45 step 112640 loss=3.3011 lr=6.97e-05 tok=14.802B dt=4308 wait=154 +20:12:29 pushed ckpt_w92.pt,log_w92.txt +20:13:30 waited 61s for global r=360 +20:13:36 global r=360 load 4.306 -> 4.290 +20:14:09 step 112660 loss=3.4113 lr=6.97e-05 tok=14.805B dt=4452 wait=154 +20:15:32 step 112680 loss=3.3352 lr=6.97e-05 tok=14.807B dt=4535 wait=154 +20:16:36 step 112700 loss=3.3479 lr=6.96e-05 tok=14.810B dt=4599 wait=155 +20:17:39 step 112720 loss=3.4860 lr=6.96e-05 tok=14.812B dt=4662 wait=155 +20:18:42 step 112740 loss=3.3317 lr=6.96e-05 tok=14.815B dt=4726 wait=155 +20:19:46 step 112760 loss=3.4511 lr=6.96e-05 tok=14.818B dt=4789 wait=156 +20:20:50 step 112780 loss=3.3613 lr=6.96e-05 tok=14.820B dt=4853 wait=156 +20:21:53 step 112800 loss=3.3463 lr=6.95e-05 tok=14.823B dt=4917 wait=156 +20:22:06 pushed ckpt_w92.pt,log_w92.txt +20:22:51 waited 45s for global r=361 +20:23:00 global r=361 load 4.287 -> 4.289 +20:24:12 step 112820 loss=3.3279 lr=6.95e-05 tok=14.826B dt=5055 wait=157 +20:25:28 step 112840 loss=3.4083 lr=6.95e-05 tok=14.828B dt=5131 wait=157 +20:26:31 step 112860 loss=3.3827 lr=6.95e-05 tok=14.831B dt=5194 wait=157 +20:27:34 step 112880 loss=3.3564 lr=6.95e-05 tok=14.833B dt=5257 wait=158 +20:28:38 step 112900 loss=3.3473 lr=6.94e-05 tok=14.836B dt=5322 wait=158 +20:29:42 step 112920 loss=3.4626 lr=6.94e-05 tok=14.839B dt=5385 wait=158 +20:30:45 step 112940 loss=3.3274 lr=6.94e-05 tok=14.841B dt=5449 wait=159 +20:31:29 pushed ckpt_w92.pt,log_w92.txt +20:32:15 waited 45s for global r=362 +20:32:23 global r=362 load 4.315 -> 4.299 +20:32:56 step 112960 loss=3.4147 lr=6.94e-05 tok=14.844B dt=5579 wait=159 +20:34:07 step 112980 loss=3.2589 lr=6.94e-05 tok=14.847B dt=5651 wait=159 +20:35:23 step 113000 loss=3.3414 lr=6.93e-05 tok=14.849B dt=5726 wait=160 +20:36:26 step 113020 loss=3.3882 lr=6.93e-05 tok=14.852B dt=5790 wait=160 +20:37:30 step 113040 loss=3.2787 lr=6.93e-05 tok=14.854B dt=5853 wait=160 +20:38:33 step 113060 loss=3.3483 lr=6.93e-05 tok=14.857B dt=5916 wait=160 +20:39:37 step 113080 loss=3.3410 lr=6.92e-05 tok=14.860B dt=5980 wait=161 +20:40:41 step 113100 loss=3.4390 lr=6.92e-05 tok=14.862B dt=6044 wait=161 +20:40:53 pushed ckpt_w92.pt,log_w92.txt +20:41:38 waited 45s for global r=363 +20:41:49 global r=363 load 4.275 -> 4.289 +20:42:54 step 113120 loss=3.3780 lr=6.92e-05 tok=14.865B dt=6177 wait=161 +20:44:15 step 113140 loss=3.3991 lr=6.92e-05 tok=14.867B dt=6258 wait=162 +20:45:21 step 113160 loss=3.3796 lr=6.92e-05 tok=14.870B dt=6324 wait=162 +20:46:25 step 113180 loss=3.3886 lr=6.91e-05 tok=14.873B dt=6388 wait=162 +20:47:28 step 113200 loss=3.3966 lr=6.91e-05 tok=14.875B dt=6451 wait=163 +20:48:32 step 113220 loss=3.5003 lr=6.91e-05 tok=14.878B dt=6515 wait=163 +20:49:36 step 113240 loss=3.3216 lr=6.91e-05 tok=14.881B dt=6579 wait=163 +20:50:23 pushed ckpt_w92.pt,log_w92.txt +20:51:08 waited 45s for global r=364 +20:51:14 global r=364 load 4.282 -> 4.284 +20:51:47 step 113260 loss=3.3998 lr=6.91e-05 tok=14.883B dt=6710 wait=164 +20:52:56 step 113280 loss=3.3126 lr=6.90e-05 tok=14.886B dt=6779 wait=164 +20:54:13 step 113300 loss=3.2865 lr=6.90e-05 tok=14.888B dt=6856 wait=164 +20:55:17 step 113320 loss=3.3957 lr=6.90e-05 tok=14.891B dt=6920 wait=165 +20:56:20 step 113340 loss=3.3656 lr=6.90e-05 tok=14.894B dt=6984 wait=165 +20:57:24 step 113360 loss=3.3567 lr=6.90e-05 tok=14.896B dt=7047 wait=165 +20:58:29 step 113380 loss=3.2799 lr=6.89e-05 tok=14.899B dt=7112 wait=166 +20:59:32 step 113400 loss=3.3957 lr=6.89e-05 tok=14.902B dt=7175 wait=166 +20:59:46 pushed ckpt_w92.pt,log_w92.txt +21:00:32 waited 45s for global r=365 +21:00:39 global r=365 load 4.314 -> 4.298 +21:01:44 step 113420 loss=3.3030 lr=6.89e-05 tok=14.904B dt=7307 wait=166 +21:02:59 step 113440 loss=3.3588 lr=6.89e-05 tok=14.907B dt=7382 wait=167 +21:04:11 step 113460 loss=3.3861 lr=6.89e-05 tok=14.909B dt=7454 wait=167 +21:05:14 step 113480 loss=3.3189 lr=6.88e-05 tok=14.912B dt=7518 wait=167 +21:06:18 step 113500 loss=3.3926 lr=6.88e-05 tok=14.915B dt=7581 wait=167 +21:07:22 step 113520 loss=3.4765 lr=6.88e-05 tok=14.917B dt=7645 wait=168 +21:08:25 step 113540 loss=3.3412 lr=6.88e-05 tok=14.920B dt=7708 wait=168 +21:09:13 pushed ckpt_w92.pt,log_w92.txt +21:10:13 waited 60s for global r=366 +21:10:22 global r=366 load 4.305 -> 4.288 +21:10:55 step 113560 loss=3.4491 lr=6.88e-05 tok=14.923B dt=7858 wait=168 +21:12:06 step 113580 loss=3.3131 lr=6.87e-05 tok=14.925B dt=7929 wait=169 +21:13:21 step 113600 loss=3.3094 lr=6.87e-05 tok=14.928B dt=8005 wait=169 +21:14:25 step 113620 loss=3.4265 lr=6.87e-05 tok=14.930B dt=8068 wait=169 +21:15:29 step 113640 loss=3.1720 lr=6.87e-05 tok=14.933B dt=8132 wait=170 +21:16:32 step 113660 loss=3.3453 lr=6.86e-05 tok=14.936B dt=8195 wait=170 +21:17:36 step 113680 loss=3.3050 lr=6.86e-05 tok=14.938B dt=8259 wait=170 +21:18:39 step 113700 loss=3.3899 lr=6.86e-05 tok=14.941B dt=8322 wait=171 +21:18:54 pushed ckpt_w92.pt,log_w92.txt +21:19:39 waited 45s for global r=367 +21:19:45 global r=367 load 4.280 -> 4.285 +21:20:50 step 113720 loss=3.3745 lr=6.86e-05 tok=14.944B dt=8453 wait=171 +21:22:07 step 113740 loss=3.3658 lr=6.86e-05 tok=14.946B dt=8530 wait=171 +21:23:17 step 113760 loss=3.3345 lr=6.85e-05 tok=14.949B dt=8600 wait=172 +21:24:20 step 113780 loss=3.3674 lr=6.85e-05 tok=14.951B dt=8664 wait=172 +21:25:24 step 113800 loss=3.4112 lr=6.85e-05 tok=14.954B dt=8728 wait=172 +21:26:28 step 113820 loss=3.4440 lr=6.85e-05 tok=14.957B dt=8791 wait=173 +21:27:31 step 113840 loss=3.3412 lr=6.85e-05 tok=14.959B dt=8854 wait=173 +21:28:15 pushed ckpt_w92.pt,log_w92.txt +21:29:01 waited 45s for global r=368 +21:29:07 global r=368 load 4.300 -> 4.284 +21:29:40 step 113860 loss=3.4380 lr=6.84e-05 tok=14.962B dt=8984 wait=173 +21:30:44 step 113880 loss=3.3668 lr=6.84e-05 tok=14.964B dt=9047 wait=174 +21:32:07 step 113900 loss=3.2892 lr=6.84e-05 tok=14.967B dt=9130 wait=174 +21:33:11 step 113920 loss=3.3567 lr=6.84e-05 tok=14.970B dt=9194 wait=174 +21:34:14 step 113940 loss=3.2859 lr=6.84e-05 tok=14.972B dt=9258 wait=175 +21:35:18 step 113960 loss=3.4278 lr=6.83e-05 tok=14.975B dt=9322 wait=175 +21:36:22 step 113980 loss=3.3017 lr=6.83e-05 tok=14.978B dt=9385 wait=175 +21:37:26 step 114000 loss=3.2589 lr=6.83e-05 tok=14.980B dt=9449 wait=176 +21:37:44 pushed ckpt_w92.pt,log_w92.txt +21:38:14 waited 30s for global r=369 +21:38:19 global r=369 load 4.281 -> 4.302 +21:39:24 step 114020 loss=3.3497 lr=6.83e-05 tok=14.983B dt=9567 wait=176 +21:40:33 step 114040 loss=3.3679 lr=6.83e-05 tok=14.985B dt=9636 wait=176 +21:41:50 step 114060 loss=3.3861 lr=6.82e-05 tok=14.988B dt=9714 wait=177 +21:42:54 step 114080 loss=3.3373 lr=6.82e-05 tok=14.991B dt=9777 wait=177 +21:43:58 step 114100 loss=3.3517 lr=6.82e-05 tok=14.993B dt=9841 wait=177 +21:45:01 step 114120 loss=3.4142 lr=6.82e-05 tok=14.996B dt=9904 wait=177 +21:46:05 step 114140 loss=3.3014 lr=6.81e-05 tok=14.999B dt=9968 wait=178 diff --git a/log_w93.txt b/log_w93.txt new file mode 100644 index 0000000000000000000000000000000000000000..92f7b9cc0e446dd72b2e336fe6f3adfcb0cc5e4a --- /dev/null +++ b/log_w93.txt @@ -0,0 +1,383 @@ +launch +18:57:45 name=w93 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 40/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:57:51 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:57:51 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:58:00 params 1.08B +18:58:02 resumed step=111600 seen=14.67B round=354 +18:58:02 stream salt=111600 +18:59:44 blocks compiled +18:59:44 start: accum=4, 131072 tokens per optimiser step +19:02:33 step 111601 loss=3.2260 lr=7.08e-05 tok=14.666B dt=168 wait=124 +19:02:37 step 111602 loss=3.2421 lr=7.08e-05 tok=14.666B dt=172 wait=125 +19:03:43 step 111620 loss=3.4953 lr=7.08e-05 tok=14.668B dt=239 wait=125 +19:04:49 step 111640 loss=3.4267 lr=7.07e-05 tok=14.671B dt=304 wait=126 +19:05:53 step 111660 loss=3.4924 lr=7.07e-05 tok=14.674B dt=369 wait=126 +19:06:58 step 111680 loss=3.4503 lr=7.07e-05 tok=14.676B dt=433 wait=127 +19:08:02 step 111700 loss=3.5759 lr=7.07e-05 tok=14.679B dt=498 wait=127 +19:09:06 step 111720 loss=3.4842 lr=7.07e-05 tok=14.681B dt=562 wait=127 +19:10:10 step 111740 loss=3.4755 lr=7.06e-05 tok=14.684B dt=626 wait=128 +19:10:56 pushed ckpt_w93.pt,log_w93.txt +19:11:57 waited 60s for global r=355 +19:12:11 global r=355 load 4.297 -> 4.287 +19:12:46 step 111760 loss=3.5357 lr=7.06e-05 tok=14.687B dt=781 wait=128 +19:13:50 step 111780 loss=3.3947 lr=7.06e-05 tok=14.689B dt=846 wait=128 +19:14:55 step 111800 loss=3.3802 lr=7.06e-05 tok=14.692B dt=910 wait=129 +19:15:59 step 111820 loss=3.4748 lr=7.06e-05 tok=14.694B dt=974 wait=129 +19:17:03 step 111840 loss=3.4078 lr=7.05e-05 tok=14.697B dt=1038 wait=129 +19:18:07 step 111860 loss=3.4257 lr=7.05e-05 tok=14.700B dt=1103 wait=130 +19:19:11 step 111880 loss=3.4102 lr=7.05e-05 tok=14.702B dt=1166 wait=130 +19:20:15 step 111900 loss=3.4533 lr=7.05e-05 tok=14.705B dt=1230 wait=130 +19:20:30 pushed ckpt_w93.pt,log_w93.txt +19:24:38 step 111920 loss=3.4196 lr=7.05e-05 tok=14.708B dt=1494 wait=131 +19:25:43 step 111940 loss=3.4329 lr=7.04e-05 tok=14.710B dt=1558 wait=131 +19:26:47 step 111960 loss=3.3889 lr=7.04e-05 tok=14.713B dt=1622 wait=131 +19:27:50 step 111980 loss=3.3612 lr=7.04e-05 tok=14.715B dt=1686 wait=132 +19:28:55 step 112000 loss=3.3297 lr=7.04e-05 tok=14.718B dt=1750 wait=132 +19:29:59 step 112020 loss=3.4722 lr=7.04e-05 tok=14.721B dt=1815 wait=133 +19:31:03 step 112040 loss=3.4209 lr=7.03e-05 tok=14.723B dt=1879 wait=133 +19:31:48 pushed ckpt_w93.pt,log_w93.txt +19:33:04 waited 76s for global r=356 +19:33:16 global r=356 load 4.303 -> 4.281 +19:33:50 step 112060 loss=3.5438 lr=7.03e-05 tok=14.726B dt=2046 wait=133 +19:34:54 step 112080 loss=3.3811 lr=7.03e-05 tok=14.729B dt=2110 wait=134 +19:35:58 step 112100 loss=3.3806 lr=7.03e-05 tok=14.731B dt=2174 wait=134 +19:37:03 step 112120 loss=3.4932 lr=7.02e-05 tok=14.734B dt=2238 wait=134 +19:38:07 step 112140 loss=3.3396 lr=7.02e-05 tok=14.736B dt=2303 wait=135 +19:39:11 step 112160 loss=3.4823 lr=7.02e-05 tok=14.739B dt=2367 wait=135 +19:40:15 step 112180 loss=3.3941 lr=7.02e-05 tok=14.742B dt=2431 wait=135 +19:41:19 step 112200 loss=3.4373 lr=7.02e-05 tok=14.744B dt=2495 wait=136 +19:41:31 pushed ckpt_w93.pt,log_w93.txt +19:42:32 waited 60s for global r=357 +19:42:39 global r=357 load 4.281 -> 4.278 +19:43:44 step 112220 loss=3.3916 lr=7.01e-05 tok=14.747B dt=2640 wait=136 +19:44:49 step 112240 loss=3.4759 lr=7.01e-05 tok=14.750B dt=2704 wait=136 +19:45:53 step 112260 loss=3.4891 lr=7.01e-05 tok=14.752B dt=2769 wait=137 +19:46:57 step 112280 loss=3.3922 lr=7.01e-05 tok=14.755B dt=2833 wait=137 +19:48:01 step 112300 loss=3.4508 lr=7.01e-05 tok=14.757B dt=2897 wait=138 +19:49:06 step 112320 loss=3.4990 lr=7.00e-05 tok=14.760B dt=2961 wait=138 +19:50:10 step 112340 loss=3.3644 lr=7.00e-05 tok=14.763B dt=3025 wait=138 +19:50:54 pushed ckpt_w93.pt,log_w93.txt +19:52:09 waited 76s for global r=358 +19:52:17 global r=358 load 4.296 -> 4.291 +19:52:51 step 112360 loss=3.5498 lr=7.00e-05 tok=14.765B dt=3186 wait=139 +19:53:55 step 112380 loss=3.3717 lr=7.00e-05 tok=14.768B dt=3250 wait=139 +19:54:59 step 112400 loss=3.4113 lr=7.00e-05 tok=14.771B dt=3315 wait=139 +19:56:03 step 112420 loss=3.5133 lr=6.99e-05 tok=14.773B dt=3379 wait=140 +19:57:08 step 112440 loss=3.4490 lr=6.99e-05 tok=14.776B dt=3443 wait=140 +19:58:12 step 112460 loss=3.5088 lr=6.99e-05 tok=14.778B dt=3507 wait=140 +19:59:16 step 112480 loss=3.4284 lr=6.99e-05 tok=14.781B dt=3572 wait=141 +20:00:20 step 112500 loss=3.4718 lr=6.99e-05 tok=14.784B dt=3636 wait=141 +20:00:32 pushed ckpt_w93.pt,log_w93.txt +20:04:12 global r=359 load 4.271 -> 4.281 +20:05:17 step 112520 loss=3.3935 lr=6.98e-05 tok=14.788B dt=3932 wait=142 +20:06:21 step 112540 loss=3.3931 lr=6.98e-05 tok=14.790B dt=3996 wait=142 +20:07:25 step 112560 loss=3.4166 lr=6.98e-05 tok=14.793B dt=4060 wait=142 +20:08:29 step 112580 loss=3.3855 lr=6.98e-05 tok=14.795B dt=4124 wait=143 +20:09:33 step 112600 loss=3.3821 lr=6.97e-05 tok=14.798B dt=4188 wait=143 +20:10:37 step 112620 loss=3.5286 lr=6.97e-05 tok=14.801B dt=4253 wait=143 +20:11:41 step 112640 loss=3.3625 lr=6.97e-05 tok=14.803B dt=4317 wait=144 +20:12:28 pushed ckpt_w93.pt,log_w93.txt +20:13:28 waited 60s for global r=360 +20:13:33 global r=360 load 4.298 -> 4.290 +20:14:07 step 112660 loss=3.5441 lr=6.97e-05 tok=14.806B dt=4462 wait=144 +20:15:10 step 112680 loss=3.4070 lr=6.97e-05 tok=14.809B dt=4526 wait=144 +20:16:14 step 112700 loss=3.3541 lr=6.96e-05 tok=14.811B dt=4590 wait=145 +20:17:18 step 112720 loss=3.4175 lr=6.96e-05 tok=14.814B dt=4654 wait=145 +20:18:27 step 112740 loss=3.3960 lr=6.96e-05 tok=14.816B dt=4722 wait=145 +20:19:31 step 112760 loss=3.4736 lr=6.96e-05 tok=14.819B dt=4787 wait=146 +20:20:35 step 112780 loss=3.4386 lr=6.96e-05 tok=14.822B dt=4851 wait=146 +20:21:39 step 112800 loss=3.4488 lr=6.95e-05 tok=14.824B dt=4914 wait=147 +20:21:51 pushed ckpt_w93.pt,log_w93.txt +20:22:52 waited 61s for global r=361 +20:22:58 global r=361 load 4.289 -> 4.289 +20:24:03 step 112820 loss=3.3426 lr=6.95e-05 tok=14.827B dt=5059 wait=147 +20:25:07 step 112840 loss=3.3995 lr=6.95e-05 tok=14.829B dt=5122 wait=147 +20:26:11 step 112860 loss=3.4559 lr=6.95e-05 tok=14.832B dt=5187 wait=148 +20:27:15 step 112880 loss=3.3849 lr=6.95e-05 tok=14.835B dt=5250 wait=148 +20:28:18 step 112900 loss=3.4044 lr=6.94e-05 tok=14.837B dt=5314 wait=148 +20:29:22 step 112920 loss=3.4848 lr=6.94e-05 tok=14.840B dt=5378 wait=149 +20:30:25 step 112940 loss=3.3576 lr=6.94e-05 tok=14.843B dt=5441 wait=149 +20:31:09 pushed ckpt_w93.pt,log_w93.txt +20:32:10 waited 61s for global r=362 +20:32:20 global r=362 load 4.324 -> 4.299 +20:32:53 step 112960 loss=3.4689 lr=6.94e-05 tok=14.845B dt=5589 wait=149 +20:33:57 step 112980 loss=3.3171 lr=6.94e-05 tok=14.848B dt=5652 wait=150 +20:35:00 step 113000 loss=3.3402 lr=6.93e-05 tok=14.850B dt=5716 wait=150 +20:36:04 step 113020 loss=3.4462 lr=6.93e-05 tok=14.853B dt=5779 wait=150 +20:37:08 step 113040 loss=3.3058 lr=6.93e-05 tok=14.856B dt=5843 wait=151 +20:38:11 step 113060 loss=3.4303 lr=6.93e-05 tok=14.858B dt=5907 wait=151 +20:39:15 step 113080 loss=3.3343 lr=6.92e-05 tok=14.861B dt=5970 wait=151 +20:40:19 step 113100 loss=3.3623 lr=6.92e-05 tok=14.864B dt=6034 wait=152 +20:40:31 pushed ckpt_w93.pt,log_w93.txt +20:41:46 waited 76s for global r=363 +20:41:54 global r=363 load 4.310 -> 4.289 +20:42:59 step 113120 loss=3.3543 lr=6.92e-05 tok=14.866B dt=6195 wait=152 +20:44:03 step 113140 loss=3.3724 lr=6.92e-05 tok=14.869B dt=6258 wait=152 +20:45:06 step 113160 loss=3.4166 lr=6.92e-05 tok=14.871B dt=6322 wait=153 +20:46:10 step 113180 loss=3.3503 lr=6.91e-05 tok=14.874B dt=6386 wait=153 +20:47:14 step 113200 loss=3.3261 lr=6.91e-05 tok=14.877B dt=6449 wait=154 +20:48:18 step 113220 loss=3.4746 lr=6.91e-05 tok=14.879B dt=6513 wait=154 +20:49:22 step 113240 loss=3.3014 lr=6.91e-05 tok=14.882B dt=6578 wait=154 +20:50:07 pushed ckpt_w93.pt,log_w93.txt +20:51:08 waited 61s for global r=364 +20:51:13 global r=364 load 4.299 -> 4.284 +20:51:46 step 113260 loss=3.5306 lr=6.91e-05 tok=14.885B dt=6722 wait=155 +20:52:50 step 113280 loss=3.3241 lr=6.90e-05 tok=14.887B dt=6786 wait=155 +20:53:53 step 113300 loss=3.3232 lr=6.90e-05 tok=14.890B dt=6849 wait=155 +20:54:57 step 113320 loss=3.4605 lr=6.90e-05 tok=14.892B dt=6913 wait=156 +20:56:03 step 113340 loss=2.8187 lr=6.90e-05 tok=14.895B dt=6979 wait=156 +20:57:07 step 113360 loss=3.4269 lr=6.90e-05 tok=14.898B dt=7043 wait=156 +20:58:11 step 113380 loss=3.3460 lr=6.89e-05 tok=14.900B dt=7106 wait=157 +20:59:15 step 113400 loss=3.3732 lr=6.89e-05 tok=14.903B dt=7170 wait=157 +20:59:28 pushed ckpt_w93.pt,log_w93.txt +21:00:43 waited 76s for global r=365 +21:00:53 global r=365 load 4.312 -> 4.298 +21:01:58 step 113420 loss=3.3677 lr=6.89e-05 tok=14.906B dt=7333 wait=158 +21:03:02 step 113440 loss=3.4316 lr=6.89e-05 tok=14.908B dt=7397 wait=158 +21:04:05 step 113460 loss=3.4171 lr=6.89e-05 tok=14.911B dt=7461 wait=158 +21:05:09 step 113480 loss=3.3760 lr=6.88e-05 tok=14.913B dt=7525 wait=159 +21:06:13 step 113500 loss=3.3394 lr=6.88e-05 tok=14.916B dt=7588 wait=159 +21:07:16 step 113520 loss=3.4327 lr=6.88e-05 tok=14.919B dt=7652 wait=159 +21:08:20 step 113540 loss=3.3461 lr=6.88e-05 tok=14.921B dt=7715 wait=160 +21:09:04 pushed ckpt_w93.pt,log_w93.txt +21:10:04 waited 61s for global r=366 +21:10:13 global r=366 load 4.300 -> 4.288 +21:10:46 step 113560 loss=3.4250 lr=6.88e-05 tok=14.924B dt=7861 wait=160 +21:11:50 step 113580 loss=3.3051 lr=6.87e-05 tok=14.926B dt=7925 wait=160 +21:12:53 step 113600 loss=3.3309 lr=6.87e-05 tok=14.929B dt=7988 wait=161 +21:13:57 step 113620 loss=3.4118 lr=6.87e-05 tok=14.932B dt=8052 wait=161 +21:15:00 step 113640 loss=3.3173 lr=6.87e-05 tok=14.934B dt=8116 wait=161 +21:16:04 step 113660 loss=3.4098 lr=6.86e-05 tok=14.937B dt=8179 wait=162 +21:17:07 step 113680 loss=3.3734 lr=6.86e-05 tok=14.940B dt=8243 wait=162 +21:18:11 step 113700 loss=3.4093 lr=6.86e-05 tok=14.942B dt=8306 wait=162 +21:18:23 pushed ckpt_w93.pt,log_w93.txt +21:19:39 waited 76s for global r=367 +21:19:45 global r=367 load 4.318 -> 4.285 +21:20:50 step 113720 loss=3.3266 lr=6.86e-05 tok=14.945B dt=8465 wait=163 +21:21:53 step 113740 loss=3.3666 lr=6.86e-05 tok=14.947B dt=8529 wait=163 +21:22:56 step 113760 loss=3.4255 lr=6.85e-05 tok=14.950B dt=8592 wait=163 +21:24:00 step 113780 loss=3.3425 lr=6.85e-05 tok=14.953B dt=8656 wait=164 +21:25:04 step 113800 loss=3.2822 lr=6.85e-05 tok=14.955B dt=8720 wait=164 +21:26:08 step 113820 loss=3.3290 lr=6.85e-05 tok=14.958B dt=8784 wait=164 +21:27:12 step 113840 loss=3.2945 lr=6.85e-05 tok=14.961B dt=8848 wait=165 +21:27:58 pushed ckpt_w93.pt,log_w93.txt +21:28:58 waited 61s for global r=368 +21:29:07 global r=368 load 4.311 -> 4.284 +21:29:40 step 113860 loss=3.4165 lr=6.84e-05 tok=14.963B dt=8996 wait=165 +21:30:44 step 113880 loss=3.3563 lr=6.84e-05 tok=14.966B dt=9059 wait=166 +21:31:47 step 113900 loss=3.3622 lr=6.84e-05 tok=14.968B dt=9123 wait=166 +21:32:51 step 113920 loss=3.3883 lr=6.84e-05 tok=14.971B dt=9186 wait=166 +21:33:54 step 113940 loss=3.3853 lr=6.84e-05 tok=14.974B dt=9250 wait=166 +21:34:58 step 113960 loss=3.3507 lr=6.83e-05 tok=14.976B dt=9314 wait=167 +21:36:01 step 113980 loss=3.3623 lr=6.83e-05 tok=14.979B dt=9377 wait=167 +21:37:05 step 114000 loss=3.4004 lr=6.83e-05 tok=14.982B dt=9441 wait=168 +21:37:21 pushed ckpt_w93.pt,log_w93.txt +21:38:21 waited 61s for global r=369 +21:38:26 global r=369 load 4.300 -> 4.302 +21:39:32 step 114020 loss=3.2900 lr=6.83e-05 tok=14.984B dt=9587 wait=168 +21:40:35 step 114040 loss=3.3575 lr=6.83e-05 tok=14.987B dt=9651 wait=168 +21:41:39 step 114060 loss=3.3834 lr=6.82e-05 tok=14.989B dt=9714 wait=169 +21:42:42 step 114080 loss=3.3281 lr=6.82e-05 tok=14.992B dt=9777 wait=169 +21:43:45 step 114100 loss=3.3692 lr=6.82e-05 tok=14.995B dt=9841 wait=169 +21:44:49 step 114120 loss=3.4813 lr=6.82e-05 tok=14.997B dt=9904 wait=170 +21:45:52 step 114140 loss=3.3656 lr=6.81e-05 tok=15.000B dt=9968 wait=170 +21:46:36 pushed ckpt_w93.pt,log_w93.txt +21:47:37 waited 61s for global r=370 +21:47:44 global r=370 load 4.302 -> 4.308 +21:48:17 step 114160 loss=3.4201 lr=6.81e-05 tok=15.003B dt=10112 wait=170 +21:49:20 step 114180 loss=3.3381 lr=6.81e-05 tok=15.005B dt=10176 wait=170 +21:50:23 step 114200 loss=3.3539 lr=6.81e-05 tok=15.008B dt=10239 wait=171 +21:51:28 step 114220 loss=3.4160 lr=6.81e-05 tok=15.010B dt=10304 wait=171 +21:52:31 step 114240 loss=3.3596 lr=6.80e-05 tok=15.013B dt=10367 wait=171 +21:53:35 step 114260 loss=3.4352 lr=6.80e-05 tok=15.016B dt=10431 wait=172 +21:54:39 step 114280 loss=3.3780 lr=6.80e-05 tok=15.018B dt=10495 wait=172 +21:55:43 step 114300 loss=3.4072 lr=6.80e-05 tok=15.021B dt=10558 wait=172 +21:55:57 pushed ckpt_w93.pt,log_w93.txt +22:00:02 step 114320 loss=3.4518 lr=6.80e-05 tok=15.023B dt=10818 wait=173 +22:00:39 global r=371 load 4.304 -> 4.315 +22:01:43 step 114320 loss=3.3692 lr=6.80e-05 tok=15.027B dt=10918 wait=173 +22:02:46 step 114340 loss=3.4056 lr=6.79e-05 tok=15.030B dt=10981 wait=174 +22:03:49 step 114360 loss=3.3537 lr=6.79e-05 tok=15.033B dt=11045 wait=174 +22:04:53 step 114380 loss=3.3679 lr=6.79e-05 tok=15.035B dt=11109 wait=174 +22:05:57 step 114400 loss=3.3502 lr=6.79e-05 tok=15.038B dt=11172 wait=174 +22:07:01 step 114420 loss=3.0215 lr=6.79e-05 tok=15.041B dt=11237 wait=175 +22:08:05 step 114440 loss=3.3381 lr=6.78e-05 tok=15.043B dt=11301 wait=175 +22:08:51 pushed ckpt_w93.pt,log_w93.txt +22:12:25 step 114460 loss=3.5102 lr=6.78e-05 tok=15.046B dt=11560 wait=176 +22:12:30 global r=372 load 4.292 -> 4.314 +22:13:01 step 114460 loss=3.5068 lr=6.78e-05 tok=15.047B dt=11597 wait=176 +22:14:05 step 114480 loss=3.3541 lr=6.78e-05 tok=15.050B dt=11660 wait=176 +22:15:08 step 114500 loss=3.3366 lr=6.78e-05 tok=15.052B dt=11724 wait=176 +22:16:12 step 114520 loss=3.4206 lr=6.78e-05 tok=15.055B dt=11787 wait=177 +22:17:15 step 114540 loss=3.3454 lr=6.77e-05 tok=15.058B dt=11851 wait=177 +22:18:19 step 114560 loss=3.5049 lr=6.77e-05 tok=15.060B dt=11914 wait=177 +22:19:22 step 114580 loss=3.3945 lr=6.77e-05 tok=15.063B dt=11978 wait=178 +22:20:26 step 114600 loss=3.4288 lr=6.77e-05 tok=15.065B dt=12042 wait=178 +22:20:39 pushed ckpt_w93.pt,log_w93.txt +22:21:55 waited 76s for global r=373 +22:21:59 global r=373 load 4.345 -> 4.314 +22:23:03 step 114620 loss=3.3727 lr=6.77e-05 tok=15.068B dt=12199 wait=178 +22:24:07 step 114640 loss=3.3745 lr=6.76e-05 tok=15.071B dt=12262 wait=179 +22:25:10 step 114660 loss=3.4018 lr=6.76e-05 tok=15.073B dt=12326 wait=179 +22:26:14 step 114680 loss=3.3443 lr=6.76e-05 tok=15.076B dt=12390 wait=179 +22:27:18 step 114700 loss=3.3064 lr=6.76e-05 tok=15.079B dt=12454 wait=180 +22:28:22 step 114720 loss=3.4514 lr=6.75e-05 tok=15.081B dt=12517 wait=180 +22:29:25 step 114740 loss=3.3362 lr=6.75e-05 tok=15.084B dt=12580 wait=180 +22:30:11 pushed ckpt_w93.pt,log_w93.txt +22:31:27 waited 76s for global r=374 +22:31:32 global r=374 load 4.287 -> 4.314 +22:32:04 step 114760 loss=3.4766 lr=6.75e-05 tok=15.086B dt=12740 wait=181 +22:33:08 step 114780 loss=3.3643 lr=6.75e-05 tok=15.089B dt=12804 wait=181 +22:34:12 step 114800 loss=3.3871 lr=6.75e-05 tok=15.092B dt=12867 wait=181 +22:35:15 step 114820 loss=3.4491 lr=6.74e-05 tok=15.094B dt=12931 wait=182 +22:36:19 step 114840 loss=3.3534 lr=6.74e-05 tok=15.097B dt=12994 wait=182 +22:37:22 step 114860 loss=3.4521 lr=6.74e-05 tok=15.099B dt=13057 wait=182 +22:38:26 step 114880 loss=3.3451 lr=6.74e-05 tok=15.102B dt=13121 wait=183 +22:39:30 step 114900 loss=3.6282 lr=6.74e-05 tok=15.105B dt=13185 wait=183 +22:39:41 pushed ckpt_w93.pt,log_w93.txt +22:40:57 waited 76s for global r=375 +22:41:03 global r=375 load 4.304 -> 4.311 +22:42:07 step 114920 loss=3.3480 lr=6.73e-05 tok=15.107B dt=13343 wait=183 +22:43:11 step 114940 loss=3.3288 lr=6.73e-05 tok=15.110B dt=13407 wait=184 +22:44:15 step 114960 loss=3.4409 lr=6.73e-05 tok=15.113B dt=13471 wait=184 +22:45:18 step 114980 loss=3.3305 lr=6.73e-05 tok=15.115B dt=13534 wait=184 +22:46:22 step 115000 loss=3.3555 lr=6.73e-05 tok=15.118B dt=13598 wait=185 +22:47:26 step 115020 loss=3.4171 lr=6.72e-05 tok=15.120B dt=13661 wait=185 +22:48:29 step 115040 loss=3.2962 lr=6.72e-05 tok=15.123B dt=13724 wait=185 +22:49:13 pushed ckpt_w93.pt,log_w93.txt +22:50:29 waited 76s for global r=376 +22:50:35 global r=376 load 4.344 -> 4.313 +22:51:07 step 115060 loss=3.4208 lr=6.72e-05 tok=15.126B dt=13883 wait=186 +22:52:11 step 115080 loss=3.3685 lr=6.72e-05 tok=15.128B dt=13946 wait=186 +22:53:15 step 115100 loss=3.3964 lr=6.72e-05 tok=15.131B dt=14010 wait=186 +22:54:18 step 115120 loss=3.4392 lr=6.71e-05 tok=15.134B dt=14074 wait=187 +22:55:22 step 115140 loss=3.3595 lr=6.71e-05 tok=15.136B dt=14137 wait=187 +22:56:26 step 115160 loss=3.4183 lr=6.71e-05 tok=15.139B dt=14201 wait=187 +22:57:29 step 115180 loss=3.3390 lr=6.71e-05 tok=15.141B dt=14264 wait=188 +22:58:33 step 115200 loss=3.4628 lr=6.71e-05 tok=15.144B dt=14328 wait=188 +22:58:45 pushed ckpt_w93.pt,log_w93.txt +22:59:45 waited 61s for global r=377 +22:59:53 global r=377 load 4.274 -> 4.296 +23:00:57 step 115220 loss=3.3805 lr=6.70e-05 tok=15.147B dt=14472 wait=188 +23:02:01 step 115240 loss=3.4318 lr=6.70e-05 tok=15.149B dt=14536 wait=189 +23:03:05 step 115260 loss=3.3512 lr=6.70e-05 tok=15.152B dt=14600 wait=189 +23:04:08 step 115280 loss=3.3497 lr=6.70e-05 tok=15.155B dt=14664 wait=189 +23:05:11 step 115300 loss=3.3489 lr=6.69e-05 tok=15.157B dt=14727 wait=190 +23:06:15 step 115320 loss=3.4505 lr=6.69e-05 tok=15.160B dt=14790 wait=190 +23:07:18 step 115340 loss=3.2524 lr=6.69e-05 tok=15.162B dt=14854 wait=190 +23:08:02 pushed ckpt_w93.pt,log_w93.txt +23:09:18 waited 76s for global r=378 +23:09:23 global r=378 load 4.312 -> 4.308 +23:09:55 step 115360 loss=3.4069 lr=6.69e-05 tok=15.165B dt=15010 wait=191 +23:10:58 step 115380 loss=3.3010 lr=6.69e-05 tok=15.168B dt=15074 wait=191 +23:12:01 step 115400 loss=3.3141 lr=6.68e-05 tok=15.170B dt=15137 wait=191 +23:13:04 step 115420 loss=3.4431 lr=6.68e-05 tok=15.173B dt=15200 wait=192 +23:14:08 step 115440 loss=3.3299 lr=6.68e-05 tok=15.176B dt=15264 wait=192 +23:15:12 step 115460 loss=3.4302 lr=6.68e-05 tok=15.178B dt=15327 wait=192 +23:16:15 step 115480 loss=3.3222 lr=6.68e-05 tok=15.181B dt=15391 wait=193 +23:17:19 step 115500 loss=3.4267 lr=6.67e-05 tok=15.183B dt=15454 wait=193 +23:17:34 pushed ckpt_w93.pt,log_w93.txt +23:18:49 waited 76s for global r=379 +23:18:53 global r=379 load 4.341 -> 4.306 +23:19:58 step 115520 loss=3.3900 lr=6.67e-05 tok=15.186B dt=15613 wait=193 +23:21:02 step 115540 loss=3.3736 lr=6.67e-05 tok=15.189B dt=15677 wait=194 +23:22:05 step 115560 loss=3.3887 lr=6.67e-05 tok=15.191B dt=15741 wait=194 +23:23:09 step 115580 loss=3.3391 lr=6.67e-05 tok=15.194B dt=15804 wait=194 +23:24:13 step 115600 loss=3.3522 lr=6.66e-05 tok=15.196B dt=15868 wait=195 +23:25:16 step 115620 loss=3.4709 lr=6.66e-05 tok=15.199B dt=15932 wait=195 +23:26:19 step 115640 loss=3.2796 lr=6.66e-05 tok=15.202B dt=15995 wait=195 +23:27:03 pushed ckpt_w93.pt,log_w93.txt +23:28:19 waited 76s for global r=380 +23:28:24 global r=380 load 4.308 -> 4.298 +23:28:57 step 115660 loss=3.4747 lr=6.66e-05 tok=15.204B dt=16152 wait=196 +23:30:00 step 115680 loss=3.2909 lr=6.66e-05 tok=15.207B dt=16216 wait=196 +23:31:04 step 115700 loss=3.3552 lr=6.65e-05 tok=15.210B dt=16279 wait=196 +23:32:08 step 115720 loss=3.4270 lr=6.65e-05 tok=15.212B dt=16343 wait=197 +23:33:11 step 115740 loss=3.2963 lr=6.65e-05 tok=15.215B dt=16407 wait=197 +23:34:15 step 115760 loss=3.4728 lr=6.65e-05 tok=15.217B dt=16470 wait=197 +23:35:18 step 115780 loss=3.3849 lr=6.65e-05 tok=15.220B dt=16534 wait=198 +23:36:22 step 115800 loss=3.4509 lr=6.64e-05 tok=15.223B dt=16597 wait=198 +23:36:36 pushed ckpt_w93.pt,log_w93.txt +23:37:37 waited 61s for global r=381 +23:37:41 global r=381 load 4.267 -> 4.286 +23:38:46 step 115820 loss=3.3573 lr=6.64e-05 tok=15.225B dt=16741 wait=198 +23:39:50 step 115840 loss=3.4049 lr=6.64e-05 tok=15.228B dt=16805 wait=199 +23:40:53 step 115860 loss=3.3772 lr=6.64e-05 tok=15.231B dt=16868 wait=199 +23:41:56 step 115880 loss=3.3715 lr=6.64e-05 tok=15.233B dt=16932 wait=199 +23:43:00 step 115900 loss=3.3637 lr=6.63e-05 tok=15.236B dt=16996 wait=200 +23:44:04 step 115920 loss=3.5044 lr=6.63e-05 tok=15.238B dt=17060 wait=200 +23:45:07 step 115940 loss=3.3655 lr=6.63e-05 tok=15.241B dt=17123 wait=200 +23:45:51 pushed ckpt_w93.pt,log_w93.txt +23:47:07 waited 76s for global r=382 +23:47:13 global r=382 load 4.296 -> 4.287 +23:47:45 step 115960 loss=3.4792 lr=6.63e-05 tok=15.244B dt=17280 wait=201 +23:48:49 step 115980 loss=3.3738 lr=6.62e-05 tok=15.246B dt=17344 wait=201 +23:49:52 step 116000 loss=3.3602 lr=6.62e-05 tok=15.249B dt=17407 wait=201 +23:50:55 step 116020 loss=3.4188 lr=6.62e-05 tok=15.252B dt=17471 wait=202 +23:51:59 step 116040 loss=3.2743 lr=6.62e-05 tok=15.254B dt=17535 wait=202 +23:53:03 step 116060 loss=3.4136 lr=6.62e-05 tok=15.257B dt=17598 wait=203 +23:54:06 step 116080 loss=3.3674 lr=6.61e-05 tok=15.259B dt=17662 wait=203 +23:55:10 step 116100 loss=3.3333 lr=6.61e-05 tok=15.262B dt=17726 wait=203 +23:55:25 pushed ckpt_w93.pt,log_w93.txt +23:56:41 waited 76s for global r=383 +23:56:47 global r=383 load 4.299 -> 4.287 +23:57:52 step 116120 loss=3.2849 lr=6.61e-05 tok=15.265B dt=17887 wait=203 +23:58:55 step 116140 loss=3.3895 lr=6.61e-05 tok=15.267B dt=17950 wait=204 +23:59:58 step 116160 loss=3.3571 lr=6.61e-05 tok=15.270B dt=18014 wait=204 +00:01:02 step 116180 loss=3.3683 lr=6.60e-05 tok=15.273B dt=18078 wait=204 +00:02:06 step 116200 loss=3.3263 lr=6.60e-05 tok=15.275B dt=18141 wait=205 +00:03:09 step 116220 loss=3.4430 lr=6.60e-05 tok=15.278B dt=18205 wait=205 +00:04:13 step 116240 loss=3.3371 lr=6.60e-05 tok=15.280B dt=18268 wait=205 +00:04:59 pushed ckpt_w93.pt,log_w93.txt +00:05:59 waited 61s for global r=384 +00:06:08 global r=384 load 4.320 -> 4.282 +00:06:41 step 116260 loss=3.4912 lr=6.60e-05 tok=15.283B dt=18417 wait=206 +00:07:45 step 116280 loss=3.2945 lr=6.59e-05 tok=15.286B dt=18481 wait=206 +00:08:49 step 116300 loss=3.3274 lr=6.59e-05 tok=15.288B dt=18544 wait=206 +00:09:53 step 116320 loss=3.4459 lr=6.59e-05 tok=15.291B dt=18608 wait=207 +00:10:56 step 116340 loss=3.3200 lr=6.59e-05 tok=15.293B dt=18672 wait=207 +00:12:00 step 116360 loss=3.3816 lr=6.59e-05 tok=15.296B dt=18735 wait=207 +00:13:03 step 116380 loss=3.3326 lr=6.58e-05 tok=15.299B dt=18799 wait=208 +00:14:07 step 116400 loss=3.3929 lr=6.58e-05 tok=15.301B dt=18862 wait=208 +00:14:18 pushed ckpt_w93.pt,log_w93.txt +00:15:34 waited 76s for global r=385 +00:15:39 global r=385 load 4.297 -> 4.298 +00:16:43 step 116420 loss=3.3451 lr=6.58e-05 tok=15.304B dt=19019 wait=208 +00:17:47 step 116440 loss=3.4203 lr=6.58e-05 tok=15.307B dt=19083 wait=208 +00:18:51 step 116460 loss=3.3603 lr=6.58e-05 tok=15.309B dt=19146 wait=209 +00:19:55 step 116480 loss=3.3215 lr=6.57e-05 tok=15.312B dt=19210 wait=209 +00:20:58 step 116500 loss=3.3417 lr=6.57e-05 tok=15.314B dt=19274 wait=209 +00:22:02 step 116520 loss=3.4544 lr=6.57e-05 tok=15.317B dt=19337 wait=210 +00:23:05 step 116540 loss=3.3207 lr=6.57e-05 tok=15.320B dt=19401 wait=210 +00:23:49 pushed ckpt_w93.pt,log_w93.txt +00:24:50 waited 60s for global r=386 +00:24:55 global r=386 load 4.262 -> 4.284 +00:25:28 step 116560 loss=3.4585 lr=6.57e-05 tok=15.322B dt=19543 wait=210 +00:26:31 step 116580 loss=3.3332 lr=6.56e-05 tok=15.325B dt=19607 wait=211 +00:27:34 step 116600 loss=3.3560 lr=6.56e-05 tok=15.328B dt=19670 wait=211 +00:28:38 step 116620 loss=3.4680 lr=6.56e-05 tok=15.330B dt=19734 wait=211 +00:29:42 step 116640 loss=3.3295 lr=6.56e-05 tok=15.333B dt=19797 wait=212 +00:30:46 step 116660 loss=3.4462 lr=6.56e-05 tok=15.335B dt=19861 wait=212 +00:31:49 step 116680 loss=3.3346 lr=6.55e-05 tok=15.338B dt=19925 wait=212 +00:32:53 step 116700 loss=3.3813 lr=6.55e-05 tok=15.341B dt=19988 wait=213 +00:33:06 pushed ckpt_w93.pt,log_w93.txt +00:34:22 waited 76s for global r=387 +00:34:27 global r=387 load 4.277 -> 4.274 +00:35:32 step 116720 loss=3.2907 lr=6.55e-05 tok=15.343B dt=20147 wait=213 +00:36:35 step 116740 loss=3.4372 lr=6.55e-05 tok=15.346B dt=20211 wait=213 +00:37:39 step 116760 loss=3.3661 lr=6.54e-05 tok=15.349B dt=20275 wait=214 +00:38:43 step 116780 loss=3.3622 lr=6.54e-05 tok=15.351B dt=20338 wait=214 +00:39:46 step 116800 loss=3.3585 lr=6.54e-05 tok=15.354B dt=20402 wait=214 +00:40:50 step 116820 loss=3.4766 lr=6.54e-05 tok=15.356B dt=20466 wait=215 +00:41:54 step 116840 loss=3.3095 lr=6.54e-05 tok=15.359B dt=20529 wait=215 +00:42:38 pushed ckpt_w93.pt,log_w93.txt +00:43:38 waited 61s for global r=388 +00:43:46 global r=388 load 4.325 -> 4.278 +00:44:19 step 116860 loss=3.4440 lr=6.53e-05 tok=15.362B dt=20675 wait=215 +00:45:23 step 116880 loss=3.3046 lr=6.53e-05 tok=15.364B dt=20738 wait=216 +00:46:26 step 116900 loss=3.3474 lr=6.53e-05 tok=15.367B dt=20802 wait=216 +00:47:30 step 116920 loss=3.3549 lr=6.53e-05 tok=15.370B dt=20866 wait=216 +00:48:34 step 116940 loss=3.3376 lr=6.53e-05 tok=15.372B dt=20929 wait=217 +00:49:38 step 116960 loss=3.4116 lr=6.52e-05 tok=15.375B dt=20993 wait=217 +00:50:41 step 116980 loss=3.3371 lr=6.52e-05 tok=15.377B dt=21056 wait=217 +00:51:44 step 117000 loss=3.4078 lr=6.52e-05 tok=15.380B dt=21120 wait=217 diff --git a/log_w94.txt b/log_w94.txt new file mode 100644 index 0000000000000000000000000000000000000000..631bff646e5b554be55c7525f6b7a0f7ee61fc7e --- /dev/null +++ b/log_w94.txt @@ -0,0 +1,183 @@ +launch +18:57:44 name=w94 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 41/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:57:51 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:57:51 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:58:00 params 1.08B +18:58:02 resumed step=111600 seen=14.67B round=354 +18:58:02 stream salt=111600 +18:59:52 blocks compiled +18:59:52 start: accum=4, 131072 tokens per optimiser step +19:02:18 step 111601 loss=3.1751 lr=7.08e-05 tok=14.666B dt=147 wait=104 +19:02:22 step 111602 loss=3.2661 lr=7.08e-05 tok=14.666B dt=150 wait=104 +19:03:25 step 111620 loss=3.4520 lr=7.08e-05 tok=14.668B dt=213 wait=105 +19:04:31 step 111640 loss=3.3735 lr=7.07e-05 tok=14.671B dt=280 wait=105 +19:05:35 step 111660 loss=3.4399 lr=7.07e-05 tok=14.674B dt=344 wait=106 +19:06:39 step 111680 loss=3.3787 lr=7.07e-05 tok=14.676B dt=408 wait=106 +19:07:44 step 111700 loss=3.4809 lr=7.07e-05 tok=14.679B dt=473 wait=106 +19:08:49 step 111720 loss=3.4422 lr=7.07e-05 tok=14.681B dt=537 wait=107 +19:09:52 step 111740 loss=3.4350 lr=7.06e-05 tok=14.684B dt=601 wait=107 +19:10:40 pushed ckpt_w94.pt,log_w94.txt +19:12:11 waited 91s for global r=355 +19:12:25 global r=355 load 4.321 -> 4.287 +19:13:00 step 111760 loss=3.4757 lr=7.06e-05 tok=14.687B dt=788 wait=107 +19:14:04 step 111780 loss=3.3090 lr=7.06e-05 tok=14.689B dt=852 wait=107 +19:15:08 step 111800 loss=3.3673 lr=7.06e-05 tok=14.692B dt=916 wait=108 +19:16:12 step 111820 loss=3.4434 lr=7.06e-05 tok=14.694B dt=980 wait=108 +19:17:15 step 111840 loss=3.3654 lr=7.05e-05 tok=14.697B dt=1044 wait=108 +19:18:19 step 111860 loss=3.3894 lr=7.05e-05 tok=14.700B dt=1108 wait=109 +19:19:24 step 111880 loss=3.3728 lr=7.05e-05 tok=14.702B dt=1172 wait=109 +19:20:28 step 111900 loss=3.4228 lr=7.05e-05 tok=14.705B dt=1236 wait=109 +19:20:42 pushed ckpt_w94.pt,log_w94.txt +19:24:50 step 111920 loss=3.3639 lr=7.05e-05 tok=14.708B dt=1498 wait=110 +19:25:54 step 111940 loss=3.3471 lr=7.04e-05 tok=14.710B dt=1562 wait=110 +19:26:58 step 111960 loss=3.3228 lr=7.04e-05 tok=14.713B dt=1626 wait=110 +19:28:01 step 111980 loss=3.2891 lr=7.04e-05 tok=14.715B dt=1690 wait=111 +19:29:05 step 112000 loss=3.2464 lr=7.04e-05 tok=14.718B dt=1754 wait=111 +19:30:09 step 112020 loss=3.4496 lr=7.04e-05 tok=14.721B dt=1818 wait=111 +19:31:13 step 112040 loss=3.3598 lr=7.03e-05 tok=14.723B dt=1882 wait=112 +19:32:09 pushed ckpt_w94.pt,log_w94.txt +19:33:09 waited 61s for global r=356 +19:33:17 global r=356 load 4.298 -> 4.281 +19:33:51 step 112060 loss=3.5502 lr=7.03e-05 tok=14.726B dt=2039 wait=112 +19:34:55 step 112080 loss=3.3379 lr=7.03e-05 tok=14.729B dt=2103 wait=112 +19:35:59 step 112100 loss=3.3544 lr=7.03e-05 tok=14.731B dt=2167 wait=113 +19:37:03 step 112120 loss=3.5094 lr=7.02e-05 tok=14.734B dt=2231 wait=113 +19:38:08 step 112140 loss=3.2770 lr=7.02e-05 tok=14.736B dt=2296 wait=113 +19:39:12 step 112160 loss=3.4610 lr=7.02e-05 tok=14.739B dt=2361 wait=113 +19:40:16 step 112180 loss=3.3369 lr=7.02e-05 tok=14.742B dt=2424 wait=114 +19:41:20 step 112200 loss=3.4297 lr=7.02e-05 tok=14.744B dt=2489 wait=114 +19:41:32 pushed ckpt_w94.pt,log_w94.txt +19:42:33 waited 61s for global r=357 +19:42:41 global r=357 load 4.305 -> 4.278 +19:43:46 step 112220 loss=3.3202 lr=7.01e-05 tok=14.747B dt=2634 wait=114 +19:44:50 step 112240 loss=3.3835 lr=7.01e-05 tok=14.750B dt=2698 wait=115 +19:45:54 step 112260 loss=3.3989 lr=7.01e-05 tok=14.752B dt=2763 wait=115 +19:46:58 step 112280 loss=3.3130 lr=7.01e-05 tok=14.755B dt=2827 wait=115 +19:48:02 step 112300 loss=3.3737 lr=7.01e-05 tok=14.757B dt=2891 wait=116 +19:49:07 step 112320 loss=3.4275 lr=7.00e-05 tok=14.760B dt=2955 wait=116 +19:50:11 step 112340 loss=3.3330 lr=7.00e-05 tok=14.763B dt=3019 wait=116 +19:50:57 pushed ckpt_w94.pt,log_w94.txt +19:52:12 waited 76s for global r=358 +19:52:21 global r=358 load 4.318 -> 4.291 +19:52:55 step 112360 loss=3.4305 lr=7.00e-05 tok=14.765B dt=3183 wait=117 +19:53:59 step 112380 loss=3.3821 lr=7.00e-05 tok=14.768B dt=3247 wait=117 +19:55:03 step 112400 loss=3.3559 lr=7.00e-05 tok=14.771B dt=3311 wait=117 +19:56:07 step 112420 loss=3.4592 lr=6.99e-05 tok=14.773B dt=3376 wait=118 +19:57:12 step 112440 loss=3.2836 lr=6.99e-05 tok=14.776B dt=3440 wait=118 +19:58:16 step 112460 loss=3.4356 lr=6.99e-05 tok=14.778B dt=3505 wait=118 +19:59:20 step 112480 loss=3.3258 lr=6.99e-05 tok=14.781B dt=3569 wait=119 +20:00:25 step 112500 loss=3.3959 lr=6.99e-05 tok=14.784B dt=3633 wait=119 +20:00:37 pushed ckpt_w94.pt,log_w94.txt +20:04:18 global r=359 load 4.293 -> 4.281 +20:05:22 step 112520 loss=3.3489 lr=6.98e-05 tok=14.788B dt=3930 wait=119 +20:06:26 step 112540 loss=3.3532 lr=6.98e-05 tok=14.790B dt=3994 wait=120 +20:07:30 step 112560 loss=3.3669 lr=6.98e-05 tok=14.793B dt=4058 wait=120 +20:08:34 step 112580 loss=3.3593 lr=6.98e-05 tok=14.795B dt=4123 wait=120 +20:09:38 step 112600 loss=3.4280 lr=6.97e-05 tok=14.798B dt=4187 wait=121 +20:10:43 step 112620 loss=3.4683 lr=6.97e-05 tok=14.801B dt=4251 wait=121 +20:11:47 step 112640 loss=3.3103 lr=6.97e-05 tok=14.803B dt=4316 wait=121 +20:12:34 pushed ckpt_w94.pt,log_w94.txt +20:13:19 waited 46s for global r=360 +20:13:26 global r=360 load 4.312 -> 4.290 +20:14:00 step 112660 loss=3.4054 lr=6.97e-05 tok=14.806B dt=4448 wait=122 +20:15:04 step 112680 loss=3.2898 lr=6.97e-05 tok=14.809B dt=4512 wait=122 +20:16:08 step 112700 loss=3.3512 lr=6.96e-05 tok=14.811B dt=4576 wait=122 +20:17:12 step 112720 loss=3.4610 lr=6.96e-05 tok=14.814B dt=4640 wait=123 +20:18:16 step 112740 loss=3.3209 lr=6.96e-05 tok=14.816B dt=4704 wait=123 +20:19:20 step 112760 loss=3.4397 lr=6.96e-05 tok=14.819B dt=4768 wait=123 +20:20:24 step 112780 loss=3.4045 lr=6.96e-05 tok=14.822B dt=4832 wait=123 +20:21:28 step 112800 loss=3.4149 lr=6.95e-05 tok=14.824B dt=4897 wait=124 +20:21:42 pushed ckpt_w94.pt,log_w94.txt +20:22:57 waited 76s for global r=361 +20:23:11 global r=361 load 4.340 -> 4.289 +20:24:16 step 112820 loss=3.3389 lr=6.95e-05 tok=14.827B dt=5064 wait=124 +20:25:25 step 112840 loss=3.3505 lr=6.95e-05 tok=14.829B dt=5133 wait=124 +20:26:29 step 112860 loss=3.3522 lr=6.95e-05 tok=14.832B dt=5197 wait=125 +20:27:33 step 112880 loss=3.3513 lr=6.95e-05 tok=14.835B dt=5261 wait=125 +20:28:37 step 112900 loss=3.3568 lr=6.94e-05 tok=14.837B dt=5325 wait=125 +20:29:41 step 112920 loss=3.4492 lr=6.94e-05 tok=14.840B dt=5390 wait=126 +20:30:46 step 112940 loss=3.2828 lr=6.94e-05 tok=14.843B dt=5454 wait=126 +20:31:31 pushed ckpt_w94.pt,log_w94.txt +20:32:17 waited 46s for global r=362 +20:32:23 global r=362 load 4.286 -> 4.299 +20:32:57 step 112960 loss=3.4614 lr=6.94e-05 tok=14.845B dt=5585 wait=126 +20:34:00 step 112980 loss=3.2795 lr=6.94e-05 tok=14.848B dt=5649 wait=127 +20:35:05 step 113000 loss=3.3107 lr=6.93e-05 tok=14.850B dt=5713 wait=127 +20:36:09 step 113020 loss=3.4199 lr=6.93e-05 tok=14.853B dt=5777 wait=127 +20:37:13 step 113040 loss=3.3675 lr=6.93e-05 tok=14.856B dt=5841 wait=127 +20:38:17 step 113060 loss=3.4493 lr=6.93e-05 tok=14.858B dt=5905 wait=128 +20:39:21 step 113080 loss=3.3914 lr=6.92e-05 tok=14.861B dt=5969 wait=128 +20:40:25 step 113100 loss=3.3848 lr=6.92e-05 tok=14.864B dt=6034 wait=128 +20:40:39 pushed ckpt_w94.pt,log_w94.txt +20:41:41 waited 62s for global r=363 +20:41:50 global r=363 load 4.279 -> 4.289 +20:42:57 step 113120 loss=3.3487 lr=6.92e-05 tok=14.866B dt=6185 wait=129 +20:44:01 step 113140 loss=3.4119 lr=6.92e-05 tok=14.869B dt=6250 wait=129 +20:45:06 step 113160 loss=3.2692 lr=6.92e-05 tok=14.871B dt=6315 wait=129 +20:46:10 step 113180 loss=3.3560 lr=6.91e-05 tok=14.874B dt=6379 wait=130 +20:47:14 step 113200 loss=3.3599 lr=6.91e-05 tok=14.877B dt=6443 wait=130 +20:48:19 step 113220 loss=3.3656 lr=6.91e-05 tok=14.879B dt=6507 wait=130 +20:49:23 step 113240 loss=3.2674 lr=6.91e-05 tok=14.882B dt=6572 wait=131 +20:50:10 pushed ckpt_w94.pt,log_w94.txt +20:51:11 waited 61s for global r=364 +20:51:25 global r=364 load 4.277 -> 4.284 +20:51:58 step 113260 loss=3.4434 lr=6.91e-05 tok=14.885B dt=6726 wait=131 +20:53:02 step 113280 loss=3.3110 lr=6.90e-05 tok=14.887B dt=6790 wait=131 +20:54:06 step 113300 loss=3.3143 lr=6.90e-05 tok=14.890B dt=6854 wait=132 +20:55:10 step 113320 loss=3.4468 lr=6.90e-05 tok=14.892B dt=6918 wait=132 +20:56:14 step 113340 loss=3.3265 lr=6.90e-05 tok=14.895B dt=6983 wait=132 +20:57:19 step 113360 loss=3.3982 lr=6.90e-05 tok=14.898B dt=7047 wait=133 +20:58:23 step 113380 loss=3.3741 lr=6.89e-05 tok=14.900B dt=7111 wait=133 +20:59:27 step 113400 loss=3.4043 lr=6.89e-05 tok=14.903B dt=7175 wait=133 +20:59:49 pushed ckpt_w94.pt,log_w94.txt +21:00:35 waited 46s for global r=365 +21:00:45 global r=365 load 4.292 -> 4.298 +21:01:51 step 113420 loss=3.3231 lr=6.89e-05 tok=14.906B dt=7319 wait=134 +21:02:55 step 113440 loss=3.4526 lr=6.89e-05 tok=14.908B dt=7383 wait=134 +21:03:59 step 113460 loss=3.3503 lr=6.89e-05 tok=14.911B dt=7448 wait=134 +21:05:03 step 113480 loss=3.3930 lr=6.88e-05 tok=14.913B dt=7512 wait=134 +21:06:08 step 113500 loss=3.3504 lr=6.88e-05 tok=14.916B dt=7576 wait=135 +21:07:13 step 113520 loss=3.4159 lr=6.88e-05 tok=14.919B dt=7641 wait=135 +21:08:17 step 113540 loss=3.3139 lr=6.88e-05 tok=14.921B dt=7705 wait=135 +21:09:01 pushed ckpt_w94.pt,log_w94.txt +21:10:02 waited 61s for global r=366 +21:10:11 global r=366 load 4.292 -> 4.288 +21:10:45 step 113560 loss=3.4252 lr=6.88e-05 tok=14.924B dt=7853 wait=136 +21:11:48 step 113580 loss=3.3088 lr=6.87e-05 tok=14.926B dt=7917 wait=136 +21:12:52 step 113600 loss=3.2543 lr=6.87e-05 tok=14.929B dt=7981 wait=136 +21:13:57 step 113620 loss=3.4321 lr=6.87e-05 tok=14.932B dt=8045 wait=137 +21:15:01 step 113640 loss=3.2972 lr=6.87e-05 tok=14.934B dt=8109 wait=137 +21:16:06 step 113660 loss=3.0984 lr=6.86e-05 tok=14.937B dt=8174 wait=137 +21:17:11 step 113680 loss=3.2058 lr=6.86e-05 tok=14.940B dt=8240 wait=138 +21:18:15 step 113700 loss=3.4048 lr=6.86e-05 tok=14.942B dt=8303 wait=138 +21:18:27 pushed ckpt_w94.pt,log_w94.txt +21:19:28 waited 61s for global r=367 +21:19:36 global r=367 load 4.331 -> 4.285 +21:20:43 step 113720 loss=3.3633 lr=6.86e-05 tok=14.945B dt=8451 wait=138 +21:21:47 step 113740 loss=3.3649 lr=6.86e-05 tok=14.947B dt=8515 wait=139 +21:22:51 step 113760 loss=3.4016 lr=6.85e-05 tok=14.950B dt=8579 wait=139 +21:23:55 step 113780 loss=3.3318 lr=6.85e-05 tok=14.953B dt=8643 wait=139 +21:24:59 step 113800 loss=3.3252 lr=6.85e-05 tok=14.955B dt=8707 wait=140 +21:26:03 step 113820 loss=3.4713 lr=6.85e-05 tok=14.958B dt=8772 wait=140 +21:27:07 step 113840 loss=3.3576 lr=6.85e-05 tok=14.961B dt=8835 wait=140 +21:27:55 pushed ckpt_w94.pt,log_w94.txt +21:28:55 waited 61s for global r=368 +21:29:03 global r=368 load 4.269 -> 4.284 +21:29:36 step 113860 loss=3.4886 lr=6.84e-05 tok=14.963B dt=8985 wait=140 +21:30:40 step 113880 loss=3.3360 lr=6.84e-05 tok=14.966B dt=9049 wait=141 +21:31:44 step 113900 loss=3.3460 lr=6.84e-05 tok=14.968B dt=9113 wait=141 +21:32:49 step 113920 loss=3.3777 lr=6.84e-05 tok=14.971B dt=9178 wait=141 +21:33:53 step 113940 loss=3.3596 lr=6.84e-05 tok=14.974B dt=9242 wait=142 +21:34:57 step 113960 loss=3.4384 lr=6.83e-05 tok=14.976B dt=9306 wait=142 +21:36:01 step 113980 loss=3.3926 lr=6.83e-05 tok=14.979B dt=9370 wait=142 +21:37:05 step 114000 loss=3.4024 lr=6.83e-05 tok=14.982B dt=9434 wait=143 +21:37:18 pushed ckpt_w94.pt,log_w94.txt +21:38:19 waited 61s for global r=369 +21:38:24 global r=369 load 4.283 -> 4.302 +21:39:29 step 114020 loss=3.3332 lr=6.83e-05 tok=14.984B dt=9578 wait=143 +21:40:34 step 114040 loss=3.3461 lr=6.83e-05 tok=14.987B dt=9642 wait=143 +21:41:38 step 114060 loss=3.3866 lr=6.82e-05 tok=14.989B dt=9706 wait=144 +21:42:42 step 114080 loss=3.3392 lr=6.82e-05 tok=14.992B dt=9770 wait=144 +21:43:46 step 114100 loss=3.3685 lr=6.82e-05 tok=14.995B dt=9834 wait=144 +21:44:50 step 114120 loss=3.4579 lr=6.82e-05 tok=14.997B dt=9899 wait=145 +21:45:55 step 114140 loss=3.3298 lr=6.81e-05 tok=15.000B dt=9963 wait=145 diff --git a/log_w95.txt b/log_w95.txt new file mode 100644 index 0000000000000000000000000000000000000000..aa5149a4528e3800bac2d7cc277f7af65796ea1e --- /dev/null +++ b/log_w95.txt @@ -0,0 +1,380 @@ +launch +18:57:47 name=w95 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 42/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +18:57:53 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +18:57:53 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +18:58:02 params 1.08B +18:58:04 resumed step=111600 seen=14.67B round=354 +18:58:04 stream salt=111600 +18:59:51 blocks compiled +18:59:51 start: accum=4, 131072 tokens per optimiser step +19:02:33 step 111601 loss=3.1698 lr=7.08e-05 tok=14.666B dt=162 wait=122 +19:02:36 step 111602 loss=3.1765 lr=7.08e-05 tok=14.666B dt=166 wait=122 +19:03:40 step 111620 loss=3.3864 lr=7.08e-05 tok=14.668B dt=230 wait=123 +19:04:44 step 111640 loss=3.4032 lr=7.07e-05 tok=14.671B dt=293 wait=123 +19:05:47 step 111660 loss=3.4437 lr=7.07e-05 tok=14.674B dt=357 wait=123 +19:06:51 step 111680 loss=3.3980 lr=7.07e-05 tok=14.676B dt=420 wait=124 +19:07:55 step 111700 loss=3.4983 lr=7.07e-05 tok=14.679B dt=484 wait=124 +19:08:58 step 111720 loss=3.4491 lr=7.07e-05 tok=14.681B dt=548 wait=124 +19:10:01 step 111740 loss=3.3753 lr=7.06e-05 tok=14.684B dt=611 wait=125 +19:10:47 pushed ckpt_w95.pt,log_w95.txt +19:12:03 waited 76s for global r=355 +19:12:19 global r=355 load 4.317 -> 4.287 +19:12:54 step 111760 loss=3.4331 lr=7.06e-05 tok=14.687B dt=783 wait=125 +19:13:57 step 111780 loss=3.2928 lr=7.06e-05 tok=14.689B dt=846 wait=125 +19:15:00 step 111800 loss=3.3314 lr=7.06e-05 tok=14.692B dt=909 wait=126 +19:16:04 step 111820 loss=3.3770 lr=7.06e-05 tok=14.694B dt=973 wait=126 +19:17:07 step 111840 loss=3.2744 lr=7.05e-05 tok=14.697B dt=1036 wait=126 +19:18:10 step 111860 loss=3.3846 lr=7.05e-05 tok=14.700B dt=1099 wait=127 +19:19:13 step 111880 loss=3.3409 lr=7.05e-05 tok=14.702B dt=1162 wait=127 +19:20:16 step 111900 loss=3.3670 lr=7.05e-05 tok=14.705B dt=1226 wait=127 +19:20:30 pushed ckpt_w95.pt,log_w95.txt +19:24:37 step 111920 loss=3.3741 lr=7.05e-05 tok=14.708B dt=1486 wait=128 +19:25:40 step 111940 loss=3.3603 lr=7.04e-05 tok=14.710B dt=1550 wait=128 +19:26:43 step 111960 loss=3.2983 lr=7.04e-05 tok=14.713B dt=1613 wait=128 +19:27:46 step 111980 loss=3.2739 lr=7.04e-05 tok=14.715B dt=1676 wait=129 +19:28:49 step 112000 loss=3.2306 lr=7.04e-05 tok=14.718B dt=1739 wait=129 +19:29:53 step 112020 loss=3.4273 lr=7.04e-05 tok=14.721B dt=1802 wait=129 +19:30:56 step 112040 loss=3.3273 lr=7.03e-05 tok=14.723B dt=1865 wait=130 +19:31:45 pushed ckpt_w95.pt,log_w95.txt +19:33:16 waited 91s for global r=356 +19:33:25 global r=356 load 4.292 -> 4.281 +19:33:59 step 112060 loss=3.4839 lr=7.03e-05 tok=14.726B dt=2049 wait=130 +19:35:02 step 112080 loss=3.3076 lr=7.03e-05 tok=14.729B dt=2112 wait=130 +19:36:05 step 112100 loss=3.2883 lr=7.03e-05 tok=14.731B dt=2175 wait=131 +19:37:08 step 112120 loss=3.3544 lr=7.02e-05 tok=14.734B dt=2238 wait=131 +19:38:11 step 112140 loss=3.2889 lr=7.02e-05 tok=14.736B dt=2300 wait=131 +19:39:14 step 112160 loss=3.4427 lr=7.02e-05 tok=14.739B dt=2364 wait=132 +19:40:18 step 112180 loss=3.3060 lr=7.02e-05 tok=14.742B dt=2427 wait=132 +19:41:21 step 112200 loss=3.3892 lr=7.02e-05 tok=14.744B dt=2490 wait=132 +19:41:34 pushed ckpt_w95.pt,log_w95.txt +19:42:35 waited 60s for global r=357 +19:42:41 global r=357 load 4.287 -> 4.278 +19:43:46 step 112220 loss=3.2962 lr=7.01e-05 tok=14.747B dt=2635 wait=133 +19:44:49 step 112240 loss=3.3784 lr=7.01e-05 tok=14.750B dt=2698 wait=133 +19:45:52 step 112260 loss=3.3976 lr=7.01e-05 tok=14.752B dt=2762 wait=133 +19:46:56 step 112280 loss=3.3161 lr=7.01e-05 tok=14.755B dt=2825 wait=134 +19:47:59 step 112300 loss=3.3500 lr=7.01e-05 tok=14.757B dt=2888 wait=134 +19:49:03 step 112320 loss=3.4271 lr=7.00e-05 tok=14.760B dt=2952 wait=134 +19:50:06 step 112340 loss=3.2699 lr=7.00e-05 tok=14.763B dt=3015 wait=135 +19:50:50 pushed ckpt_w95.pt,log_w95.txt +19:52:20 waited 91s for global r=358 +19:52:27 global r=358 load 4.308 -> 4.291 +19:53:00 step 112360 loss=3.4232 lr=7.00e-05 tok=14.765B dt=3189 wait=135 +19:55:05 step 112380 loss=3.2681 lr=7.00e-05 tok=14.768B dt=3315 wait=135 +19:57:16 step 112400 loss=3.2537 lr=7.00e-05 tok=14.771B dt=3446 wait=136 +19:59:30 step 112420 loss=3.4429 lr=6.99e-05 tok=14.773B dt=3580 wait=136 +20:01:45 step 112440 loss=3.2471 lr=6.99e-05 tok=14.776B dt=3714 wait=136 +20:03:58 step 112460 loss=3.4280 lr=6.99e-05 tok=14.778B dt=3847 wait=137 +20:04:06 global r=359 load 4.305 -> 4.281 +20:05:39 step 112520 loss=3.4512 lr=6.98e-05 tok=14.781B dt=3949 wait=137 +20:06:42 step 112540 loss=3.3639 lr=6.98e-05 tok=14.784B dt=4012 wait=137 +20:07:45 step 112560 loss=3.3853 lr=6.98e-05 tok=14.786B dt=4074 wait=138 +20:08:48 step 112580 loss=3.3048 lr=6.98e-05 tok=14.789B dt=4137 wait=138 +20:09:52 step 112600 loss=3.2991 lr=6.97e-05 tok=14.791B dt=4201 wait=138 +20:10:56 step 112620 loss=3.3153 lr=6.97e-05 tok=14.794B dt=4265 wait=139 +20:11:59 step 112640 loss=3.3095 lr=6.97e-05 tok=14.797B dt=4328 wait=139 +20:12:44 pushed ckpt_w95.pt,log_w95.txt +20:13:30 waited 45s for global r=360 +20:13:35 global r=360 load 4.289 -> 4.290 +20:14:08 step 112660 loss=3.4062 lr=6.97e-05 tok=14.799B dt=4458 wait=139 +20:15:12 step 112680 loss=3.2856 lr=6.97e-05 tok=14.802B dt=4521 wait=140 +20:16:15 step 112700 loss=3.3045 lr=6.96e-05 tok=14.805B dt=4584 wait=140 +20:17:19 step 112720 loss=3.3874 lr=6.96e-05 tok=14.807B dt=4648 wait=140 +20:18:22 step 112740 loss=3.2759 lr=6.96e-05 tok=14.810B dt=4711 wait=141 +20:19:25 step 112760 loss=3.4155 lr=6.96e-05 tok=14.812B dt=4774 wait=141 +20:20:29 step 112780 loss=3.3070 lr=6.96e-05 tok=14.815B dt=4838 wait=141 +20:21:32 step 112800 loss=3.3639 lr=6.95e-05 tok=14.818B dt=4902 wait=142 +20:21:44 pushed ckpt_w95.pt,log_w95.txt +20:22:45 waited 61s for global r=361 +20:22:54 global r=361 load 4.298 -> 4.289 +20:23:59 step 112820 loss=3.3295 lr=6.95e-05 tok=14.820B dt=5048 wait=142 +20:25:02 step 112840 loss=3.3522 lr=6.95e-05 tok=14.823B dt=5111 wait=142 +20:26:05 step 112860 loss=3.3378 lr=6.95e-05 tok=14.826B dt=5175 wait=143 +20:27:09 step 112880 loss=3.3248 lr=6.95e-05 tok=14.828B dt=5238 wait=143 +20:28:12 step 112900 loss=3.3232 lr=6.94e-05 tok=14.831B dt=5301 wait=143 +20:29:15 step 112920 loss=3.3900 lr=6.94e-05 tok=14.833B dt=5364 wait=144 +20:30:18 step 112940 loss=3.2630 lr=6.94e-05 tok=14.836B dt=5427 wait=144 +20:31:03 pushed ckpt_w95.pt,log_w95.txt +20:32:18 waited 76s for global r=362 +20:32:24 global r=362 load 4.309 -> 4.299 +20:32:56 step 112960 loss=3.3657 lr=6.94e-05 tok=14.839B dt=5586 wait=144 +20:34:00 step 112980 loss=3.2813 lr=6.94e-05 tok=14.841B dt=5649 wait=145 +20:35:03 step 113000 loss=3.2727 lr=6.93e-05 tok=14.844B dt=5712 wait=145 +20:36:06 step 113020 loss=3.3895 lr=6.93e-05 tok=14.847B dt=5775 wait=145 +20:37:09 step 113040 loss=3.2964 lr=6.93e-05 tok=14.849B dt=5839 wait=146 +20:38:13 step 113060 loss=3.3646 lr=6.93e-05 tok=14.852B dt=5902 wait=146 +20:39:16 step 113080 loss=3.3262 lr=6.92e-05 tok=14.854B dt=5965 wait=146 +20:40:19 step 113100 loss=3.2987 lr=6.92e-05 tok=14.857B dt=6028 wait=147 +20:40:34 pushed ckpt_w95.pt,log_w95.txt +20:41:50 waited 76s for global r=363 +20:41:56 global r=363 load 4.335 -> 4.289 +20:43:01 step 113120 loss=3.2955 lr=6.92e-05 tok=14.860B dt=6191 wait=147 +20:44:04 step 113140 loss=3.3844 lr=6.92e-05 tok=14.862B dt=6254 wait=148 +20:45:08 step 113160 loss=3.3651 lr=6.92e-05 tok=14.865B dt=6317 wait=148 +20:46:11 step 113180 loss=3.3075 lr=6.91e-05 tok=14.867B dt=6380 wait=148 +20:47:14 step 113200 loss=3.3448 lr=6.91e-05 tok=14.870B dt=6444 wait=149 +20:48:18 step 113220 loss=3.3568 lr=6.91e-05 tok=14.873B dt=6507 wait=149 +20:49:21 step 113240 loss=3.2767 lr=6.91e-05 tok=14.875B dt=6570 wait=149 +20:50:08 pushed ckpt_w95.pt,log_w95.txt +20:51:09 waited 61s for global r=364 +20:51:17 global r=364 load 4.330 -> 4.284 +20:51:50 step 113260 loss=3.4378 lr=6.91e-05 tok=14.878B dt=6720 wait=150 +20:52:53 step 113280 loss=3.3842 lr=6.90e-05 tok=14.881B dt=6783 wait=150 +20:53:57 step 113300 loss=3.3074 lr=6.90e-05 tok=14.883B dt=6846 wait=150 +20:55:01 step 113320 loss=3.3326 lr=6.90e-05 tok=14.886B dt=6910 wait=151 +20:56:04 step 113340 loss=3.3125 lr=6.90e-05 tok=14.888B dt=6973 wait=151 +20:57:07 step 113360 loss=3.4096 lr=6.90e-05 tok=14.891B dt=7037 wait=151 +20:58:10 step 113380 loss=3.3146 lr=6.89e-05 tok=14.894B dt=7100 wait=151 +20:59:13 step 113400 loss=3.2980 lr=6.89e-05 tok=14.896B dt=7163 wait=152 +20:59:27 pushed ckpt_w95.pt,log_w95.txt +21:00:43 waited 76s for global r=365 +21:00:50 global r=365 load 4.323 -> 4.298 +21:01:55 step 113420 loss=3.3165 lr=6.89e-05 tok=14.899B dt=7324 wait=152 +21:02:58 step 113440 loss=3.3681 lr=6.89e-05 tok=14.902B dt=7387 wait=152 +21:04:01 step 113460 loss=3.3337 lr=6.89e-05 tok=14.904B dt=7451 wait=153 +21:05:04 step 113480 loss=3.2832 lr=6.88e-05 tok=14.907B dt=7514 wait=153 +21:06:07 step 113500 loss=3.3647 lr=6.88e-05 tok=14.909B dt=7577 wait=153 +21:07:11 step 113520 loss=3.4067 lr=6.88e-05 tok=14.912B dt=7640 wait=154 +21:08:14 step 113540 loss=3.3097 lr=6.88e-05 tok=14.915B dt=7703 wait=154 +21:09:03 pushed ckpt_w95.pt,log_w95.txt +21:10:04 waited 61s for global r=366 +21:10:13 global r=366 load 4.288 -> 4.288 +21:10:46 step 113560 loss=3.3755 lr=6.88e-05 tok=14.917B dt=7855 wait=154 +21:11:49 step 113580 loss=3.3314 lr=6.87e-05 tok=14.920B dt=7918 wait=155 +21:12:52 step 113600 loss=3.3490 lr=6.87e-05 tok=14.923B dt=7982 wait=155 +21:13:56 step 113620 loss=3.3807 lr=6.87e-05 tok=14.925B dt=8045 wait=155 +21:14:58 step 113640 loss=3.2522 lr=6.87e-05 tok=14.928B dt=8108 wait=156 +21:16:02 step 113660 loss=3.3919 lr=6.86e-05 tok=14.930B dt=8171 wait=156 +21:17:05 step 113680 loss=3.3163 lr=6.86e-05 tok=14.933B dt=8234 wait=156 +21:18:08 step 113700 loss=3.3578 lr=6.86e-05 tok=14.936B dt=8297 wait=157 +21:18:20 pushed ckpt_w95.pt,log_w95.txt +21:19:36 waited 76s for global r=367 +21:19:45 global r=367 load 4.268 -> 4.285 +21:20:50 step 113720 loss=3.2985 lr=6.86e-05 tok=14.938B dt=8459 wait=157 +21:21:53 step 113740 loss=3.3368 lr=6.86e-05 tok=14.941B dt=8523 wait=157 +21:22:57 step 113760 loss=3.3708 lr=6.85e-05 tok=14.944B dt=8586 wait=158 +21:24:00 step 113780 loss=3.3111 lr=6.85e-05 tok=14.946B dt=8649 wait=158 +21:25:03 step 113800 loss=3.3250 lr=6.85e-05 tok=14.949B dt=8712 wait=158 +21:26:06 step 113820 loss=3.4536 lr=6.85e-05 tok=14.951B dt=8776 wait=159 +21:27:09 step 113840 loss=3.2927 lr=6.85e-05 tok=14.954B dt=8839 wait=159 +21:27:58 pushed ckpt_w95.pt,log_w95.txt +21:28:58 waited 60s for global r=368 +21:29:04 global r=368 load 4.309 -> 4.284 +21:29:37 step 113860 loss=3.3445 lr=6.84e-05 tok=14.957B dt=8986 wait=159 +21:30:40 step 113880 loss=3.3190 lr=6.84e-05 tok=14.959B dt=9050 wait=160 +21:31:43 step 113900 loss=3.3179 lr=6.84e-05 tok=14.962B dt=9113 wait=160 +21:32:47 step 113920 loss=3.4060 lr=6.84e-05 tok=14.964B dt=9176 wait=160 +21:33:50 step 113940 loss=3.3123 lr=6.84e-05 tok=14.967B dt=9239 wait=161 +21:34:53 step 113960 loss=3.3823 lr=6.83e-05 tok=14.970B dt=9303 wait=161 +21:35:56 step 113980 loss=3.3504 lr=6.83e-05 tok=14.972B dt=9366 wait=161 +21:36:59 step 114000 loss=3.3185 lr=6.83e-05 tok=14.975B dt=9429 wait=162 +21:37:15 pushed ckpt_w95.pt,log_w95.txt +21:38:15 waited 60s for global r=369 +21:38:23 global r=369 load 4.295 -> 4.302 +21:39:27 step 114020 loss=3.3016 lr=6.83e-05 tok=14.978B dt=9576 wait=162 +21:40:30 step 114040 loss=3.3859 lr=6.83e-05 tok=14.980B dt=9639 wait=162 +21:41:33 step 114060 loss=3.3355 lr=6.82e-05 tok=14.983B dt=9703 wait=163 +21:42:37 step 114080 loss=3.2584 lr=6.82e-05 tok=14.985B dt=9766 wait=163 +21:43:40 step 114100 loss=3.3078 lr=6.82e-05 tok=14.988B dt=9829 wait=163 +21:44:44 step 114120 loss=3.3073 lr=6.82e-05 tok=14.991B dt=9893 wait=164 +21:45:47 step 114140 loss=3.2309 lr=6.81e-05 tok=14.993B dt=9956 wait=164 +21:46:33 pushed ckpt_w95.pt,log_w95.txt +21:47:34 waited 61s for global r=370 +21:47:42 global r=370 load 4.313 -> 4.308 +21:48:15 step 114160 loss=3.4092 lr=6.81e-05 tok=14.996B dt=10104 wait=164 +21:49:18 step 114180 loss=3.2746 lr=6.81e-05 tok=14.999B dt=10167 wait=165 +21:50:21 step 114200 loss=3.3231 lr=6.81e-05 tok=15.001B dt=10230 wait=165 +21:51:24 step 114220 loss=3.4020 lr=6.81e-05 tok=15.004B dt=10293 wait=166 +21:52:27 step 114240 loss=3.3073 lr=6.80e-05 tok=15.006B dt=10357 wait=166 +21:53:30 step 114260 loss=3.3672 lr=6.80e-05 tok=15.009B dt=10420 wait=166 +21:54:34 step 114280 loss=3.3704 lr=6.80e-05 tok=15.012B dt=10483 wait=167 +21:55:37 step 114300 loss=3.3033 lr=6.80e-05 tok=15.014B dt=10546 wait=167 +21:55:49 pushed ckpt_w95.pt,log_w95.txt +21:59:55 step 114320 loss=3.3379 lr=6.80e-05 tok=15.017B dt=10804 wait=167 +22:00:38 global r=371 load 4.323 -> 4.315 +22:01:41 step 114320 loss=3.3264 lr=6.80e-05 tok=15.021B dt=10910 wait=168 +22:02:44 step 114340 loss=3.3118 lr=6.79e-05 tok=15.023B dt=10973 wait=168 +22:03:47 step 114360 loss=3.3507 lr=6.79e-05 tok=15.026B dt=11036 wait=168 +22:04:50 step 114380 loss=3.2860 lr=6.79e-05 tok=15.029B dt=11100 wait=169 +22:05:53 step 114400 loss=3.3234 lr=6.79e-05 tok=15.031B dt=11163 wait=169 +22:06:57 step 114420 loss=3.3930 lr=6.79e-05 tok=15.034B dt=11226 wait=170 +22:08:01 step 114440 loss=2.8157 lr=6.78e-05 tok=15.037B dt=11290 wait=170 +22:08:50 pushed ckpt_w95.pt,log_w95.txt +22:12:24 step 114460 loss=3.3802 lr=6.78e-05 tok=15.039B dt=11553 wait=170 +22:12:29 global r=372 load 4.316 -> 4.314 +22:13:01 step 114460 loss=3.3879 lr=6.78e-05 tok=15.041B dt=11591 wait=170 +22:14:04 step 114480 loss=3.2992 lr=6.78e-05 tok=15.043B dt=11654 wait=171 +22:15:08 step 114500 loss=3.3444 lr=6.78e-05 tok=15.046B dt=11718 wait=171 +22:16:12 step 114520 loss=3.4115 lr=6.78e-05 tok=15.048B dt=11781 wait=171 +22:17:16 step 114540 loss=3.3185 lr=6.77e-05 tok=15.051B dt=11845 wait=172 +22:18:19 step 114560 loss=3.3490 lr=6.77e-05 tok=15.054B dt=11908 wait=172 +22:19:22 step 114580 loss=3.3131 lr=6.77e-05 tok=15.056B dt=11971 wait=172 +22:20:25 step 114600 loss=3.3368 lr=6.77e-05 tok=15.059B dt=12034 wait=172 +22:20:37 pushed ckpt_w95.pt,log_w95.txt +22:21:54 waited 76s for global r=373 +22:22:02 global r=373 load 4.315 -> 4.314 +22:23:07 step 114620 loss=3.2881 lr=6.77e-05 tok=15.061B dt=12196 wait=173 +22:24:11 step 114640 loss=3.3668 lr=6.76e-05 tok=15.064B dt=12260 wait=174 +22:25:14 step 114660 loss=3.3432 lr=6.76e-05 tok=15.067B dt=12324 wait=175 +22:26:17 step 114680 loss=3.3348 lr=6.76e-05 tok=15.069B dt=12386 wait=175 +22:27:20 step 114700 loss=3.3190 lr=6.76e-05 tok=15.072B dt=12450 wait=175 +22:28:23 step 114720 loss=3.4120 lr=6.75e-05 tok=15.075B dt=12513 wait=176 +22:29:27 step 114740 loss=3.2754 lr=6.75e-05 tok=15.077B dt=12576 wait=176 +22:30:11 pushed ckpt_w95.pt,log_w95.txt +22:31:27 waited 77s for global r=374 +22:31:36 global r=374 load 4.355 -> 4.314 +22:32:09 step 114760 loss=3.4365 lr=6.75e-05 tok=15.080B dt=12738 wait=176 +22:33:12 step 114780 loss=3.2915 lr=6.75e-05 tok=15.082B dt=12802 wait=177 +22:34:15 step 114800 loss=3.2988 lr=6.75e-05 tok=15.085B dt=12865 wait=177 +22:35:19 step 114820 loss=3.3749 lr=6.74e-05 tok=15.088B dt=12928 wait=177 +22:36:22 step 114840 loss=3.2558 lr=6.74e-05 tok=15.090B dt=12991 wait=178 +22:37:25 step 114860 loss=3.3871 lr=6.74e-05 tok=15.093B dt=13054 wait=178 +22:38:28 step 114880 loss=3.3155 lr=6.74e-05 tok=15.096B dt=13117 wait=178 +22:39:31 step 114900 loss=3.3709 lr=6.74e-05 tok=15.098B dt=13180 wait=179 +22:39:45 pushed ckpt_w95.pt,log_w95.txt +22:41:01 waited 76s for global r=375 +22:41:08 global r=375 load 4.320 -> 4.311 +22:42:12 step 114920 loss=3.2942 lr=6.73e-05 tok=15.101B dt=13342 wait=179 +22:43:15 step 114940 loss=3.3951 lr=6.73e-05 tok=15.103B dt=13405 wait=179 +22:44:19 step 114960 loss=3.3387 lr=6.73e-05 tok=15.106B dt=13468 wait=180 +22:45:22 step 114980 loss=3.3004 lr=6.73e-05 tok=15.109B dt=13531 wait=180 +22:46:25 step 115000 loss=3.3046 lr=6.73e-05 tok=15.111B dt=13595 wait=180 +22:47:29 step 115020 loss=3.3987 lr=6.72e-05 tok=15.114B dt=13659 wait=181 +22:48:33 step 115040 loss=3.2589 lr=6.72e-05 tok=15.117B dt=13722 wait=181 +22:49:27 pushed ckpt_w95.pt,log_w95.txt +22:50:27 waited 61s for global r=376 +22:50:37 global r=376 load 4.289 -> 4.313 +22:51:10 step 115060 loss=3.3843 lr=6.72e-05 tok=15.119B dt=13879 wait=181 +22:52:13 step 115080 loss=3.2686 lr=6.72e-05 tok=15.122B dt=13942 wait=182 +22:53:16 step 115100 loss=3.2476 lr=6.72e-05 tok=15.124B dt=14005 wait=182 +22:54:19 step 115120 loss=3.4201 lr=6.71e-05 tok=15.127B dt=14068 wait=182 +22:55:22 step 115140 loss=3.3078 lr=6.71e-05 tok=15.130B dt=14131 wait=183 +22:56:25 step 115160 loss=3.3637 lr=6.71e-05 tok=15.132B dt=14194 wait=183 +22:57:28 step 115180 loss=3.3113 lr=6.71e-05 tok=15.135B dt=14257 wait=183 +22:58:31 step 115200 loss=3.3545 lr=6.71e-05 tok=15.138B dt=14321 wait=184 +22:58:44 pushed ckpt_w95.pt,log_w95.txt +22:59:44 waited 61s for global r=377 +22:59:51 global r=377 load 4.306 -> 4.296 +23:00:56 step 115220 loss=3.2879 lr=6.70e-05 tok=15.140B dt=14466 wait=184 +23:02:00 step 115240 loss=3.2698 lr=6.70e-05 tok=15.143B dt=14529 wait=184 +23:03:03 step 115260 loss=3.3207 lr=6.70e-05 tok=15.145B dt=14592 wait=185 +23:04:06 step 115280 loss=3.2862 lr=6.70e-05 tok=15.148B dt=14656 wait=185 +23:05:10 step 115300 loss=3.3265 lr=6.69e-05 tok=15.151B dt=14719 wait=185 +23:06:13 step 115320 loss=3.4276 lr=6.69e-05 tok=15.153B dt=14783 wait=186 +23:07:16 step 115340 loss=3.3039 lr=6.69e-05 tok=15.156B dt=14846 wait=186 +23:08:01 pushed ckpt_w95.pt,log_w95.txt +23:09:21 waited 80s for global r=378 +23:09:31 global r=378 load 4.323 -> 4.308 +23:10:04 step 115360 loss=3.4138 lr=6.69e-05 tok=15.158B dt=15013 wait=186 +23:11:07 step 115380 loss=3.2154 lr=6.69e-05 tok=15.161B dt=15076 wait=187 +23:12:11 step 115400 loss=3.2526 lr=6.68e-05 tok=15.164B dt=15140 wait=187 +23:13:14 step 115420 loss=3.2709 lr=6.68e-05 tok=15.166B dt=15204 wait=187 +23:14:17 step 115440 loss=3.2814 lr=6.68e-05 tok=15.169B dt=15267 wait=188 +23:15:20 step 115460 loss=3.3684 lr=6.68e-05 tok=15.172B dt=15330 wait=188 +23:16:23 step 115480 loss=3.3625 lr=6.68e-05 tok=15.174B dt=15393 wait=188 +23:17:27 step 115500 loss=3.3397 lr=6.67e-05 tok=15.177B dt=15456 wait=189 +23:17:42 pushed ckpt_w95.pt,log_w95.txt +23:18:44 waited 63s for global r=379 +23:18:52 global r=379 load 4.336 -> 4.306 +23:19:57 step 115520 loss=3.3192 lr=6.67e-05 tok=15.179B dt=15607 wait=189 +23:21:01 step 115540 loss=3.3772 lr=6.67e-05 tok=15.182B dt=15670 wait=190 +23:22:04 step 115560 loss=3.3327 lr=6.67e-05 tok=15.185B dt=15733 wait=190 +23:23:07 step 115580 loss=3.2845 lr=6.67e-05 tok=15.187B dt=15796 wait=190 +23:24:10 step 115600 loss=3.2529 lr=6.66e-05 tok=15.190B dt=15859 wait=191 +23:25:13 step 115620 loss=3.4133 lr=6.66e-05 tok=15.193B dt=15923 wait=191 +23:26:16 step 115640 loss=3.2793 lr=6.66e-05 tok=15.195B dt=15985 wait=191 +23:27:00 pushed ckpt_w95.pt,log_w95.txt +23:28:15 waited 76s for global r=380 +23:28:26 global r=380 load 4.297 -> 4.298 +23:28:59 step 115660 loss=3.4196 lr=6.66e-05 tok=15.198B dt=16148 wait=192 +23:30:02 step 115680 loss=3.2480 lr=6.66e-05 tok=15.200B dt=16212 wait=192 +23:31:06 step 115700 loss=3.3121 lr=6.65e-05 tok=15.203B dt=16275 wait=192 +23:32:09 step 115720 loss=3.3282 lr=6.65e-05 tok=15.206B dt=16338 wait=193 +23:33:12 step 115740 loss=3.2902 lr=6.65e-05 tok=15.208B dt=16402 wait=193 +23:34:15 step 115760 loss=3.3624 lr=6.65e-05 tok=15.211B dt=16465 wait=193 +23:35:19 step 115780 loss=3.2961 lr=6.65e-05 tok=15.214B dt=16528 wait=194 +23:36:22 step 115800 loss=3.3668 lr=6.64e-05 tok=15.216B dt=16591 wait=194 +23:36:39 pushed ckpt_w95.pt,log_w95.txt +23:37:41 waited 62s for global r=381 +23:37:52 global r=381 load 4.341 -> 4.286 +23:38:57 step 115820 loss=3.3501 lr=6.64e-05 tok=15.219B dt=16746 wait=194 +23:40:01 step 115840 loss=3.3469 lr=6.64e-05 tok=15.221B dt=16810 wait=195 +23:41:04 step 115860 loss=3.2851 lr=6.64e-05 tok=15.224B dt=16873 wait=195 +23:42:07 step 115880 loss=3.2844 lr=6.64e-05 tok=15.227B dt=16936 wait=195 +23:43:10 step 115900 loss=3.3703 lr=6.63e-05 tok=15.229B dt=16999 wait=196 +23:44:13 step 115920 loss=3.4157 lr=6.63e-05 tok=15.232B dt=17063 wait=196 +23:45:17 step 115940 loss=3.2260 lr=6.63e-05 tok=15.234B dt=17126 wait=196 +23:46:05 pushed ckpt_w95.pt,log_w95.txt +23:47:05 waited 61s for global r=382 +23:47:13 global r=382 load 4.308 -> 4.287 +23:47:47 step 115960 loss=3.4092 lr=6.63e-05 tok=15.237B dt=17276 wait=197 +23:48:50 step 115980 loss=3.3131 lr=6.62e-05 tok=15.240B dt=17340 wait=197 +23:49:53 step 116000 loss=3.2837 lr=6.62e-05 tok=15.242B dt=17403 wait=197 +23:50:56 step 116020 loss=3.3850 lr=6.62e-05 tok=15.245B dt=17466 wait=198 +23:52:00 step 116040 loss=3.2808 lr=6.62e-05 tok=15.248B dt=17529 wait=198 +23:53:03 step 116060 loss=3.3571 lr=6.62e-05 tok=15.250B dt=17592 wait=199 +23:54:06 step 116080 loss=3.2978 lr=6.61e-05 tok=15.253B dt=17656 wait=199 +23:55:10 step 116100 loss=3.3090 lr=6.61e-05 tok=15.255B dt=17719 wait=199 +23:55:22 pushed ckpt_w95.pt,log_w95.txt +23:56:39 waited 77s for global r=383 +23:56:47 global r=383 load 4.295 -> 4.287 +23:57:52 step 116120 loss=3.3171 lr=6.61e-05 tok=15.258B dt=17881 wait=200 +23:58:55 step 116140 loss=3.3586 lr=6.61e-05 tok=15.261B dt=17945 wait=200 +23:59:58 step 116160 loss=3.3210 lr=6.61e-05 tok=15.263B dt=18007 wait=200 +00:01:02 step 116180 loss=3.2779 lr=6.60e-05 tok=15.266B dt=18071 wait=201 +00:02:05 step 116200 loss=3.3025 lr=6.60e-05 tok=15.269B dt=18134 wait=201 +00:03:09 step 116220 loss=3.2785 lr=6.60e-05 tok=15.271B dt=18198 wait=201 +00:04:12 step 116240 loss=3.2379 lr=6.60e-05 tok=15.274B dt=18261 wait=202 +00:05:04 pushed ckpt_w95.pt,log_w95.txt +00:06:06 waited 62s for global r=384 +00:06:16 global r=384 load 4.303 -> 4.282 +00:06:49 step 116260 loss=3.4166 lr=6.60e-05 tok=15.276B dt=18419 wait=202 +00:07:53 step 116280 loss=3.3718 lr=6.59e-05 tok=15.279B dt=18482 wait=202 +00:08:56 step 116300 loss=3.3189 lr=6.59e-05 tok=15.282B dt=18545 wait=203 +00:09:59 step 116320 loss=3.3205 lr=6.59e-05 tok=15.284B dt=18608 wait=203 +00:11:02 step 116340 loss=3.2824 lr=6.59e-05 tok=15.287B dt=18672 wait=203 +00:12:06 step 116360 loss=3.4179 lr=6.59e-05 tok=15.290B dt=18735 wait=203 +00:13:09 step 116380 loss=3.2688 lr=6.58e-05 tok=15.292B dt=18799 wait=204 +00:14:13 step 116400 loss=3.3443 lr=6.58e-05 tok=15.295B dt=18862 wait=204 +00:14:28 pushed ckpt_w95.pt,log_w95.txt +00:15:29 waited 61s for global r=385 +00:15:37 global r=385 load 4.293 -> 4.298 +00:16:42 step 116420 loss=3.2758 lr=6.58e-05 tok=15.297B dt=19011 wait=205 +00:17:45 step 116440 loss=3.2983 lr=6.58e-05 tok=15.300B dt=19075 wait=205 +00:18:49 step 116460 loss=3.3417 lr=6.58e-05 tok=15.303B dt=19138 wait=205 +00:19:52 step 116480 loss=3.3087 lr=6.57e-05 tok=15.305B dt=19201 wait=206 +00:20:55 step 116500 loss=3.3277 lr=6.57e-05 tok=15.308B dt=19264 wait=206 +00:21:58 step 116520 loss=3.4016 lr=6.57e-05 tok=15.311B dt=19327 wait=206 +00:23:01 step 116540 loss=3.2708 lr=6.57e-05 tok=15.313B dt=19390 wait=207 +00:23:48 pushed ckpt_w95.pt,log_w95.txt +00:24:49 waited 61s for global r=386 +00:24:57 global r=386 load 4.313 -> 4.284 +00:25:31 step 116560 loss=3.3406 lr=6.57e-05 tok=15.316B dt=19540 wait=207 +00:26:34 step 116580 loss=3.2611 lr=6.56e-05 tok=15.318B dt=19603 wait=207 +00:27:37 step 116600 loss=3.2837 lr=6.56e-05 tok=15.321B dt=19666 wait=208 +00:28:40 step 116620 loss=3.4143 lr=6.56e-05 tok=15.324B dt=19729 wait=208 +00:29:43 step 116640 loss=3.3264 lr=6.56e-05 tok=15.326B dt=19792 wait=208 +00:30:46 step 116660 loss=3.4210 lr=6.56e-05 tok=15.329B dt=19855 wait=209 +00:31:49 step 116680 loss=3.3144 lr=6.55e-05 tok=15.331B dt=19918 wait=209 +00:32:53 step 116700 loss=3.3505 lr=6.55e-05 tok=15.334B dt=19982 wait=209 +00:33:07 pushed ckpt_w95.pt,log_w95.txt +00:34:23 waited 76s for global r=387 +00:34:30 global r=387 load 4.287 -> 4.274 +00:35:34 step 116720 loss=3.2708 lr=6.55e-05 tok=15.337B dt=20144 wait=210 +00:36:38 step 116740 loss=3.3267 lr=6.55e-05 tok=15.339B dt=20207 wait=210 +00:37:41 step 116760 loss=3.3390 lr=6.54e-05 tok=15.342B dt=20271 wait=210 +00:38:44 step 116780 loss=3.3348 lr=6.54e-05 tok=15.345B dt=20334 wait=210 +00:39:48 step 116800 loss=3.3394 lr=6.54e-05 tok=15.347B dt=20397 wait=211 +00:40:51 step 116820 loss=3.3963 lr=6.54e-05 tok=15.350B dt=20460 wait=211 +00:41:54 step 116840 loss=3.3044 lr=6.54e-05 tok=15.352B dt=20523 wait=212 +00:42:42 pushed ckpt_w95.pt,log_w95.txt +00:43:43 waited 61s for global r=388 +00:43:50 global r=388 load 4.271 -> 4.278 +00:44:23 step 116860 loss=3.4328 lr=6.53e-05 tok=15.355B dt=20673 wait=212 +00:45:26 step 116880 loss=3.2740 lr=6.53e-05 tok=15.358B dt=20736 wait=212 +00:46:30 step 116900 loss=3.2787 lr=6.53e-05 tok=15.360B dt=20799 wait=213 +00:47:32 step 116920 loss=3.3692 lr=6.53e-05 tok=15.363B dt=20862 wait=213 +00:48:36 step 116940 loss=3.2551 lr=6.53e-05 tok=15.366B dt=20925 wait=213 +00:49:39 step 116960 loss=3.3667 lr=6.52e-05 tok=15.368B dt=20988 wait=214 +00:50:42 step 116980 loss=3.3450 lr=6.52e-05 tok=15.371B dt=21052 wait=214 +00:51:46 step 117000 loss=3.3221 lr=6.52e-05 tok=15.373B dt=21115 wait=214 diff --git a/log_w96.txt b/log_w96.txt new file mode 100644 index 0000000000000000000000000000000000000000..a625a828912e69640b5deb5926b115041fb54d54 --- /dev/null +++ b/log_w96.txt @@ -0,0 +1,203 @@ +launch +22:02:30 name=w96 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 43/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:02:36 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:02:36 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +22:02:46 params 1.08B +22:02:47 resumed step=114300 seen=15.02B round=371 +22:02:47 stream salt=114300 +22:04:35 blocks compiled +22:04:35 start: accum=4, 131072 tokens per optimiser step +22:06:42 step 114301 loss=3.0441 lr=6.80e-05 tok=15.021B dt=128 wait=87 +22:06:46 step 114302 loss=3.0384 lr=6.80e-05 tok=15.021B dt=131 wait=87 +22:07:50 step 114320 loss=3.3737 lr=6.80e-05 tok=15.023B dt=195 wait=87 +22:08:56 step 114340 loss=3.3516 lr=6.79e-05 tok=15.026B dt=261 wait=88 +22:09:59 step 114360 loss=3.4087 lr=6.79e-05 tok=15.029B dt=325 wait=88 +22:11:03 step 114380 loss=3.3371 lr=6.79e-05 tok=15.031B dt=388 wait=88 +22:12:06 step 114400 loss=3.4739 lr=6.79e-05 tok=15.034B dt=451 wait=89 +22:12:57 global r=372 load 4.351 -> 4.314 +22:13:29 step 114460 loss=3.4288 lr=6.78e-05 tok=15.037B dt=534 wait=89 +22:14:32 step 114480 loss=3.3079 lr=6.78e-05 tok=15.039B dt=597 wait=89 +22:15:47 step 114500 loss=3.1748 lr=6.78e-05 tok=15.042B dt=672 wait=90 +22:16:50 step 114520 loss=3.3851 lr=6.78e-05 tok=15.044B dt=736 wait=90 +22:17:53 step 114540 loss=3.2790 lr=6.77e-05 tok=15.047B dt=799 wait=90 +22:18:56 step 114560 loss=3.4074 lr=6.77e-05 tok=15.050B dt=862 wait=90 +22:19:59 step 114580 loss=3.3406 lr=6.77e-05 tok=15.052B dt=924 wait=91 +22:21:02 step 114600 loss=3.2771 lr=6.77e-05 tok=15.055B dt=987 wait=91 +22:21:15 pushed ckpt_w96.pt,log_w96.txt +22:22:01 waited 45s for global r=373 +22:22:08 global r=373 load 4.295 -> 4.314 +22:23:13 step 114620 loss=3.2661 lr=6.77e-05 tok=15.058B dt=1118 wait=91 +22:24:24 step 114640 loss=3.3461 lr=6.76e-05 tok=15.060B dt=1189 wait=92 +22:25:38 step 114660 loss=3.3087 lr=6.76e-05 tok=15.063B dt=1264 wait=92 +22:26:41 step 114680 loss=3.2932 lr=6.76e-05 tok=15.065B dt=1327 wait=92 +22:27:45 step 114700 loss=3.3815 lr=6.76e-05 tok=15.068B dt=1390 wait=93 +22:28:48 step 114720 loss=3.3682 lr=6.75e-05 tok=15.071B dt=1453 wait=93 +22:29:51 step 114740 loss=3.2571 lr=6.75e-05 tok=15.073B dt=1516 wait=93 +22:30:34 pushed ckpt_w96.pt,log_w96.txt +22:31:20 waited 46s for global r=374 +22:31:25 global r=374 load 4.328 -> 4.314 +22:31:58 step 114760 loss=3.3903 lr=6.75e-05 tok=15.076B dt=1643 wait=94 +22:33:01 step 114780 loss=3.3156 lr=6.75e-05 tok=15.079B dt=1707 wait=94 +22:34:19 step 114800 loss=3.3196 lr=6.75e-05 tok=15.081B dt=1784 wait=94 +22:35:27 step 114820 loss=3.4493 lr=6.74e-05 tok=15.084B dt=1852 wait=95 +22:36:30 step 114840 loss=3.3248 lr=6.74e-05 tok=15.086B dt=1915 wait=95 +22:37:33 step 114860 loss=3.4586 lr=6.74e-05 tok=15.089B dt=1978 wait=95 +22:38:36 step 114880 loss=3.3478 lr=6.74e-05 tok=15.092B dt=2042 wait=96 +22:39:39 step 114900 loss=3.3777 lr=6.74e-05 tok=15.094B dt=2105 wait=96 +22:39:52 pushed ckpt_w96.pt,log_w96.txt +22:40:52 waited 60s for global r=375 +22:40:58 global r=375 load 4.331 -> 4.311 +22:42:02 step 114920 loss=3.2971 lr=6.73e-05 tok=15.097B dt=2248 wait=96 +22:43:12 step 114940 loss=3.3572 lr=6.73e-05 tok=15.099B dt=2318 wait=97 +22:44:29 step 114960 loss=3.3488 lr=6.73e-05 tok=15.102B dt=2394 wait=97 +22:45:32 step 114980 loss=3.1702 lr=6.73e-05 tok=15.105B dt=2458 wait=97 +22:46:35 step 115000 loss=3.3938 lr=6.73e-05 tok=15.107B dt=2521 wait=98 +22:47:39 step 115020 loss=3.4578 lr=6.72e-05 tok=15.110B dt=2584 wait=98 +22:48:42 step 115040 loss=3.3026 lr=6.72e-05 tok=15.113B dt=2647 wait=98 +22:49:27 pushed ckpt_w96.pt,log_w96.txt +22:50:27 waited 60s for global r=376 +22:50:32 global r=376 load 4.316 -> 4.313 +22:51:05 step 115060 loss=3.4650 lr=6.72e-05 tok=15.115B dt=2790 wait=98 +22:52:08 step 115080 loss=3.3070 lr=6.72e-05 tok=15.118B dt=2853 wait=99 +22:53:27 step 115100 loss=3.2957 lr=6.72e-05 tok=15.120B dt=2932 wait=99 +22:54:34 step 115120 loss=3.4281 lr=6.71e-05 tok=15.123B dt=2999 wait=100 +22:55:38 step 115140 loss=3.2889 lr=6.71e-05 tok=15.126B dt=3063 wait=100 +22:56:46 step 115160 loss=3.3533 lr=6.71e-05 tok=15.128B dt=3131 wait=100 +22:57:49 step 115180 loss=3.3502 lr=6.71e-05 tok=15.131B dt=3194 wait=100 +22:58:52 step 115200 loss=3.3706 lr=6.71e-05 tok=15.134B dt=3257 wait=101 +22:59:04 pushed ckpt_w96.pt,log_w96.txt +22:59:50 waited 45s for global r=377 +22:59:58 global r=377 load 4.299 -> 4.296 +23:01:02 step 115220 loss=3.3212 lr=6.70e-05 tok=15.136B dt=3387 wait=101 +23:02:11 step 115240 loss=3.3241 lr=6.70e-05 tok=15.139B dt=3457 wait=101 +23:03:28 step 115260 loss=3.2840 lr=6.70e-05 tok=15.141B dt=3533 wait=102 +23:04:31 step 115280 loss=3.2906 lr=6.70e-05 tok=15.144B dt=3596 wait=102 +23:05:34 step 115300 loss=3.3504 lr=6.69e-05 tok=15.147B dt=3659 wait=102 +23:06:37 step 115320 loss=3.4254 lr=6.69e-05 tok=15.149B dt=3722 wait=103 +23:07:41 step 115340 loss=3.2553 lr=6.69e-05 tok=15.152B dt=3786 wait=103 +23:08:24 pushed ckpt_w96.pt,log_w96.txt +23:09:10 waited 45s for global r=378 +23:09:18 global r=378 load 4.339 -> 4.308 +23:09:59 step 115360 loss=3.4239 lr=6.69e-05 tok=15.155B dt=3924 wait=103 +23:11:14 step 115380 loss=3.2674 lr=6.69e-05 tok=15.157B dt=3999 wait=104 +23:12:17 step 115400 loss=3.3116 lr=6.68e-05 tok=15.160B dt=4062 wait=104 +23:13:20 step 115420 loss=3.3702 lr=6.68e-05 tok=15.162B dt=4125 wait=104 +23:14:23 step 115440 loss=3.2885 lr=6.68e-05 tok=15.165B dt=4188 wait=105 +23:15:26 step 115460 loss=3.4099 lr=6.68e-05 tok=15.168B dt=4252 wait=105 +23:16:29 step 115480 loss=3.3271 lr=6.68e-05 tok=15.170B dt=4315 wait=105 +23:17:33 step 115500 loss=3.3316 lr=6.67e-05 tok=15.173B dt=4378 wait=106 +23:17:48 pushed ckpt_w96.pt,log_w96.txt +23:18:48 waited 61s for global r=379 +23:18:57 global r=379 load 4.317 -> 4.306 +23:20:21 step 115520 loss=3.3385 lr=6.67e-05 tok=15.176B dt=4547 wait=106 +23:21:25 step 115540 loss=3.3297 lr=6.67e-05 tok=15.178B dt=4610 wait=106 +23:22:27 step 115560 loss=3.3083 lr=6.67e-05 tok=15.181B dt=4673 wait=107 +23:23:31 step 115580 loss=3.2844 lr=6.67e-05 tok=15.183B dt=4736 wait=107 +23:24:33 step 115600 loss=3.3209 lr=6.66e-05 tok=15.186B dt=4799 wait=107 +23:25:36 step 115620 loss=3.4250 lr=6.66e-05 tok=15.189B dt=4862 wait=108 +23:26:39 step 115640 loss=3.2764 lr=6.66e-05 tok=15.191B dt=4925 wait=108 +23:27:24 pushed ckpt_w96.pt,log_w96.txt +23:28:25 waited 61s for global r=380 +23:28:32 global r=380 load 4.314 -> 4.298 +23:29:13 step 115660 loss=3.3636 lr=6.66e-05 tok=15.194B dt=5078 wait=108 +23:30:28 step 115680 loss=3.2507 lr=6.66e-05 tok=15.196B dt=5153 wait=109 +23:31:31 step 115700 loss=3.3022 lr=6.65e-05 tok=15.199B dt=5216 wait=109 +23:32:34 step 115720 loss=3.3874 lr=6.65e-05 tok=15.202B dt=5280 wait=109 +23:33:37 step 115740 loss=3.3146 lr=6.65e-05 tok=15.204B dt=5343 wait=110 +23:34:40 step 115760 loss=3.3844 lr=6.65e-05 tok=15.207B dt=5406 wait=110 +23:35:43 step 115780 loss=3.3322 lr=6.65e-05 tok=15.210B dt=5468 wait=110 +23:36:46 step 115800 loss=3.3702 lr=6.64e-05 tok=15.212B dt=5531 wait=111 +23:37:01 pushed ckpt_w96.pt,log_w96.txt +23:37:46 waited 45s for global r=381 +23:37:53 global r=381 load 4.314 -> 4.286 +23:39:13 step 115820 loss=3.2726 lr=6.64e-05 tok=15.215B dt=5678 wait=111 +23:40:19 step 115840 loss=3.3614 lr=6.64e-05 tok=15.217B dt=5745 wait=111 +23:41:23 step 115860 loss=3.3367 lr=6.64e-05 tok=15.220B dt=5808 wait=112 +23:42:27 step 115880 loss=3.3056 lr=6.64e-05 tok=15.223B dt=5872 wait=112 +23:43:30 step 115900 loss=3.3422 lr=6.63e-05 tok=15.225B dt=5935 wait=112 +23:44:33 step 115920 loss=3.3479 lr=6.63e-05 tok=15.228B dt=5998 wait=113 +23:45:38 step 115940 loss=2.8238 lr=6.63e-05 tok=15.231B dt=6064 wait=113 +23:46:22 pushed ckpt_w96.pt,log_w96.txt +23:47:08 waited 45s for global r=382 +23:47:15 global r=382 load 4.293 -> 4.287 +23:47:53 step 115960 loss=3.3295 lr=6.63e-05 tok=15.233B dt=6199 wait=113 +23:49:10 step 115980 loss=3.3187 lr=6.62e-05 tok=15.236B dt=6276 wait=114 +23:50:13 step 116000 loss=3.2589 lr=6.62e-05 tok=15.238B dt=6339 wait=114 +23:51:17 step 116020 loss=3.3954 lr=6.62e-05 tok=15.241B dt=6402 wait=114 +23:52:20 step 116040 loss=3.3051 lr=6.62e-05 tok=15.244B dt=6465 wait=114 +23:53:23 step 116060 loss=3.3907 lr=6.62e-05 tok=15.246B dt=6528 wait=115 +23:54:26 step 116080 loss=3.3451 lr=6.61e-05 tok=15.249B dt=6591 wait=115 +23:55:30 step 116100 loss=3.3101 lr=6.61e-05 tok=15.252B dt=6655 wait=115 +23:55:44 pushed ckpt_w96.pt,log_w96.txt +23:56:45 waited 61s for global r=383 +23:56:50 global r=383 load 4.296 -> 4.287 +23:58:03 step 116120 loss=3.3109 lr=6.61e-05 tok=15.254B dt=6808 wait=116 +23:59:18 step 116140 loss=3.3743 lr=6.61e-05 tok=15.257B dt=6883 wait=116 +00:00:22 step 116160 loss=3.3070 lr=6.61e-05 tok=15.259B dt=6947 wait=116 +00:01:25 step 116180 loss=3.2927 lr=6.60e-05 tok=15.262B dt=7010 wait=117 +00:02:28 step 116200 loss=3.3413 lr=6.60e-05 tok=15.265B dt=7073 wait=117 +00:03:31 step 116220 loss=3.4354 lr=6.60e-05 tok=15.267B dt=7137 wait=117 +00:04:34 step 116240 loss=3.3184 lr=6.60e-05 tok=15.270B dt=7200 wait=118 +00:05:19 pushed ckpt_w96.pt,log_w96.txt +00:06:04 waited 45s for global r=384 +00:06:10 global r=384 load 4.280 -> 4.282 +00:06:43 step 116260 loss=3.4377 lr=6.60e-05 tok=15.273B dt=7328 wait=118 +00:08:04 step 116280 loss=3.2760 lr=6.59e-05 tok=15.275B dt=7409 wait=118 +00:09:10 step 116300 loss=3.3325 lr=6.59e-05 tok=15.278B dt=7475 wait=119 +00:10:13 step 116320 loss=3.3959 lr=6.59e-05 tok=15.280B dt=7538 wait=119 +00:11:17 step 116340 loss=3.2993 lr=6.59e-05 tok=15.283B dt=7602 wait=119 +00:12:20 step 116360 loss=3.3506 lr=6.59e-05 tok=15.286B dt=7665 wait=120 +00:13:23 step 116380 loss=3.3252 lr=6.58e-05 tok=15.288B dt=7728 wait=120 +00:14:26 step 116400 loss=3.3587 lr=6.58e-05 tok=15.291B dt=7791 wait=120 +00:14:37 pushed ckpt_w96.pt,log_w96.txt +00:15:23 waited 45s for global r=385 +00:15:29 global r=385 load 4.307 -> 4.298 +00:16:38 step 116420 loss=3.2019 lr=6.58e-05 tok=15.293B dt=7923 wait=121 +00:17:57 step 116440 loss=3.2978 lr=6.58e-05 tok=15.296B dt=8002 wait=121 +00:19:00 step 116460 loss=3.3622 lr=6.58e-05 tok=15.299B dt=8065 wait=121 +00:20:03 step 116480 loss=3.2986 lr=6.57e-05 tok=15.301B dt=8128 wait=122 +00:21:06 step 116500 loss=3.2295 lr=6.57e-05 tok=15.304B dt=8192 wait=122 +00:22:09 step 116520 loss=3.4184 lr=6.57e-05 tok=15.307B dt=8254 wait=122 +00:23:12 step 116540 loss=3.2271 lr=6.57e-05 tok=15.309B dt=8317 wait=123 +00:23:56 pushed ckpt_w96.pt,log_w96.txt +00:24:56 waited 60s for global r=386 +00:25:01 global r=386 load 4.319 -> 4.284 +00:25:34 step 116560 loss=3.3292 lr=6.57e-05 tok=15.312B dt=8460 wait=123 +00:26:45 step 116580 loss=3.2729 lr=6.56e-05 tok=15.314B dt=8530 wait=123 +00:28:00 step 116600 loss=3.2556 lr=6.56e-05 tok=15.317B dt=8605 wait=124 +00:29:03 step 116620 loss=3.3887 lr=6.56e-05 tok=15.320B dt=8668 wait=124 +00:30:06 step 116640 loss=3.2549 lr=6.56e-05 tok=15.322B dt=8731 wait=124 +00:31:10 step 116660 loss=3.3853 lr=6.56e-05 tok=15.325B dt=8795 wait=124 +00:32:12 step 116680 loss=3.3663 lr=6.55e-05 tok=15.328B dt=8858 wait=125 +00:33:15 step 116700 loss=3.3276 lr=6.55e-05 tok=15.330B dt=8921 wait=125 +00:33:27 pushed ckpt_w96.pt,log_w96.txt +00:34:13 waited 46s for global r=387 +00:34:20 global r=387 load 4.292 -> 4.274 +00:35:25 step 116720 loss=3.2204 lr=6.55e-05 tok=15.333B dt=9050 wait=125 +00:36:43 step 116740 loss=3.3334 lr=6.55e-05 tok=15.335B dt=9129 wait=126 +00:37:50 step 116760 loss=3.3354 lr=6.54e-05 tok=15.338B dt=9195 wait=126 +00:38:53 step 116780 loss=3.2488 lr=6.54e-05 tok=15.341B dt=9259 wait=126 +00:39:57 step 116800 loss=3.3234 lr=6.54e-05 tok=15.343B dt=9322 wait=127 +00:41:00 step 116820 loss=3.4164 lr=6.54e-05 tok=15.346B dt=9385 wait=127 +00:42:03 step 116840 loss=3.2629 lr=6.54e-05 tok=15.349B dt=9448 wait=127 +00:42:48 pushed ckpt_w96.pt,log_w96.txt +00:43:48 waited 61s for global r=388 +00:43:55 global r=388 load 4.280 -> 4.278 +00:44:28 step 116860 loss=3.4125 lr=6.53e-05 tok=15.351B dt=9593 wait=128 +00:45:39 step 116880 loss=3.3234 lr=6.53e-05 tok=15.354B dt=9664 wait=128 +00:46:54 step 116900 loss=3.3089 lr=6.53e-05 tok=15.356B dt=9739 wait=128 +00:47:57 step 116920 loss=3.4228 lr=6.53e-05 tok=15.359B dt=9803 wait=129 +00:49:01 step 116940 loss=3.3039 lr=6.53e-05 tok=15.362B dt=9866 wait=129 +00:50:04 step 116960 loss=3.3819 lr=6.52e-05 tok=15.364B dt=9929 wait=129 +00:51:07 step 116980 loss=3.3843 lr=6.52e-05 tok=15.367B dt=9992 wait=130 +00:52:11 step 117000 loss=3.3784 lr=6.52e-05 tok=15.370B dt=10056 wait=130 +00:52:22 pushed ckpt_w96.pt,log_w96.txt +00:53:08 waited 45s for global r=389 +00:53:17 global r=389 load 4.282 -> 4.281 +00:54:21 step 117020 loss=3.3365 lr=6.52e-05 tok=15.372B dt=10186 wait=130 +00:55:38 step 117040 loss=3.2999 lr=6.52e-05 tok=15.375B dt=10263 wait=131 +00:56:47 step 117060 loss=3.3691 lr=6.51e-05 tok=15.377B dt=10333 wait=131 +00:57:51 step 117080 loss=3.3266 lr=6.51e-05 tok=15.380B dt=10396 wait=131 +00:58:54 step 117100 loss=3.2624 lr=6.51e-05 tok=15.383B dt=10459 wait=132 +00:59:57 step 117120 loss=3.4232 lr=6.51e-05 tok=15.385B dt=10523 wait=132 +01:01:00 step 117140 loss=3.3097 lr=6.51e-05 tok=15.388B dt=10586 wait=132 diff --git a/log_w97.txt b/log_w97.txt new file mode 100644 index 0000000000000000000000000000000000000000..c71f94b65a45dd9ac80c14ec9c3a7af94eb3f109 --- /dev/null +++ b/log_w97.txt @@ -0,0 +1,203 @@ +launch +22:02:28 name=w97 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 44/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:02:34 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:02:34 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +22:02:44 params 1.08B +22:02:45 resumed step=114300 seen=15.02B round=371 +22:02:45 stream salt=114300 +22:04:28 blocks compiled +22:04:28 start: accum=4, 131072 tokens per optimiser step +22:06:45 step 114301 loss=2.9235 lr=6.80e-05 tok=15.021B dt=136 wait=91 +22:06:48 step 114302 loss=3.0468 lr=6.80e-05 tok=15.021B dt=140 wait=91 +22:07:53 step 114320 loss=3.3338 lr=6.80e-05 tok=15.023B dt=205 wait=91 +22:09:00 step 114340 loss=3.4308 lr=6.79e-05 tok=15.026B dt=272 wait=92 +22:10:04 step 114360 loss=3.4040 lr=6.79e-05 tok=15.029B dt=335 wait=92 +22:11:08 step 114380 loss=3.3674 lr=6.79e-05 tok=15.031B dt=400 wait=93 +22:12:12 step 114400 loss=3.4906 lr=6.79e-05 tok=15.034B dt=464 wait=93 +22:12:23 global r=372 load 4.289 -> 4.314 +22:12:54 step 114460 loss=3.5906 lr=6.78e-05 tok=15.035B dt=506 wait=93 +22:13:58 step 114480 loss=3.4180 lr=6.78e-05 tok=15.038B dt=570 wait=93 +22:15:02 step 114500 loss=3.3415 lr=6.78e-05 tok=15.041B dt=634 wait=94 +22:16:06 step 114520 loss=3.4625 lr=6.78e-05 tok=15.043B dt=698 wait=94 +22:17:10 step 114540 loss=3.3387 lr=6.77e-05 tok=15.046B dt=761 wait=94 +22:18:14 step 114560 loss=3.3747 lr=6.77e-05 tok=15.048B dt=826 wait=95 +22:19:18 step 114580 loss=3.2963 lr=6.77e-05 tok=15.051B dt=889 wait=95 +22:20:22 step 114600 loss=3.3944 lr=6.77e-05 tok=15.054B dt=953 wait=95 +22:20:35 pushed ckpt_w97.pt,log_w97.txt +22:21:51 waited 76s for global r=373 +22:21:57 global r=373 load 4.329 -> 4.314 +22:23:03 step 114620 loss=3.3433 lr=6.77e-05 tok=15.056B dt=1114 wait=96 +22:24:06 step 114640 loss=3.3769 lr=6.76e-05 tok=15.059B dt=1178 wait=96 +22:25:10 step 114660 loss=3.3537 lr=6.76e-05 tok=15.061B dt=1241 wait=97 +22:26:14 step 114680 loss=3.3113 lr=6.76e-05 tok=15.064B dt=1305 wait=97 +22:27:17 step 114700 loss=3.3736 lr=6.76e-05 tok=15.067B dt=1369 wait=97 +22:28:21 step 114720 loss=3.3907 lr=6.75e-05 tok=15.069B dt=1433 wait=97 +22:29:25 step 114740 loss=3.3194 lr=6.75e-05 tok=15.072B dt=1497 wait=98 +22:30:14 pushed ckpt_w97.pt,log_w97.txt +22:31:30 waited 76s for global r=374 +22:31:36 global r=374 load 4.339 -> 4.314 +22:32:09 step 114760 loss=3.4314 lr=6.75e-05 tok=15.075B dt=1661 wait=98 +22:33:13 step 114780 loss=3.3024 lr=6.75e-05 tok=15.077B dt=1724 wait=99 +22:34:16 step 114800 loss=3.3438 lr=6.75e-05 tok=15.080B dt=1788 wait=99 +22:35:20 step 114820 loss=3.3460 lr=6.74e-05 tok=15.082B dt=1852 wait=99 +22:36:23 step 114840 loss=3.3249 lr=6.74e-05 tok=15.085B dt=1915 wait=100 +22:37:28 step 114860 loss=3.4191 lr=6.74e-05 tok=15.088B dt=1979 wait=100 +22:38:32 step 114880 loss=3.3781 lr=6.74e-05 tok=15.090B dt=2043 wait=100 +22:39:35 step 114900 loss=3.4222 lr=6.74e-05 tok=15.093B dt=2107 wait=101 +22:39:47 pushed ckpt_w97.pt,log_w97.txt +22:40:48 waited 61s for global r=375 +22:40:54 global r=375 load 4.314 -> 4.311 +22:41:59 step 114920 loss=3.3012 lr=6.73e-05 tok=15.096B dt=2250 wait=101 +22:43:03 step 114940 loss=3.3729 lr=6.73e-05 tok=15.098B dt=2314 wait=101 +22:44:07 step 114960 loss=3.3465 lr=6.73e-05 tok=15.101B dt=2378 wait=102 +22:45:11 step 114980 loss=3.3157 lr=6.73e-05 tok=15.103B dt=2442 wait=102 +22:46:15 step 115000 loss=3.3300 lr=6.73e-05 tok=15.106B dt=2506 wait=102 +22:47:18 step 115020 loss=3.4377 lr=6.72e-05 tok=15.109B dt=2570 wait=103 +22:48:22 step 115040 loss=3.3437 lr=6.72e-05 tok=15.111B dt=2634 wait=103 +22:49:08 pushed ckpt_w97.pt,log_w97.txt +22:50:24 waited 76s for global r=376 +22:50:30 global r=376 load 4.320 -> 4.313 +22:51:03 step 115060 loss=3.3449 lr=6.72e-05 tok=15.114B dt=2795 wait=103 +22:52:07 step 115080 loss=3.2892 lr=6.72e-05 tok=15.117B dt=2859 wait=104 +22:53:11 step 115100 loss=3.3291 lr=6.72e-05 tok=15.119B dt=2922 wait=104 +22:54:14 step 115120 loss=3.3883 lr=6.71e-05 tok=15.122B dt=2986 wait=104 +22:55:18 step 115140 loss=3.3195 lr=6.71e-05 tok=15.124B dt=3050 wait=105 +22:56:22 step 115160 loss=3.4158 lr=6.71e-05 tok=15.127B dt=3113 wait=105 +22:57:25 step 115180 loss=3.3724 lr=6.71e-05 tok=15.130B dt=3177 wait=105 +22:58:29 step 115200 loss=3.3954 lr=6.71e-05 tok=15.132B dt=3241 wait=106 +22:58:44 pushed ckpt_w97.pt,log_w97.txt +22:59:44 waited 61s for global r=377 +22:59:52 global r=377 load 4.323 -> 4.296 +23:00:56 step 115220 loss=3.3382 lr=6.70e-05 tok=15.135B dt=3388 wait=106 +23:02:00 step 115240 loss=3.3324 lr=6.70e-05 tok=15.138B dt=3451 wait=106 +23:03:04 step 115260 loss=3.3121 lr=6.70e-05 tok=15.140B dt=3515 wait=107 +23:04:07 step 115280 loss=3.2838 lr=6.70e-05 tok=15.143B dt=3579 wait=107 +23:05:11 step 115300 loss=3.3106 lr=6.69e-05 tok=15.145B dt=3643 wait=107 +23:06:15 step 115320 loss=3.3986 lr=6.69e-05 tok=15.148B dt=3706 wait=108 +23:07:19 step 115340 loss=3.2565 lr=6.69e-05 tok=15.151B dt=3771 wait=108 +23:08:05 pushed ckpt_w97.pt,log_w97.txt +23:09:21 waited 76s for global r=378 +23:09:28 global r=378 load 4.315 -> 4.308 +23:10:01 step 115360 loss=3.3647 lr=6.69e-05 tok=15.153B dt=3933 wait=108 +23:11:05 step 115380 loss=3.3047 lr=6.69e-05 tok=15.156B dt=3996 wait=109 +23:12:08 step 115400 loss=3.3284 lr=6.68e-05 tok=15.158B dt=4060 wait=109 +23:13:12 step 115420 loss=3.4296 lr=6.68e-05 tok=15.161B dt=4124 wait=109 +23:14:16 step 115440 loss=3.3039 lr=6.68e-05 tok=15.164B dt=4187 wait=110 +23:15:20 step 115460 loss=3.4170 lr=6.68e-05 tok=15.166B dt=4251 wait=110 +23:16:23 step 115480 loss=3.2472 lr=6.68e-05 tok=15.169B dt=4315 wait=110 +23:17:27 step 115500 loss=3.3936 lr=6.67e-05 tok=15.172B dt=4379 wait=110 +23:17:41 pushed ckpt_w97.pt,log_w97.txt +23:18:42 waited 61s for global r=379 +23:18:50 global r=379 load 4.304 -> 4.306 +23:19:54 step 115520 loss=3.3757 lr=6.67e-05 tok=15.174B dt=4526 wait=111 +23:20:58 step 115540 loss=3.3171 lr=6.67e-05 tok=15.177B dt=4590 wait=111 +23:22:02 step 115560 loss=3.3531 lr=6.67e-05 tok=15.179B dt=4653 wait=111 +23:23:05 step 115580 loss=3.3309 lr=6.67e-05 tok=15.182B dt=4717 wait=112 +23:24:09 step 115600 loss=3.3335 lr=6.66e-05 tok=15.185B dt=4781 wait=112 +23:25:14 step 115620 loss=3.4572 lr=6.66e-05 tok=15.187B dt=4845 wait=112 +23:26:18 step 115640 loss=3.1602 lr=6.66e-05 tok=15.190B dt=4910 wait=113 +23:27:03 pushed ckpt_w97.pt,log_w97.txt +23:28:19 waited 76s for global r=380 +23:28:28 global r=380 load 4.304 -> 4.298 +23:29:01 step 115660 loss=3.4203 lr=6.66e-05 tok=15.193B dt=5073 wait=113 +23:30:05 step 115680 loss=3.2756 lr=6.66e-05 tok=15.195B dt=5136 wait=113 +23:31:09 step 115700 loss=3.3166 lr=6.65e-05 tok=15.198B dt=5200 wait=114 +23:32:12 step 115720 loss=3.4043 lr=6.65e-05 tok=15.200B dt=5264 wait=114 +23:33:16 step 115740 loss=3.2518 lr=6.65e-05 tok=15.203B dt=5328 wait=114 +23:34:20 step 115760 loss=3.4415 lr=6.65e-05 tok=15.206B dt=5391 wait=115 +23:35:24 step 115780 loss=3.3278 lr=6.65e-05 tok=15.208B dt=5455 wait=115 +23:36:28 step 115800 loss=3.3971 lr=6.64e-05 tok=15.211B dt=5519 wait=115 +23:36:41 pushed ckpt_w97.pt,log_w97.txt +23:37:42 waited 60s for global r=381 +23:37:48 global r=381 load 4.287 -> 4.286 +23:38:53 step 115820 loss=3.3111 lr=6.64e-05 tok=15.214B dt=5665 wait=116 +23:39:57 step 115840 loss=3.4006 lr=6.64e-05 tok=15.216B dt=5729 wait=116 +23:41:00 step 115860 loss=3.4002 lr=6.64e-05 tok=15.219B dt=5792 wait=116 +23:42:05 step 115880 loss=3.3086 lr=6.64e-05 tok=15.221B dt=5856 wait=116 +23:43:08 step 115900 loss=3.3540 lr=6.63e-05 tok=15.224B dt=5920 wait=117 +23:44:12 step 115920 loss=3.4263 lr=6.63e-05 tok=15.227B dt=5984 wait=117 +23:45:15 step 115940 loss=3.3176 lr=6.63e-05 tok=15.229B dt=6047 wait=117 +23:46:01 pushed ckpt_w97.pt,log_w97.txt +23:47:02 waited 60s for global r=382 +23:47:07 global r=382 load 4.294 -> 4.287 +23:47:40 step 115960 loss=3.4135 lr=6.63e-05 tok=15.232B dt=6192 wait=118 +23:48:44 step 115980 loss=3.2865 lr=6.62e-05 tok=15.234B dt=6256 wait=118 +23:49:48 step 116000 loss=3.3613 lr=6.62e-05 tok=15.237B dt=6319 wait=118 +23:50:52 step 116020 loss=3.4232 lr=6.62e-05 tok=15.240B dt=6383 wait=119 +23:51:55 step 116040 loss=3.2898 lr=6.62e-05 tok=15.242B dt=6447 wait=119 +23:52:59 step 116060 loss=3.3909 lr=6.62e-05 tok=15.245B dt=6511 wait=119 +23:54:03 step 116080 loss=3.3360 lr=6.61e-05 tok=15.248B dt=6575 wait=120 +23:55:07 step 116100 loss=3.3448 lr=6.61e-05 tok=15.250B dt=6638 wait=120 +23:55:24 pushed ckpt_w97.pt,log_w97.txt +23:56:39 waited 76s for global r=383 +23:56:45 global r=383 load 4.306 -> 4.287 +23:57:50 step 116120 loss=3.3067 lr=6.61e-05 tok=15.253B dt=6802 wait=120 +23:58:54 step 116140 loss=3.3848 lr=6.61e-05 tok=15.255B dt=6865 wait=121 +23:59:57 step 116160 loss=3.3997 lr=6.61e-05 tok=15.258B dt=6929 wait=121 +00:01:01 step 116180 loss=3.2664 lr=6.60e-05 tok=15.261B dt=6993 wait=121 +00:02:05 step 116200 loss=3.2838 lr=6.60e-05 tok=15.263B dt=7056 wait=122 +00:03:09 step 116220 loss=3.3876 lr=6.60e-05 tok=15.266B dt=7120 wait=122 +00:04:12 step 116240 loss=3.2968 lr=6.60e-05 tok=15.269B dt=7184 wait=122 +00:04:58 pushed ckpt_w97.pt,log_w97.txt +00:05:59 waited 60s for global r=384 +00:06:05 global r=384 load 4.285 -> 4.282 +00:06:38 step 116260 loss=3.4522 lr=6.60e-05 tok=15.271B dt=7329 wait=123 +00:07:41 step 116280 loss=3.2834 lr=6.59e-05 tok=15.274B dt=7393 wait=123 +00:08:45 step 116300 loss=3.3328 lr=6.59e-05 tok=15.276B dt=7456 wait=123 +00:09:48 step 116320 loss=3.3847 lr=6.59e-05 tok=15.279B dt=7520 wait=124 +00:10:52 step 116340 loss=3.2588 lr=6.59e-05 tok=15.282B dt=7584 wait=124 +00:11:56 step 116360 loss=3.4235 lr=6.59e-05 tok=15.284B dt=7648 wait=124 +00:13:00 step 116380 loss=3.3351 lr=6.58e-05 tok=15.287B dt=7711 wait=124 +00:14:04 step 116400 loss=3.3848 lr=6.58e-05 tok=15.290B dt=7775 wait=125 +00:14:18 pushed ckpt_w97.pt,log_w97.txt +00:15:33 waited 76s for global r=385 +00:15:41 global r=385 load 4.326 -> 4.298 +00:16:46 step 116420 loss=3.3654 lr=6.58e-05 tok=15.292B dt=7937 wait=125 +00:17:49 step 116440 loss=3.3541 lr=6.58e-05 tok=15.295B dt=8001 wait=125 +00:18:53 step 116460 loss=3.4061 lr=6.58e-05 tok=15.297B dt=8065 wait=126 +00:19:58 step 116480 loss=3.1630 lr=6.57e-05 tok=15.300B dt=8130 wait=126 +00:21:02 step 116500 loss=3.3316 lr=6.57e-05 tok=15.303B dt=8193 wait=126 +00:22:06 step 116520 loss=3.4348 lr=6.57e-05 tok=15.305B dt=8257 wait=127 +00:23:09 step 116540 loss=3.2954 lr=6.57e-05 tok=15.308B dt=8321 wait=127 +00:23:55 pushed ckpt_w97.pt,log_w97.txt +00:24:56 waited 60s for global r=386 +00:25:01 global r=386 load 4.309 -> 4.284 +00:25:34 step 116560 loss=3.4489 lr=6.57e-05 tok=15.311B dt=8466 wait=127 +00:26:38 step 116580 loss=3.2944 lr=6.56e-05 tok=15.313B dt=8529 wait=127 +00:27:41 step 116600 loss=3.2409 lr=6.56e-05 tok=15.316B dt=8593 wait=128 +00:28:45 step 116620 loss=3.4130 lr=6.56e-05 tok=15.318B dt=8657 wait=128 +00:29:49 step 116640 loss=3.3364 lr=6.56e-05 tok=15.321B dt=8721 wait=128 +00:30:53 step 116660 loss=3.3554 lr=6.56e-05 tok=15.324B dt=8785 wait=129 +00:31:57 step 116680 loss=3.3240 lr=6.55e-05 tok=15.326B dt=8848 wait=129 +00:33:00 step 116700 loss=3.3610 lr=6.55e-05 tok=15.329B dt=8912 wait=129 +00:33:16 pushed ckpt_w97.pt,log_w97.txt +00:34:16 waited 60s for global r=387 +00:34:26 global r=387 load 4.290 -> 4.274 +00:35:31 step 116720 loss=3.3424 lr=6.55e-05 tok=15.331B dt=9062 wait=130 +00:36:34 step 116740 loss=3.3648 lr=6.55e-05 tok=15.334B dt=9126 wait=130 +00:37:38 step 116760 loss=3.3448 lr=6.54e-05 tok=15.337B dt=9189 wait=130 +00:38:41 step 116780 loss=3.3489 lr=6.54e-05 tok=15.339B dt=9253 wait=131 +00:39:45 step 116800 loss=3.3372 lr=6.54e-05 tok=15.342B dt=9317 wait=131 +00:40:50 step 116820 loss=3.3802 lr=6.54e-05 tok=15.345B dt=9381 wait=131 +00:41:54 step 116840 loss=3.3203 lr=6.54e-05 tok=15.347B dt=9445 wait=132 +00:42:38 pushed ckpt_w97.pt,log_w97.txt +00:43:53 waited 76s for global r=388 +00:44:02 global r=388 load 4.291 -> 4.278 +00:44:35 step 116860 loss=3.3980 lr=6.53e-05 tok=15.350B dt=9606 wait=132 +00:45:38 step 116880 loss=3.2914 lr=6.53e-05 tok=15.352B dt=9670 wait=132 +00:46:42 step 116900 loss=3.2794 lr=6.53e-05 tok=15.355B dt=9733 wait=132 +00:47:45 step 116920 loss=3.3784 lr=6.53e-05 tok=15.358B dt=9797 wait=133 +00:48:49 step 116940 loss=3.2498 lr=6.53e-05 tok=15.360B dt=9861 wait=133 +00:49:53 step 116960 loss=3.3485 lr=6.52e-05 tok=15.363B dt=9924 wait=133 +00:50:56 step 116980 loss=3.3344 lr=6.52e-05 tok=15.366B dt=9988 wait=134 +00:52:00 step 117000 loss=3.3372 lr=6.52e-05 tok=15.368B dt=10052 wait=134 +00:52:12 pushed ckpt_w97.pt,log_w97.txt +00:53:12 waited 60s for global r=389 +00:53:17 global r=389 load 4.281 -> 4.281 +00:54:22 step 117020 loss=3.3159 lr=6.52e-05 tok=15.371B dt=10194 wait=134 +00:55:26 step 117040 loss=3.3841 lr=6.52e-05 tok=15.373B dt=10257 wait=135 +00:56:29 step 117060 loss=3.3649 lr=6.51e-05 tok=15.376B dt=10321 wait=135 +00:57:33 step 117080 loss=3.3482 lr=6.51e-05 tok=15.379B dt=10385 wait=135 +00:58:37 step 117100 loss=3.3493 lr=6.51e-05 tok=15.381B dt=10448 wait=136 +00:59:40 step 117120 loss=3.4065 lr=6.51e-05 tok=15.384B dt=10512 wait=136 +01:00:44 step 117140 loss=3.2931 lr=6.51e-05 tok=15.387B dt=10575 wait=136 diff --git a/log_w98.txt b/log_w98.txt new file mode 100644 index 0000000000000000000000000000000000000000..4943c9dd6a7763f07c49e97e3c2c923aff54c37b --- /dev/null +++ b/log_w98.txt @@ -0,0 +1,193 @@ +launch +22:02:25 name=w98 dim=2048 layers=18 seq=2048 bs=64 mb=16 shard 45/4 ds=openbmb/Ultra-FineWeb-L3 cfg=Ultra-FineWeb-L3-en-Multi-Style-Synthetic,Ultra-FineWeb-L3-en-QA-Synthetic +22:02:32 tokenizer openbmb/MiniCPM4-0.5B vocab=73440 model_vocab=73760 +22:02:32 ssm: 0.25 of rows, 10884 capitalised / 127 digit pieces +22:02:41 params 1.08B +22:02:43 resumed step=114300 seen=15.02B round=371 +22:02:43 stream salt=114300 +22:04:26 blocks compiled +22:04:26 start: accum=4, 131072 tokens per optimiser step +22:06:54 step 114301 loss=3.0255 lr=6.80e-05 tok=15.021B dt=148 wait=104 +22:06:58 step 114302 loss=3.0930 lr=6.80e-05 tok=15.021B dt=152 wait=104 +22:08:02 step 114320 loss=3.3749 lr=6.80e-05 tok=15.023B dt=216 wait=105 +22:09:07 step 114340 loss=3.3986 lr=6.79e-05 tok=15.026B dt=281 wait=105 +22:10:10 step 114360 loss=3.3609 lr=6.79e-05 tok=15.029B dt=344 wait=105 +22:11:13 step 114380 loss=3.4427 lr=6.79e-05 tok=15.031B dt=407 wait=106 +22:12:17 step 114400 loss=3.5054 lr=6.79e-05 tok=15.034B dt=471 wait=106 +22:12:27 global r=372 load 4.316 -> 4.314 +22:12:59 step 114460 loss=3.5665 lr=6.78e-05 tok=15.035B dt=513 wait=106 +22:14:02 step 114480 loss=3.3418 lr=6.78e-05 tok=15.038B dt=576 wait=106 +22:15:05 step 114500 loss=3.3422 lr=6.78e-05 tok=15.041B dt=639 wait=107 +22:16:08 step 114520 loss=3.3877 lr=6.78e-05 tok=15.043B dt=702 wait=107 +22:17:12 step 114540 loss=3.3437 lr=6.77e-05 tok=15.046B dt=766 wait=107 +22:18:15 step 114560 loss=3.3859 lr=6.77e-05 tok=15.048B dt=829 wait=107 +22:19:18 step 114580 loss=3.2953 lr=6.77e-05 tok=15.051B dt=892 wait=108 +22:20:22 step 114600 loss=3.3378 lr=6.77e-05 tok=15.054B dt=956 wait=108 +22:20:41 pushed ckpt_w98.pt,log_w98.txt +22:21:56 waited 76s for global r=373 +22:22:02 global r=373 load 4.327 -> 4.314 +22:23:07 step 114620 loss=3.3063 lr=6.77e-05 tok=15.056B dt=1121 wait=108 +22:24:10 step 114640 loss=3.3631 lr=6.76e-05 tok=15.059B dt=1184 wait=109 +22:25:14 step 114660 loss=3.3452 lr=6.76e-05 tok=15.061B dt=1248 wait=109 +22:26:17 step 114680 loss=3.2734 lr=6.76e-05 tok=15.064B dt=1311 wait=109 +22:27:20 step 114700 loss=3.3472 lr=6.76e-05 tok=15.067B dt=1374 wait=110 +22:28:24 step 114720 loss=3.4314 lr=6.75e-05 tok=15.069B dt=1438 wait=110 +22:29:27 step 114740 loss=3.3229 lr=6.75e-05 tok=15.072B dt=1501 wait=110 +22:30:10 pushed ckpt_w98.pt,log_w98.txt +22:31:26 waited 76s for global r=374 +22:31:32 global r=374 load 4.308 -> 4.314 +22:32:05 step 114760 loss=3.4247 lr=6.75e-05 tok=15.075B dt=1659 wait=111 +22:33:08 step 114780 loss=3.2948 lr=6.75e-05 tok=15.077B dt=1722 wait=111 +22:34:11 step 114800 loss=3.3073 lr=6.75e-05 tok=15.080B dt=1785 wait=111 +22:35:15 step 114820 loss=3.3891 lr=6.74e-05 tok=15.082B dt=1849 wait=112 +22:36:18 step 114840 loss=3.3069 lr=6.74e-05 tok=15.085B dt=1912 wait=112 +22:37:22 step 114860 loss=3.3707 lr=6.74e-05 tok=15.088B dt=1976 wait=112 +22:38:25 step 114880 loss=3.3264 lr=6.74e-05 tok=15.090B dt=2039 wait=112 +22:39:29 step 114900 loss=3.4202 lr=6.74e-05 tok=15.093B dt=2103 wait=113 +22:39:41 pushed ckpt_w98.pt,log_w98.txt +22:40:56 waited 76s for global r=375 +22:41:02 global r=375 load 4.322 -> 4.311 +22:42:05 step 114920 loss=3.2912 lr=6.73e-05 tok=15.096B dt=2259 wait=113 +22:43:09 step 114940 loss=3.3401 lr=6.73e-05 tok=15.098B dt=2323 wait=113 +22:44:12 step 114960 loss=3.3547 lr=6.73e-05 tok=15.101B dt=2386 wait=114 +22:45:16 step 114980 loss=3.2882 lr=6.73e-05 tok=15.103B dt=2449 wait=114 +22:46:19 step 115000 loss=3.3170 lr=6.73e-05 tok=15.106B dt=2513 wait=114 +22:47:22 step 115020 loss=3.4500 lr=6.72e-05 tok=15.109B dt=2576 wait=115 +22:48:26 step 115040 loss=3.3130 lr=6.72e-05 tok=15.111B dt=2640 wait=115 +22:49:11 pushed ckpt_w98.pt,log_w98.txt +22:50:26 waited 76s for global r=376 +22:50:33 global r=376 load 4.334 -> 4.313 +22:51:06 step 115060 loss=3.3620 lr=6.72e-05 tok=15.114B dt=2800 wait=115 +22:52:10 step 115080 loss=3.3239 lr=6.72e-05 tok=15.117B dt=2863 wait=116 +22:53:13 step 115100 loss=3.2822 lr=6.72e-05 tok=15.119B dt=2927 wait=116 +22:54:16 step 115120 loss=3.3848 lr=6.71e-05 tok=15.122B dt=2990 wait=116 +22:55:20 step 115140 loss=3.2311 lr=6.71e-05 tok=15.124B dt=3054 wait=117 +22:56:23 step 115160 loss=3.4307 lr=6.71e-05 tok=15.127B dt=3117 wait=117 +22:57:26 step 115180 loss=3.2863 lr=6.71e-05 tok=15.130B dt=3180 wait=117 +22:58:30 step 115200 loss=3.3641 lr=6.71e-05 tok=15.132B dt=3244 wait=118 +22:58:44 pushed ckpt_w98.pt,log_w98.txt +22:59:44 waited 60s for global r=377 +22:59:51 global r=377 load 4.338 -> 4.296 +23:00:56 step 115220 loss=3.3112 lr=6.70e-05 tok=15.135B dt=3390 wait=118 +23:01:59 step 115240 loss=3.3857 lr=6.70e-05 tok=15.138B dt=3453 wait=118 +23:03:03 step 115260 loss=3.3454 lr=6.70e-05 tok=15.140B dt=3517 wait=119 +23:04:06 step 115280 loss=3.3296 lr=6.70e-05 tok=15.143B dt=3580 wait=119 +23:05:09 step 115300 loss=3.2873 lr=6.69e-05 tok=15.145B dt=3643 wait=119 +23:06:12 step 115320 loss=3.4161 lr=6.69e-05 tok=15.148B dt=3706 wait=120 +23:07:15 step 115340 loss=3.2856 lr=6.69e-05 tok=15.151B dt=3769 wait=120 +23:07:59 pushed ckpt_w98.pt,log_w98.txt +23:09:14 waited 76s for global r=378 +23:09:20 global r=378 load 4.297 -> 4.308 +23:09:53 step 115360 loss=3.4141 lr=6.69e-05 tok=15.153B dt=3927 wait=120 +23:10:56 step 115380 loss=3.2919 lr=6.69e-05 tok=15.156B dt=3990 wait=121 +23:11:59 step 115400 loss=3.2935 lr=6.68e-05 tok=15.158B dt=4053 wait=121 +23:13:03 step 115420 loss=3.3481 lr=6.68e-05 tok=15.161B dt=4117 wait=121 +23:14:07 step 115440 loss=3.2057 lr=6.68e-05 tok=15.164B dt=4181 wait=121 +23:15:10 step 115460 loss=3.4098 lr=6.68e-05 tok=15.166B dt=4244 wait=122 +23:16:13 step 115480 loss=3.3039 lr=6.68e-05 tok=15.169B dt=4307 wait=122 +23:17:17 step 115500 loss=3.3541 lr=6.67e-05 tok=15.172B dt=4371 wait=122 +23:17:29 pushed ckpt_w98.pt,log_w98.txt +23:18:45 waited 76s for global r=379 +23:18:53 global r=379 load 4.290 -> 4.306 +23:19:56 step 115520 loss=3.3218 lr=6.67e-05 tok=15.174B dt=4530 wait=123 +23:20:59 step 115540 loss=3.3731 lr=6.67e-05 tok=15.177B dt=4593 wait=123 +23:22:03 step 115560 loss=3.3802 lr=6.67e-05 tok=15.179B dt=4657 wait=123 +23:23:06 step 115580 loss=3.3088 lr=6.67e-05 tok=15.182B dt=4720 wait=124 +23:24:09 step 115600 loss=3.3147 lr=6.66e-05 tok=15.185B dt=4783 wait=124 +23:25:12 step 115620 loss=3.4049 lr=6.66e-05 tok=15.187B dt=4846 wait=124 +23:26:15 step 115640 loss=3.2778 lr=6.66e-05 tok=15.190B dt=4909 wait=125 +23:27:01 pushed ckpt_w98.pt,log_w98.txt +23:28:16 waited 76s for global r=380 +23:28:21 global r=380 load 4.316 -> 4.298 +23:28:53 step 115660 loss=3.3786 lr=6.66e-05 tok=15.193B dt=5067 wait=125 +23:29:56 step 115680 loss=3.3082 lr=6.66e-05 tok=15.195B dt=5130 wait=125 +23:30:59 step 115700 loss=3.2695 lr=6.65e-05 tok=15.198B dt=5193 wait=126 +23:32:03 step 115720 loss=3.3681 lr=6.65e-05 tok=15.200B dt=5257 wait=126 +23:33:07 step 115740 loss=3.2769 lr=6.65e-05 tok=15.203B dt=5321 wait=126 +23:34:10 step 115760 loss=3.3733 lr=6.65e-05 tok=15.206B dt=5384 wait=126 +23:35:14 step 115780 loss=3.2897 lr=6.65e-05 tok=15.208B dt=5448 wait=127 +23:36:17 step 115800 loss=3.3466 lr=6.64e-05 tok=15.211B dt=5511 wait=127 +23:36:29 pushed ckpt_w98.pt,log_w98.txt +23:37:44 waited 76s for global r=381 +23:37:49 global r=381 load 4.262 -> 4.286 +23:38:53 step 115820 loss=3.2329 lr=6.64e-05 tok=15.214B dt=5667 wait=127 +23:39:56 step 115840 loss=3.3709 lr=6.64e-05 tok=15.216B dt=5730 wait=128 +23:41:00 step 115860 loss=3.3546 lr=6.64e-05 tok=15.219B dt=5794 wait=128 +23:42:06 step 115880 loss=2.8725 lr=6.64e-05 tok=15.221B dt=5860 wait=128 +23:43:10 step 115900 loss=3.3174 lr=6.63e-05 tok=15.224B dt=5924 wait=129 +23:44:13 step 115920 loss=3.4225 lr=6.63e-05 tok=15.227B dt=5987 wait=129 +23:45:16 step 115940 loss=3.2587 lr=6.63e-05 tok=15.229B dt=6050 wait=129 +23:46:00 pushed ckpt_w98.pt,log_w98.txt +23:47:01 waited 61s for global r=382 +23:47:06 global r=382 load 4.309 -> 4.287 +23:47:38 step 115960 loss=3.3751 lr=6.63e-05 tok=15.232B dt=6192 wait=129 +23:48:42 step 115980 loss=3.2887 lr=6.62e-05 tok=15.234B dt=6256 wait=130 +23:49:45 step 116000 loss=3.3132 lr=6.62e-05 tok=15.237B dt=6319 wait=130 +23:50:49 step 116020 loss=3.3993 lr=6.62e-05 tok=15.240B dt=6383 wait=130 +23:51:52 step 116040 loss=3.2996 lr=6.62e-05 tok=15.242B dt=6446 wait=131 +23:52:55 step 116060 loss=3.3753 lr=6.62e-05 tok=15.245B dt=6509 wait=131 +23:53:59 step 116080 loss=3.2700 lr=6.61e-05 tok=15.248B dt=6573 wait=131 +23:55:03 step 116100 loss=3.3647 lr=6.61e-05 tok=15.250B dt=6637 wait=132 +23:55:18 pushed ckpt_w98.pt,log_w98.txt +23:56:34 waited 76s for global r=383 +23:56:42 global r=383 load 4.291 -> 4.287 +23:57:46 step 116120 loss=3.3011 lr=6.61e-05 tok=15.253B dt=6800 wait=132 +23:58:50 step 116140 loss=3.3294 lr=6.61e-05 tok=15.255B dt=6863 wait=132 +23:59:53 step 116160 loss=3.3417 lr=6.61e-05 tok=15.258B dt=6927 wait=133 +00:00:57 step 116180 loss=3.2774 lr=6.60e-05 tok=15.261B dt=6991 wait=133 +00:02:00 step 116200 loss=3.3545 lr=6.60e-05 tok=15.263B dt=7054 wait=133 +00:03:04 step 116220 loss=3.3541 lr=6.60e-05 tok=15.266B dt=7118 wait=134 +00:04:08 step 116240 loss=3.2753 lr=6.60e-05 tok=15.269B dt=7182 wait=134 +00:04:51 pushed ckpt_w98.pt,log_w98.txt +00:06:07 waited 76s for global r=384 +00:06:12 global r=384 load 4.286 -> 4.282 +00:06:44 step 116260 loss=3.4089 lr=6.60e-05 tok=15.271B dt=7338 wait=134 +00:07:48 step 116280 loss=3.3083 lr=6.59e-05 tok=15.274B dt=7402 wait=135 +00:08:51 step 116300 loss=3.2744 lr=6.59e-05 tok=15.276B dt=7465 wait=135 +00:09:54 step 116320 loss=3.3885 lr=6.59e-05 tok=15.279B dt=7528 wait=135 +00:10:58 step 116340 loss=3.3403 lr=6.59e-05 tok=15.282B dt=7592 wait=135 +00:12:01 step 116360 loss=3.2489 lr=6.59e-05 tok=15.284B dt=7655 wait=136 +00:13:05 step 116380 loss=3.2147 lr=6.58e-05 tok=15.287B dt=7719 wait=136 +00:14:09 step 116400 loss=3.3510 lr=6.58e-05 tok=15.290B dt=7783 wait=136 +00:14:21 pushed ckpt_w98.pt,log_w98.txt +00:15:37 waited 76s for global r=385 +00:15:42 global r=385 load 4.327 -> 4.298 +00:16:47 step 116420 loss=3.2003 lr=6.58e-05 tok=15.292B dt=7941 wait=137 +00:17:50 step 116440 loss=3.3470 lr=6.58e-05 tok=15.295B dt=8004 wait=137 +00:18:54 step 116460 loss=3.3516 lr=6.58e-05 tok=15.297B dt=8068 wait=137 +00:19:57 step 116480 loss=3.2849 lr=6.57e-05 tok=15.300B dt=8131 wait=138 +00:21:01 step 116500 loss=3.2521 lr=6.57e-05 tok=15.303B dt=8195 wait=138 +00:22:05 step 116520 loss=3.3796 lr=6.57e-05 tok=15.305B dt=8259 wait=138 +00:23:08 step 116540 loss=3.2798 lr=6.57e-05 tok=15.308B dt=8322 wait=139 +00:23:52 pushed ckpt_w98.pt,log_w98.txt +00:24:52 waited 60s for global r=386 +00:25:01 global r=386 load 4.282 -> 4.284 +00:25:33 step 116560 loss=3.3992 lr=6.57e-05 tok=15.311B dt=8467 wait=139 +00:26:36 step 116580 loss=3.2726 lr=6.56e-05 tok=15.313B dt=8530 wait=139 +00:27:41 step 116600 loss=3.2368 lr=6.56e-05 tok=15.316B dt=8595 wait=139 +00:28:45 step 116620 loss=3.4181 lr=6.56e-05 tok=15.318B dt=8659 wait=140 +00:29:48 step 116640 loss=3.3252 lr=6.56e-05 tok=15.321B dt=8722 wait=140 +00:30:52 step 116660 loss=3.3913 lr=6.56e-05 tok=15.324B dt=8786 wait=140 +00:31:56 step 116680 loss=3.3885 lr=6.55e-05 tok=15.326B dt=8850 wait=141 +00:33:00 step 116700 loss=3.3276 lr=6.55e-05 tok=15.329B dt=8914 wait=141 +00:33:13 pushed ckpt_w98.pt,log_w98.txt +00:34:14 waited 60s for global r=387 +00:34:19 global r=387 load 4.287 -> 4.274 +00:35:24 step 116720 loss=3.3334 lr=6.55e-05 tok=15.331B dt=9058 wait=141 +00:36:31 step 116740 loss=3.3649 lr=6.55e-05 tok=15.334B dt=9125 wait=142 +00:37:38 step 116760 loss=3.3795 lr=6.54e-05 tok=15.337B dt=9192 wait=142 +00:38:41 step 116780 loss=3.2831 lr=6.54e-05 tok=15.339B dt=9255 wait=142 +00:39:44 step 116800 loss=3.3039 lr=6.54e-05 tok=15.342B dt=9318 wait=143 +00:40:48 step 116820 loss=3.4081 lr=6.54e-05 tok=15.345B dt=9381 wait=143 +00:41:51 step 116840 loss=3.2625 lr=6.54e-05 tok=15.347B dt=9444 wait=143 +00:42:34 pushed ckpt_w98.pt,log_w98.txt +00:43:49 waited 76s for global r=388 +00:43:54 global r=388 load 4.287 -> 4.278 +00:44:26 step 116860 loss=3.4020 lr=6.53e-05 tok=15.350B dt=9600 wait=144 +00:45:30 step 116880 loss=3.2989 lr=6.53e-05 tok=15.352B dt=9664 wait=144 +00:46:33 step 116900 loss=3.3410 lr=6.53e-05 tok=15.355B dt=9727 wait=144 +00:47:36 step 116920 loss=3.3426 lr=6.53e-05 tok=15.358B dt=9790 wait=145 +00:48:40 step 116940 loss=3.3372 lr=6.53e-05 tok=15.360B dt=9854 wait=145 +00:49:44 step 116960 loss=3.3404 lr=6.52e-05 tok=15.363B dt=9918 wait=145 +00:50:47 step 116980 loss=3.3177 lr=6.52e-05 tok=15.366B dt=9981 wait=145 +00:51:51 step 117000 loss=3.2701 lr=6.52e-05 tok=15.368B dt=10045 wait=146 diff --git a/pilots/ardec.log b/pilots/ardec.log new file mode 100644 index 0000000000000000000000000000000000000000..f370ac647879c2c745580e5089527051e66f657a --- /dev/null +++ b/pilots/ardec.log @@ -0,0 +1,24 @@ +EMA round 506 step 135000 +FACTS ar1 exam 27/60 (45%) popqa 27/200 (13.5%) 18s +FACTS blk4 exam 27/60 (45%) popqa 27/200 (13.5%) 18s +FACTS blk8 exam 27/60 (45%) popqa 27/200 (13.5%) 18s +FACTS par exam 27/60 (45%) popqa 27/200 (13.5%) 18s +[ar1] The capital of France is ||| the city of Paris, the capital of the Guelts-Santhes culture. It is named after the French colonel, Pierre-Jean-Dameu-Louis, who was born in 1610. The French capital is Paris, France, located at 18. +[ar1] In the morning, the old fisherman walked down to ||| his fisherman and caught the fish at the sea. He sat there for a while, watching the fish wryly. “Was you thinking about me?” he asked. “I’m so tired of my friends,” he said, “and I’m so sorry for your lifelessness. +[ar1] Photosynthesis is the process by which ||| plants convert food into energy through the process. This process occurs when plants absorb sunlight, which can be broken down into oxygen and carbon dioxide. When plants produce energy, these substances are used in the form of carbohydrates and water, which are essential nutrients for plant growth +[ar1] The main causes of the First World War were ||| the diplomatic and political differences between France and Germany. The war was resolved by the Treaty of Strasbourg. The war ended in 1915, and the conflict was still ongoing. The outbreak of World War I was marked by the defeat of the Allies, including the Battle of the Somme, +[ar1] To make a simple omelette, first ||| create a simple boundary layer of paraffin wax, then again cut the outer edges and layer of wax. Cut the wax into two halves, one with paraffin wax and one without wax, and cut the edges of the wax, leaving the leftover parts to be removed. If needed, cut the wax into a +TEXT ar1 distinct-2 0.933 +[blk8] The capital of France is ||| the city of Paris, located in the heart of the French desert. The capital of France is Paris, USA, located in the heart of the French desert. The capital of France is Paris, France, located in the heart of the French desert. The capital of France is Paris, France, located in +[blk8] In the morning, the old fisherman walked down to ||| his house, swayed against the banks of the river, and began to make his way back home. He quickly sank into the water, and he tried to find a home. However, as the water got deeper, he found a strange sound. He tried to move to another house, but he did not +[blk8] Photosynthesis is the process by which ||| plants convert light energy into chemical energy through the production of carbon dioxide. This process occurs when plants absorb sunlight, which is absorbed by their leaves, and release carbon dioxide into the atmosphere. Carbon dioxide is a byproduct of photosynthesis, and when plants absorb some +[blk8] The main causes of the First World War were ||| the fall of the Ottoman Empire, the outbreak of the Second World War, and the outbreak of the First World War. The fall of the Ottoman Empire was the primary cause of the First World War. The outbreak of the First World War was primarily caused by the outbreak of the Second World War. The outbreak o +[blk8] To make a simple omelette, first ||| prepare the ingredients in a mixing bowl. Begin by combining the eggs, milk, and sugar in a large bowl. Add the milk, the sugar, and the eggs in a small bowl. Stir in the milk, the sugar, and the eggs. Grain the mixture to a large bowl and mix well. Add the +TEXT blk8 distinct-2 0.660 +[par] The capital of France is ||| the city of Lorraine. History The capital of France was established in 1891 with the creation of the French Republic in 1891 and of the Republic of France in 1914. References Capitals of the French Republic (state) +[par] In the morning, the old fisherman walked down to ||| his home, Hudson Lake, on the far shore of the lake. He was struck by the beauty, beauty, and mystery of the lake. He stepped out of the boat, walked to the shore, and returned to the lake. On October 15, 1909, the 2 +[par] Photosynthesis is the process by which ||| plants obtain energy from sunlight in the form of glucose. This process occurs when plants use sunlight to produce energy, which is then converted into chemical energy in the form of carbon dioxide. Unlike the energy produced through photosynthesis, the energy released during photosynthesis does not +[par] The main causes of the First World War were ||| the outbreak of World War I and the fall of the Soviet Union. The outbreak of World War I was caused by the collapse of the Soviet Union and the fall of the Soviet Union. The outbreak of World War II was caused by the collapse of the Soviet Union and the fall of the Soviet Union. The outbreak of the +[par] To make a simple omelette, first ||| gather the ingredients: 1,1,2,2,3,4,4,5,5,6,7,8,9,1,1,2,2,3,4,4,5,5,6,7,8,9,1,1,2,2 +TEXT par distinct-2 0.648 +ARDEC DONE diff --git a/pilots/decode_pilot.json b/pilots/decode_pilot.json new file mode 100644 index 0000000000000000000000000000000000000000..f21c60832ceb72fc12b035ba12252e81b36f6602 --- /dev/null +++ b/pilots/decode_pilot.json @@ -0,0 +1,72 @@ +{ + "span": { + "one_pass": { + "ce@0.3": 2.7062206268310547, + "ce@0.6": 3.5128703117370605, + "pair": 0.15048712595685457, + "tok": 0.3763739085772984, + "ce@0.9": 5.025173664093018, + "ce": 3.7480882008870444, + "ce_units@0.6": 3.5128703117370605 + }, + "t0.5_K4": [ + 0.4241751432418823, + 0.2007444202899933 + ], + "t0.5_K8": [ + 0.4246068596839905, + 0.2075682431459427 + ], + "t0.5_K16": [ + 0.4234350919723511, + 0.21104218065738678 + ], + "t0.9_K4": [ + 0.14572523534297943, + 0.04976913332939148 + ], + "t0.9_K8": [ + 0.13873910903930664, + 0.05241844058036804 + ], + "t0.9_K16": [ + 0.1360260397195816, + 0.05635455250740051 + ] + }, + "indep": { + "one_pass": { + "ce@0.3": 2.541595935821533, + "ce@0.6": 3.409759521484375, + "pair": 0.16301322199025747, + "tok": 0.3892141756548536, + "ce@0.9": 5.002624988555908, + "ce": 3.651326815287272, + "ce_units@0.6": 3.409759521484375 + }, + "t0.5_K4": [ + 0.44323158264160156, + 0.21476426720619202 + ], + "t0.5_K8": [ + 0.4416281282901764, + 0.22258064150810242 + ], + "t0.5_K16": [ + 0.43872958421707153, + 0.22332505881786346 + ], + "t0.9_K4": [ + 0.14833655953407288, + 0.05037468671798706 + ], + "t0.9_K8": [ + 0.142706960439682, + 0.0537809394299984 + ], + "t0.9_K16": [ + 0.13690778613090515, + 0.05514344200491905 + ] + } +} \ No newline at end of file diff --git a/pilots/decode_pilot.log b/pilots/decode_pilot.log new file mode 100644 index 0000000000000000000000000000000000000000..c4fa7e082007dcfeefdf1c49069c11886086f84c --- /dev/null +++ b/pilots/decode_pilot.log @@ -0,0 +1,9 @@ +18:35:16 span one-pass ce 3.7481 pair 0.1505 tok 0.3764 (expect 3.7481) | 296s +18:35:17 span decode t=0.5 K=4 tok 0.4242 pair 0.2007 K=8 tok 0.4246 pair 0.2076 K=16 tok 0.4234 pair 0.2110 +18:35:18 span decode t=0.9 K=4 tok 0.1457 pair 0.0498 K=8 tok 0.1387 pair 0.0524 K=16 tok 0.1360 pair 0.0564 +18:39:32 indep one-pass ce 3.6513 pair 0.1630 tok 0.3892 (expect 3.6513) | 253s +18:39:33 indep decode t=0.5 K=4 tok 0.4432 pair 0.2148 K=8 tok 0.4416 pair 0.2226 K=16 tok 0.4387 pair 0.2233 +18:39:34 indep decode t=0.9 K=4 tok 0.1483 pair 0.0504 K=8 tok 0.1427 pair 0.0538 K=16 tok 0.1369 pair 0.0551 +18:39:34 --- decode, span minus indep (positive = span better) --- +18:39:34 t=0.5 K=4 dtok -0.0191 dpair -0.0140 K=8 dtok -0.0170 dpair -0.0150 K=16 dtok -0.0153 dpair -0.0123 +18:39:34 t=0.9 K=4 dtok -0.0026 dpair -0.0006 K=8 dtok -0.0040 dpair -0.0014 K=16 dtok -0.0009 dpair +0.0012 diff --git a/pilots/engram1b.log b/pilots/engram1b.log new file mode 100644 index 0000000000000000000000000000000000000000..b1b5a534b575f786bfb882ff003ad3ad2d4c7fec --- /dev/null +++ b/pilots/engram1b.log @@ -0,0 +1,52 @@ +13:35:15 engram test from global round 506 step 135000: 1000 steps x 64 rows, arms ['mem', 'ctrl'], slots 1000000 x 256 +13:50:09 data ready in 894s +13:50:21 mem arm: 256M table params, 12.6M projection params, inject before blocks [1, 8] +13:53:08 mem step 50 loss 3.3029 (167s) |wv| 3.974,3.925 +13:55:48 mem step 100 loss 3.7715 (327s) |wv| 6.333,6.528 +13:58:28 mem step 150 loss 3.1712 (487s) |wv| 8.301,8.584 +14:01:08 mem step 200 loss 3.3070 (647s) |wv| 9.927,10.253 +14:03:48 mem step 250 loss 2.9819 (807s) |wv| 11.281,11.650 +14:03:55 EVAL mem step 250 | facts mc 0.5505 nll 6.0279 em 0.0752 +14:06:35 mem step 300 loss 2.9541 (974s) |wv| 12.476,12.890 +14:09:15 mem step 350 loss 3.3530 (1134s) |wv| 13.582,14.027 +14:11:55 mem step 400 loss 3.2561 (1294s) |wv| 14.549,15.057 +14:14:35 mem step 450 loss 3.2113 (1454s) |wv| 15.438,16.016 +14:17:15 mem step 500 loss 2.9051 (1614s) |wv| 16.297,16.906 +14:17:16 EVAL mem step 500 | facts mc 0.5400 nll 6.0650 em 0.0790 +14:19:56 mem step 550 loss 3.9082 (1775s) |wv| 17.106,17.728 +14:22:36 mem step 600 loss 3.0027 (1935s) |wv| 17.876,18.521 +14:25:17 mem step 650 loss 3.0703 (2095s) |wv| 18.583,19.276 +14:27:57 mem step 700 loss 2.9264 (2256s) |wv| 19.285,20.003 +14:30:37 mem step 750 loss 3.5324 (2416s) |wv| 19.940,20.697 +14:30:38 EVAL mem step 750 | facts mc 0.5571 nll 6.0943 em 0.0695 +14:33:18 mem step 800 loss 4.0787 (2577s) |wv| 20.574,21.377 +14:35:58 mem step 850 loss 3.5568 (2737s) |wv| 21.167,22.024 +14:38:38 mem step 900 loss 2.9475 (2897s) |wv| 21.744,22.648 +14:41:18 mem step 950 loss 3.4908 (3057s) |wv| 22.311,23.252 +14:43:58 mem step 1000 loss 3.0032 (3217s) |wv| 22.894,23.842 +14:44:01 EVAL mem step 1000 | facts mc 0.5419 nll 6.1187 em 0.0810 long 2.9095 short 4.0137 +14:46:46 ctrl step 50 loss 3.3031 (156s) +14:49:23 ctrl step 100 loss 3.7692 (312s) +14:51:59 ctrl step 150 loss 3.1691 (468s) +14:54:35 ctrl step 200 loss 3.3060 (624s) +14:57:11 ctrl step 250 loss 2.9781 (781s) +14:57:12 EVAL ctrl step 250 | facts mc 0.5476 nll 6.0658 em 0.0724 +14:59:49 ctrl step 300 loss 2.9535 (939s) +15:02:25 ctrl step 350 loss 3.3512 (1095s) +15:05:02 ctrl step 400 loss 3.2523 (1251s) +15:07:38 ctrl step 450 loss 3.2085 (1407s) +15:10:14 ctrl step 500 loss 2.9019 (1563s) +15:10:15 EVAL ctrl step 500 | facts mc 0.5371 nll 6.1237 em 0.0752 +15:12:52 ctrl step 550 loss 3.9052 (1721s) +15:15:28 ctrl step 600 loss 3.0007 (1878s) +15:18:05 ctrl step 650 loss 3.0664 (2034s) +15:20:41 ctrl step 700 loss 2.9228 (2190s) +15:23:17 ctrl step 750 loss 3.5292 (2347s) +15:23:19 EVAL ctrl step 750 | facts mc 0.5495 nll 6.1099 em 0.0667 +15:25:55 ctrl step 800 loss 4.0768 (2505s) +15:28:31 ctrl step 850 loss 3.5518 (2661s) +15:31:08 ctrl step 900 loss 2.9456 (2817s) +15:33:44 ctrl step 950 loss 3.4854 (2973s) +15:36:20 ctrl step 1000 loss 3.0000 (3130s) +15:36:22 EVAL ctrl step 1000 | facts mc 0.5419 nll 6.1120 em 0.0790 long 2.9052 short 4.0058 +15:36:23 ENGRAM DONE diff --git a/pilots/facts.json b/pilots/facts.json new file mode 100644 index 0000000000000000000000000000000000000000..d95a6504005fd3e82044f0bf9e112b0e569f772a --- /dev/null +++ b/pilots/facts.json @@ -0,0 +1 @@ +[{"prop": "author", "subj": "Facebook", "obj": "Chris Hughes", "pop": 1551049, "ids": [8149, 1523, 5336, 1520, 13765, 41106, 72], "a": 4, "n": 2, "alts": ["eduardo saverin", "eduardo luiz saverin", "mark zuckerberg", "mark elliott zuckerberg", "zuck", "zuckerberg", "mark elliot zuckerberg", "dustin moskovitz", "chris hughes"], "dis": [[27971, 18407], [29041, 25321], [7524, 47649]]}, {"prop": "author", "subj": "James Bond", "obj": "Ian Fleming", "pop": 671954, "ids": [8266, 30188, 1523, 5336, 1520, 31235, 36120, 4764, 72], "a": 5, "n": 3, "alts": ["ian fleming", "ian lancaster fleming"], "dis": [[11238, 25346, 14452], [29151, 4154, 59321], [27863, 46407, 2968]]}, {"prop": "author", "subj": "LinkedIn", "obj": "Reid Hoffman", "pop": 278148, "ids": [44583, 1523, 5336, 1520, 44332, 46351, 2450, 72], "a": 4, "n": 3, "alts": ["reid hoffman", "reid garrett hoffman", "reid g. hoffman"], "dis": [[19562, 16433, 59332], [13765, 9525, 1543], [25747, 46233, 7662]]}, {"prop": "author", "subj": "Nineteen Eighty-Four", "obj": "George Orwell", "pop": 237868, "ids": [1515, 29973, 13727, 28185, 59339, 63, 42700, 1523, 5336, 1520, 9595, 3271, 9058, 72], "a": 10, "n": 3, "alts": ["george orwell", "eric blair", "p. s. burton", "eric arthur blair", "john freeman"], "dis": [[15816, 22328, 1640], [36027, 1445, 1371], [10255, 4172, 45896]]}, {"prop": "author", "subj": "One Piece", "obj": "Eiichiro Oda", "pop": 216773, "ids": [4502, 56681, 1523, 5336, 1520, 1529, 59325, 1620, 9355, 1549, 3903, 72], "a": 5, "n": 6, "alts": ["eiichiro oda", "eiichir\u014d oda", "eiichirou oda", "oda eiichiro", "oda eiichir\u014d", "oda eiichirou"], "dis": [[21341, 1491, 1375, 15263, 13163, 59393], [1493, 7587, 29976, 1413, 1693, 59326], [4255, 1487, 52747, 2664, 9429, 59326]]}, {"prop": "author", "subj": "Beloved", "obj": "Toni Morrison", "pop": 133630, "ids": [7276, 5383, 1523, 5336, 1520, 1408, 18602, 52402, 72], "a": 5, "n": 3, "alts": ["toni morrison", "chloe ardelia wofford", "chloe anthony wofford-morrison", "chloe anthony wofford", "morrison", "chloe anthony wofford morrison"], "dis": [[8266, 44891, 6329], [7524, 1491, 1579], [42897, 1493, 58625]]}, {"prop": "author", "subj": "Rent", "obj": "Jonathan Larson", "pop": 123783, "ids": [43715, 1523, 5336, 1520, 29151, 27976, 2162, 72], "a": 4, "n": 3, "alts": ["jonathan larson", "jonathan david larson"], "dis": [[11033, 13836, 3061], [31235, 36120, 4764], [39716, 25346, 14452]]}, {"prop": "author", "subj": "representational state transfer", "obj": "Roy Fielding", "pop": 112732, "ids": [3593, 1988, 2506, 6008, 1523, 5336, 1520, 22219, 10943, 1371, 72], "a": 7, "n": 3, "alts": ["roy fielding", "roy t. fielding", "roy thomas fielding"], "dis": [[8710, 1778, 31627], [14185, 5082, 1853], [7524, 1874, 1640]]}, {"prop": "author", "subj": "Wicked", "obj": "Winnie Holzman", "pop": 111439, "ids": [1495, 33919, 1523, 5336, 1520, 42328, 1778, 6875, 59398, 2450, 72], "a": 5, "n": 5, "alts": ["winnie holzman"], "dis": [[1494, 72, 10677, 15508, 1451], [9753, 11238, 3474, 1353, 1961], [23020, 1483, 1609, 72, 7868]]}, {"prop": "author", "subj": "GNU General Public License", "obj": "Richard Stallman", "pop": 106394, "ids": [9177, 5297, 5578, 2862, 1523, 5336, 1520, 11646, 1716, 1541, 2450, 72], "a": 7, "n": 4, "alts": ["free software foundation", "fsf", "free software foundation, inc.", "richard stallman", "richard m. stallman", "rms", "richard matthew stallman", "st. ignucius", "chief gnuisance"], "dis": [[35219, 5281, 27941, 1363], [4063, 42697, 47068, 26490], [15402, 1352, 13827, 59321]]}, {"prop": "author", "subj": "It", "obj": "Stephen King", "pop": 101430, "ids": [1826, 1523, 5336, 1520, 19378, 6132, 72], "a": 4, "n": 2, "alts": ["stephen king", "stephen edwin king", "richard bachman", "john swithen"], "dis": [[30433, 41106], [22219, 45215], [13765, 41106]]}, {"prop": "author", "subj": "Stack Overflow", "obj": "Joel Spolsky", "pop": 76948, "ids": [22701, 7351, 5014, 1523, 5336, 1520, 46319, 2870, 1404, 16872, 72], "a": 6, "n": 4, "alts": ["joel spolsky", "avram joel spolsky"], "dis": [[33686, 27467, 2655, 59321], [26750, 1358, 53653, 1378], [22774, 4384, 1788, 1636]]}, {"prop": "author", "subj": "New Testament", "obj": "Luke the Evangelist", "pop": 76533, "ids": [2404, 51686, 1523, 5336, 1520, 33348, 1358, 8860, 17300, 1466, 72], "a": 5, "n": 5, "alts": ["luke the evangelist", "saint luke", "luke", "lukas", "lucanus", "peter", "saint peter", "simon peter", "shimon", "simeon", "simon", "petrus", "peter the apostle", "pope peter", "saint peter the apostle", "matthew the apostle", "saint matthew", "matthew", "matthew the evangelist", "james, son of alphaeus", "james the less", "james the lesser", "paul", "paul the apostle", "paul of tarsus", "saint paul", "st. paul", "st paul", "saul", "saul of tarsus", "the apostle paul", "paulus", "saint paulus", "paulus von tarsus", "of tarsus, the apostle, saint paul", "the apostle, saint paul", "saint paul the apostle", "apostle paul"], "dis": [[28860, 1534, 22941, 18036, 11130], [5859, 1391, 1419, 1589, 1435], [1491, 2231, 1495, 34709, 2450]]}, {"prop": "author", "subj": "King James Version", "obj": "Lancelot Andrewes", "pop": 71185, "ids": [6132, 8266, 7128, 1523, 5336, 1520, 1494, 12926, 1411, 15054, 1359, 72], "a": 6, "n": 5, "alts": ["lancelot andrewes"], "dis": [[31264, 12150, 59324, 61599, 26546], [1419, 1994, 1405, 5281, 57416], [1515, 8077, 1494, 2174, 1356]]}, {"prop": "author", "subj": "Cats", "obj": "Andrew Lloyd Webber", "pop": 68737, "ids": [1419, 2790, 1523, 5336, 1520, 15054, 37787, 6987, 1640, 72], "a": 5, "n": 4, "alts": ["andrew lloyd webber", "baron lloyd-webber", "sir andrew lloyd webber", "andrew lloyd webber, baron lloyd-webber", "the rt hon. the lord lloyd-webber", "the lord lloyd-webber", "lord lloyd-webber"], "dis": [[19562, 9114, 44885, 1360], [50021, 1821, 2785, 17948], [1413, 5555, 1469, 5994]]}, {"prop": "author", "subj": "Thunderball", "obj": "Ian Fleming", "pop": 61249, "ids": [36322, 2626, 1523, 5336, 1520, 31235, 36120, 4764, 72], "a": 5, "n": 3, "alts": ["ian fleming", "ian lancaster fleming"], "dis": [[42060, 19045, 6730], [28143, 14972, 14603], [14943, 25950, 2233]]}, {"prop": "author", "subj": "The Stand", "obj": "Stephen King", "pop": 58670, "ids": [1507, 10017, 1523, 5336, 1520, 19378, 6132, 72], "a": 5, "n": 2, "alts": ["stephen king", "stephen edwin king", "richard bachman", "john swithen"], "dis": [[16067, 56915], [22976, 52402], [8351, 30692]]}, {"prop": "author", "subj": "The Book of Mormon", "obj": "Robert Lopez", "pop": 58655, "ids": [1507, 7631, 1379, 1445, 1994, 1350, 1523, 5336, 1520, 9812, 58825, 72], "a": 9, "n": 2, "alts": ["robert lopez", "robert joseph lopez", "matt stone", "matthew richard stone", "trey parker", "randolph severn parker", "randolph parker", "randolph severn parker iii"], "dis": [[13457, 10459], [58128, 27254], [15089, 51847]]}, {"prop": "author", "subj": "King Lear", "obj": "William Shakespeare", "pop": 56971, "ids": [6132, 10078, 1523, 5336, 1520, 9753, 43860, 72], "a": 5, "n": 2, "alts": ["william shakespeare", "shakespeare", "the bard", "the bard of avon", "william shakspere", "swan of avon", "bard of avon", "shakespere", "shakespear", "shakspeare", "shackspeare", "william shake\u2010\u017fpeare"], "dis": [[16067, 56915], [10804, 5547], [50961, 30692]]}, {"prop": "author", "subj": "Great Expectations", "obj": "Charles Dickens", "pop": 54249, "ids": [7566, 46486, 1641, 1523, 5336, 1520, 12583, 25346, 1643, 72], "a": 6, "n": 3, "alts": ["charles dickens", "dickens", "c.dickens", "charles john huffam dickens", "boz"], "dis": [[23794, 1681, 2611], [7078, 1681, 13667], [5583, 1469, 43457]]}, {"prop": "author", "subj": "Treasure Island", "obj": "Robert Louis Stevenson", "pop": 51623, "ids": [11748, 8448, 8991, 1523, 5336, 1520, 9812, 10674, 37325, 1350, 72], "a": 6, "n": 4, "alts": ["robert louis stevenson", "robert lewis balfour stevenson", "robert luis stivensoni", "shih-ti-wen-sheng", "stivenson", "robert loui sitivensin", "robert louis balfour stevenson", "robert lui stivenson", "rl stivenson", "rl stevenson", "rls"], "dis": [[13463, 13993, 1804, 11579], [29708, 8390, 21172, 1437], [27971, 52400, 1349, 6593]]}, {"prop": "author", "subj": "Rich Dad Poor Dad", "obj": "Robert Kiyosaki", "pop": 48457, "ids": [8213, 39550, 51719, 39550, 1523, 5336, 1520, 9812, 1681, 11976, 1472, 14770, 72], "a": 7, "n": 5, "alts": ["robert kiyosaki", "robert toru kiyosaki", "robert t. kiyosaki"], "dis": [[1527, 6466, 1469, 7839, 3972], [8959, 1408, 1898, 1363, 28094], [32392, 1363, 5281, 14653, 1604]]}, {"prop": "author", "subj": "The Lottery", "obj": "Shirley Jackson", "pop": 46478, "ids": [1507, 27733, 22613, 1523, 5336, 1520, 41181, 3389, 15670, 72], "a": 6, "n": 3, "alts": ["shirley jackson", "shirley hardie jackson"], "dis": [[13187, 10714, 2175], [7821, 16260, 50082], [32905, 43655, 20414]]}, {"prop": "author", "subj": "The Boys", "obj": "Darick Robertson", "pop": 44105, "ids": [1507, 27820, 1523, 5336, 1520, 15319, 1809, 50831, 72], "a": 5, "n": 3, "alts": ["darick robertson"], "dis": [[14943, 25950, 2233], [32740, 1874, 1360], [9753, 44879, 15119]]}, {"prop": "author", "subj": "Emma", "obj": "Jane Austen", "pop": 43382, "ids": [40813, 1523, 5336, 1520, 21872, 4060, 1356, 72], "a": 4, "n": 3, "alts": ["jane austen"], "dis": [[39716, 25346, 14452], [32932, 1479, 47192], [36336, 44891, 6329]]}, {"prop": "author", "subj": "Gospel of Matthew", "obj": "Matthew the Apostle", "pop": 42624, "ids": [51030, 1379, 26750, 1523, 5336, 1520, 26750, 1358, 53653, 1378, 72], "a": 6, "n": 4, "alts": ["matthew the apostle", "saint matthew", "matthew", "matthew the evangelist"], "dis": [[11003, 15818, 5517, 59323], [22774, 4384, 17627, 44664], [2899, 4554, 53673, 1579]]}, {"prop": "author", "subj": "Gin Tama", "obj": "Hideaki Sorachi", "pop": 40613, "ids": [1527, 1347, 1408, 4251, 1523, 5336, 1520, 46164, 14770, 36735, 29949, 72], "a": 7, "n": 4, "alts": ["hideaki sorachi"], "dis": [[42267, 2234, 29228, 2450], [18223, 1527, 1512, 59339], [13344, 2341, 26728, 3582]]}, {"prop": "author", "subj": "Hair", "obj": "Gerome Ragni", "pop": 40292, "ids": [35079, 1523, 5336, 1520, 22328, 1653, 1490, 3396, 59325, 72], "a": 4, "n": 5, "alts": ["galt macdermot", "arthur terence galt macdermot", "gerome ragni", "jerome bernard ragni", "james rado", "james alexander radomski"], "dis": [[36221, 28516, 1493, 1424, 3061], [4915, 1402, 1469, 1434, 2232], [8201, 1527, 1511, 2643, 59324]]}, {"prop": "author", "subj": "The Adventures of Tom Sawyer", "obj": "Mark Twain", "pop": 38388, "ids": [1507, 56104, 1379, 9549, 40373, 25706, 1523, 5336, 1520, 5378, 12239, 1485, 72], "a": 9, "n": 3, "alts": ["mark twain", "samuel langhorne clemens", "samuel l. clemens", "samuel clemens"], "dis": [[9595, 6905, 26470], [9595, 33863, 36635], [9595, 29228, 2450]]}, {"prop": "author", "subj": "The Walking Dead", "obj": "Robert Kirkman", "pop": 38079, "ids": [1507, 51081, 21981, 1523, 5336, 1520, 9812, 33778, 2450, 72], "a": 6, "n": 3, "alts": ["robert kirkman"], "dis": [[2419, 3418, 8080], [42060, 19045, 6730], [50377, 1452, 3287]]}, {"prop": "author", "subj": "Passing", "obj": "Nella Larsen", "pop": 36495, "ids": [11751, 1371, 1523, 5336, 1520, 1515, 8077, 1494, 2174, 1356, 72], "a": 5, "n": 5, "alts": ["nella larsen", "nella larsen imes"], "dis": [[5929, 14525, 52471, 1452, 5323], [24505, 1355, 59334, 7213, 59325], [1491, 11287, 1604, 17353, 1473]]}, {"prop": "author", "subj": "The Road", "obj": "Cormac McCarthy", "pop": 33713, "ids": [1507, 8289, 1523, 5336, 1520, 1419, 1994, 1405, 5281, 57416, 72], "a": 5, "n": 5, "alts": ["cormac mccarthy", "charles mccarthy"], "dis": [[1515, 3972, 17627, 44664, 59326], [1609, 1643, 1494, 46986, 1406], [1690, 72, 1419, 72, 46150]]}, {"prop": "author", "subj": "Into the Woods", "obj": "James Lapine", "pop": 33405, "ids": [50324, 1358, 37501, 1523, 5336, 1520, 8266, 50666, 1534, 72], "a": 6, "n": 3, "alts": ["stephen sondheim", "stephen joshua sondheim", "james lapine", "james elliot lapine"], "dis": [[4516, 59324, 26064], [5583, 1469, 43457], [32740, 5929, 2162]]}, {"prop": "author", "subj": "On the Road", "obj": "Jack Kerouac", "pop": 33066, "ids": [3154, 1358, 8289, 1523, 5336, 1520, 8710, 36239, 1375, 1405, 72], "a": 6, "n": 4, "alts": ["jack kerouac", "jean-louis lebris de k\u00e9rouac", "jean-louis lebris de kerouac", "kerouac", "jean-louis kerouac"], "dis": [[19378, 1413, 1810, 22901], [30791, 1527, 1424, 8235], [10399, 11010, 19761, 7323]]}, {"prop": "author", "subj": "The Book Thief", "obj": "Markus Zusak", "pop": 32842, "ids": [1507, 7631, 1632, 4920, 1523, 5336, 1520, 5378, 1406, 2572, 1406, 1622, 72], "a": 7, "n": 5, "alts": ["markus zusak"], "dis": [[29041, 46151, 3308, 1540, 1350], [4801, 9525, 1699, 1350, 47385], [4156, 5517, 3318, 1487, 2230]]}, {"prop": "author", "subj": "The Jungle", "obj": "Upton Sinclair", "pop": 30635, "ids": [1507, 1609, 39540, 1523, 5336, 1520, 1612, 16061, 20362, 55330, 72], "a": 6, "n": 4, "alts": ["upton sinclair", "upton beall sinclair", "clarke fitch", "frederick garrison", "arthur stirling", "upton sinclair jr."], "dis": [[41138, 1491, 2871, 2450], [21927, 28791, 38911, 26155], [37837, 45418, 1430, 2232]]}, {"prop": "author", "subj": "Utopia", "obj": "Thomas More", "pop": 30368, "ids": [1612, 3899, 1627, 1523, 5336, 1520, 10804, 5547, 72], "a": 6, "n": 2, "alts": ["thomas more", "sir thomas more", "saint thomas more", "thomas morus", "thomas, saint more", "thomas, sir more"], "dis": [[19488, 31947], [33133, 19201], [40813, 23347]]}, {"prop": "author", "subj": "Anything Goes", "obj": "Howard Lindsay", "pop": 30025, "ids": [52297, 4395, 1359, 1523, 5336, 1520, 24987, 1494, 55147, 72], "a": 6, "n": 3, "alts": ["howard lindsay", "herman nelke", "russel crouse", "p. g. wodehouse", "pelham grenville wodehouse", "sir pelham grenville wodehouse", "p.g. wodehouse", "guy bolton", "h. b. trevelyan", "guy reginald bolton"], "dis": [[50789, 25346, 1643], [45573, 51589, 9744], [15816, 1515, 4443]]}, {"prop": "author", "subj": "Through the Looking-Glass", "obj": "Lewis Carroll", "pop": 28871, "ids": [13401, 1358, 28036, 63, 59394, 14015, 1523, 5336, 1520, 19739, 48480, 72], "a": 9, "n": 2, "alts": ["lewis carroll", "charles dodgson", "lewis caroll", "lewis carroll dodgson", "charles lutwidge dodgson", "lewis carroll (charles lutwidge dodgson)", "rev. c. l. dodgson", "charles l. dodgson"], "dis": [[9114, 19739], [22976, 52402], [9812, 20169]]}, {"prop": "author", "subj": "The Way of Kings", "obj": "Brandon Sanderson", "pop": 28577, "ids": [1507, 13360, 1379, 26968, 1523, 5336, 1520, 43953, 11388, 2332, 72], "a": 7, "n": 3, "alts": ["brandon sanderson"], "dis": [[27971, 1491, 47638], [36221, 52570, 1677], [44332, 46351, 2450]]}, {"prop": "author", "subj": "Gospel of Luke", "obj": "Luke the Evangelist", "pop": 28461, "ids": [51030, 1379, 33348, 1523, 5336, 1520, 33348, 1358, 8860, 17300, 1466, 72], "a": 6, "n": 5, "alts": ["luke the evangelist", "saint luke", "luke", "lukas", "lucanus"], "dis": [[9753, 34683, 1493, 1424, 3061], [2861, 1611, 21401, 1486, 1391], [5102, 7380, 2270, 3469, 6591]]}, {"prop": "author", "subj": "Carousel", "obj": "Oscar Hammerstein II", "pop": 28292, "ids": [3684, 31418, 1523, 5336, 1520, 36197, 54423, 14976, 4401, 72], "a": 5, "n": 4, "alts": ["oscar hammerstein ii", "hammerstien", "oscar greely clendenning hammerstein ii", "oscar hammerstein", "oscar greeley clendenning hammerstein ii"], "dis": [[1469, 2500, 5162, 1349], [9812, 21712, 1452, 9725], [4801, 5082, 59327, 1861]]}, {"prop": "author", "subj": "The Secret", "obj": "Rhonda Byrne", "pop": 28156, "ids": [1507, 11681, 1523, 5336, 1520, 13807, 16099, 5082, 59327, 1861, 72], "a": 5, "n": 5, "alts": ["rhonda byrne"], "dis": [[4304, 1543, 15367, 2642, 2874], [11181, 1359, 5098, 1356, 1350], [2270, 4125, 15289, 8053, 1350]]}, {"prop": "author", "subj": "The Crow", "obj": "James O'Barr", "pop": 27285, "ids": [1507, 31831, 1523, 5336, 1520, 8266, 1549, 59361, 59378, 3418, 72], "a": 5, "n": 5, "alts": ["james o'barr"], "dis": [[9753, 46769, 1445, 5590, 1398], [19488, 1469, 1769, 1431, 2036], [1491, 2231, 1495, 34709, 2450]]}, {"prop": "author", "subj": "Moneyball", "obj": "Michael Lewis", "pop": 27186, "ids": [25530, 2626, 1523, 5336, 1520, 9114, 19739, 72], "a": 5, "n": 2, "alts": ["michael lewis", "michael m. lewis", "michael monroe lewis"], "dis": [[9753, 31219], [10804, 37501], [15089, 51847]]}, {"prop": "author", "subj": "Vagabond", "obj": "Takehiko Inoue", "pop": 26729, "ids": [1690, 1442, 1456, 1810, 1523, 5336, 1520, 13893, 59331, 26487, 1646, 1375, 59321, 72], "a": 7, "n": 6, "alts": ["takehiko inoue", "inoue takehiko"], "dis": [[1445, 1655, 1501, 3243, 8213, 3061], [25214, 1494, 72, 2861, 1422, 8573], [45220, 1543, 1493, 5262, 1484, 1822]]}, {"prop": "author", "subj": "The Trial", "obj": "Franz Kafka", "pop": 24829, "ids": [1507, 37204, 1523, 5336, 1520, 56696, 1681, 21300, 72], "a": 5, "n": 3, "alts": ["franz kafka", "franti\u0161ek kafka", "kafka"], "dis": [[31235, 36120, 4764], [39327, 15289, 4014], [55678, 55751, 32094]]}, {"prop": "author", "subj": "Persuasion", "obj": "Jane Austen", "pop": 24513, "ids": [17784, 59334, 9552, 1523, 5336, 1520, 21872, 4060, 1356, 72], "a": 6, "n": 3, "alts": ["jane austen"], "dis": [[5378, 12239, 1485], [1609, 14474, 29624], [7821, 16260, 50082]]}, {"prop": "author", "subj": "From Hell letter", "obj": "Jack the Ripper", "pop": 24403, "ids": [6051, 29775, 9072, 1523, 5336, 1520, 8710, 1358, 1490, 34954, 72], "a": 6, "n": 4, "alts": ["jack the ripper", "the whitechapel murderer", "whitechapel murderer", "leather apron"], "dis": [[50822, 1367, 1521, 15073], [7821, 2572, 1585, 1391], [43028, 1469, 1446, 2450]]}, {"prop": "author", "subj": "Company", "obj": "George Furth", "pop": 23448, "ids": [8465, 1523, 5336, 1520, 9595, 58283, 1440, 72], "a": 4, "n": 3, "alts": ["george furth"], "dis": [[35187, 1469, 39735], [7821, 2548, 20084], [19255, 8848, 1378]]}, {"prop": "author", "subj": null, "obj": "Peter David", "pop": 23061, "ids": [4090, 1523, 5336, 1520, 11238, 7524, 72], "a": 4, "n": 2, "alts": ["peter david", "peter allen david", "pad", "jacob ben gunter", "david peters"], "dis": [[9383, 59325], [27816, 56522], [9812, 20169]]}, {"prop": "author", "subj": "Christine", "obj": "Stephen King", "pop": 21978, "ids": [53715, 1523, 5336, 1520, 19378, 6132, 72], "a": 4, "n": 2, "alts": ["stephen king", "stephen edwin king", "richard bachman", "john swithen"], "dis": [[7524, 40753], [4801, 36389], [13457, 10459]]}, {"prop": "author", "subj": "Left Behind", "obj": "Jerry B. Jenkins", "pop": 21053, "ids": [26887, 3594, 32212, 1523, 5336, 1520, 32905, 1469, 72, 42279, 72], "a": 6, "n": 4, "alts": ["jerry b. jenkins", "jerry jenkins", "jerry bruce jenkins", "tim lahaye", "timothy lahaye", "timothy f. \"tim\" lahaye", "timothy francis lahaye", "tim francis lahaye"], "dis": [[7524, 7276, 1382, 3116], [4801, 25900, 2420, 3097], [27229, 5281, 5315, 2770]]}, {"prop": "author", "subj": "Holes", "obj": "Louis Sachar", "pop": 20940, "ids": [1491, 5305, 1523, 5336, 1520, 10674, 55917, 1370, 72], "a": 5, "n": 3, "alts": ["louis sachar"], "dis": [[9812, 17861, 7662], [4801, 18475, 7009], [55678, 55751, 32094]]}, {"prop": "author", "subj": "And did those feet in ancient time", "obj": "William Blake", "pop": 20338, "ids": [2738, 2579, 2608, 9499, 1377, 15626, 1817, 1523, 5336, 1520, 9753, 43556, 72], "a": 10, "n": 2, "alts": ["william blake", "w. blake", "uil'iam bleik", "blake"], "dis": [[4801, 36389], [50961, 30692], [29151, 21191]]}, {"prop": "author", "subj": "The Prophet", "obj": "Kahlil Gibran", "pop": 19860, "ids": [1507, 13977, 10151, 1523, 5336, 1520, 1681, 28040, 1394, 1527, 3260, 1363, 72], "a": 6, "n": 6, "alts": ["kahlil gibran", "gibran khalil gibran", "jubran khalil jubran"], "dis": [[39327, 19498, 36076, 1412, 11891, 1359], [1490, 1451, 26487, 30865, 2420, 27357], [2814, 2201, 59339, 1716, 9477, 1640]]}, {"prop": "author", "subj": "Elements", "obj": "Euclid", "pop": 19671, "ids": [48589, 1523, 5336, 1520, 1529, 31967, 72], "a": 4, "n": 2, "alts": ["euclid", "euclides", "euclid of alexandria"], "dis": [[8710, 37050], [15089, 51847], [8710, 44562]]}, {"prop": "author", "subj": "Midnight's Children", "obj": "Salman Rushdie", "pop": 19128, "ids": [13454, 9813, 59361, 59328, 14161, 1523, 5336, 1520, 8486, 2450, 44024, 29334, 72], "a": 8, "n": 4, "alts": ["salman rushdie", "rushdie", "joseph anton", "sir ahmed salman rushdie"], "dis": [[53752, 26202, 1738, 1687], [11646, 1716, 1541, 2450], [15089, 5281, 14037, 7304]]}, {"prop": "author", "subj": "The Neverending Story", "obj": "Michael Ende", "pop": 18610, "ids": [1507, 17607, 3551, 19650, 1523, 5336, 1520, 9114, 9946, 59321, 72], "a": 7, "n": 3, "alts": ["michael ende", "michael andreas helmuth ende"], "dis": [[22313, 1469, 4376], [9549, 28821, 3555], [4801, 18475, 7009]]}, {"prop": "author", "subj": "Gerald's Game", "obj": "Stephen King", "pop": 17670, "ids": [49530, 59361, 59328, 10164, 1523, 5336, 1520, 19378, 6132, 72], "a": 7, "n": 2, "alts": ["stephen king", "stephen edwin king", "richard bachman", "john swithen"], "dis": [[41266, 10082], [36140, 51092], [7524, 47649]]}, {"prop": "author", "subj": "Sweet Tooth", "obj": "Jeff Lemire", "pop": 17561, "ids": [30015, 3015, 1760, 1523, 5336, 1520, 13457, 27494, 1604, 72], "a": 6, "n": 3, "alts": ["jeff lemire"], "dis": [[37507, 43972, 16814], [7524, 2270, 14069], [19255, 8848, 1378]]}, {"prop": "author", "subj": "Symposium", "obj": "Plato", "pop": 16901, "ids": [47939, 1523, 5336, 1520, 2836, 8678, 72], "a": 4, "n": 2, "alts": ["plato", "platon", "aristocles"], "dis": [[13765, 41106], [18432, 12793], [25747, 47606]]}, {"prop": "author", "subj": "Urusei Yatsura", "obj": "Rumiko Takahashi", "pop": 16899, "ids": [21903, 1848, 59325, 1788, 2790, 9124, 1523, 5336, 1520, 1490, 1451, 26487, 30865, 2420, 27357, 72], "a": 9, "n": 6, "alts": ["rumiko takahashi", "takahashi rumiko"], "dis": [[19357, 1402, 1363, 3013, 31697, 59334], [25101, 1585, 6275, 1598, 18257, 1512], [1529, 59325, 1620, 9355, 1549, 3903]]}, {"prop": "author", "subj": "Our Town", "obj": "Thornton Wilder", "pop": 16434, "ids": [3804, 8962, 1523, 5336, 1520, 35689, 45126, 13337, 1349, 72], "a": 5, "n": 4, "alts": ["thornton wilder", "thornton niven wilder"], "dis": [[8710, 1358, 1490, 34954], [1646, 2683, 4516, 32801], [5929, 31677, 4429, 59336]]}, {"prop": "author", "subj": "Histories", "obj": "Herodotos", "pop": 16315, "ids": [32428, 3415, 1523, 5336, 1520, 6555, 1438, 1411, 1472, 72], "a": 5, "n": 4, "alts": ["herodotos", "herodotus of halicarnassus", "herodotus", "father of history"], "dis": [[6248, 13953, 13866, 4125], [14943, 27971, 17537, 1718], [31088, 12471, 1597, 59323]]}, {"prop": "author", "subj": "Scary Godmother", "obj": "Jill Thompson", "pop": 15195, "ids": [2732, 1629, 6274, 30070, 1523, 5336, 1520, 58128, 27254, 72], "a": 7, "n": 2, "alts": ["jill thompson"], "dis": [[18223, 30188], [9812, 34688], [30922, 58462]]}, {"prop": "author", "subj": "Spice and Wolf", "obj": "Isuna Hasekura", "pop": 14796, "ids": [2870, 1613, 1384, 26064, 1523, 5336, 1520, 2899, 12406, 1491, 1582, 59346, 9124, 72], "a": 7, "n": 6, "alts": ["isuna hasekura"], "dis": [[55738, 10665, 34668, 16519, 1391, 9266], [1529, 59325, 1620, 9355, 1549, 3903], [1408, 28558, 6745, 1445, 2079, 16423]]}, {"prop": "author", "subj": "White Noise", "obj": "Don DeLillo", "pop": 14735, "ids": [8080, 54919, 1523, 5336, 1520, 7063, 2748, 59376, 25324, 72], "a": 5, "n": 4, "alts": ["don delillo", "don delillo", "donald richard delillo"], "dis": [[26560, 35024, 38313, 9124], [21165, 9510, 1350, 22536], [36545, 1469, 1444, 1347]]}, {"prop": "author", "subj": "Bone", "obj": "Jeff Smith", "pop": 14437, "ids": [46181, 1523, 5336, 1520, 13457, 10459, 72], "a": 4, "n": 2, "alts": ["jeff smith"], "dis": [[43261, 14603], [21942, 8080], [25747, 47606]]}, {"prop": "author", "subj": "New Mutants", "obj": "Bill Sienkiewicz", "pop": 13903, "ids": [2404, 23643, 2417, 1523, 5336, 1520, 10518, 1413, 7607, 59346, 1800, 39024, 72], "a": 6, "n": 6, "alts": ["chris claremont", "bill sienkiewicz", "boleslav william felix robert sienkiewicz"], "dis": [[13893, 59331, 26487, 1646, 1375, 59321], [55247, 44218, 1568, 1632, 2355, 3389], [2572, 1622, 11458, 33224, 1378, 12904]]}, {"prop": "capital", "subj": "United States of America", "obj": "Washington, D.C.", "pop": 1629691, "ids": [1507, 8107, 1379, 3532, 4025, 1379, 6263, 1410, 8225, 59342, 1479, 72, 59357, 72, 72], "a": 8, "n": 6, "alts": ["washington, d.c.", "washington", "washington dc", "washington, dc", "dc", "d.c.", "district of columbia", "washington, district of columbia", "washington d.c.", "the district"], "dis": [[1491, 8678, 24558, 2664, 2518, 59339], [10541, 1440, 6444, 1408, 3198, 2435], [1408, 1719, 1511, 59338, 5818, 59326]]}, {"prop": "capital", "subj": "India", "obj": "New Delhi", "pop": 1301086, "ids": [1507, 8107, 1379, 5995, 1410, 2404, 20603, 72], "a": 5, "n": 2, "alts": ["new delhi", "new delhi district", "nayi dilli"], "dis": [[29099, 30595], [2387, 4493], [17441, 2874]]}, {"prop": "capital", "subj": "United Kingdom", "obj": "London", "pop": 1066107, "ids": [1507, 8107, 1379, 3532, 11690, 1410, 6098, 72], "a": 6, "n": 1, "alts": ["london", "london, uk", "london, united kingdom", "london, england", "modern babylon"], "dis": [[16016], [51987], [32438]]}, {"prop": "capital", "subj": "Taiwan", "obj": "Taipei", "pop": 978231, "ids": [1507, 8107, 1379, 24569, 1410, 18911, 5711, 59325, 72], "a": 5, "n": 3, "alts": ["taipei", "the city of azaleas", "taipei city", "taibei", "city of taipei"], "dis": [[1491, 3357, 1351], [1690, 4093, 20414], [8275, 3839, 11105]]}, {"prop": "capital", "subj": "Canada", "obj": "Ottawa", "pop": 816653, "ids": [1507, 8107, 1379, 6991, 1410, 36732, 72], "a": 5, "n": 1, "alts": ["ottawa", "ottawa (ontario)", "bytown"], "dis": [[37742], [53684], [32438]]}, {"prop": "capital", "subj": "California", "obj": "Sacramento", "pop": 766733, "ids": [1507, 8107, 1379, 6293, 1410, 48617, 72], "a": 5, "n": 1, "alts": ["sacramento", "sac-town", "sac", "the indomitable city", "city of trees", "the big tomato", "the capitol city", "sacramento, california", "camellia city", "river city"], "dis": [[58433], [37711], [52563]]}, {"prop": "capital", "subj": "People's Republic of China", "obj": "Beijing", "pop": 724517, "ids": [1507, 8107, 1379, 11038, 59361, 59328, 7121, 1379, 6213, 1410, 26823, 72], "a": 10, "n": 1, "alts": ["beijing", "peking", "beiping", "peiping", "yanjing", "zhongdu", "khanbaliq", "bj", "shun tian fu", "pekin", "beijing"], "dis": [[54793], [53823], [21513]]}, {"prop": "capital", "subj": "Australia", "obj": "Canberra", "pop": 713574, "ids": [1507, 8107, 1379, 6690, 1410, 3426, 1640, 1765, 72], "a": 5, "n": 3, "alts": ["canberra"], "dis": [[40596, 3269, 1443], [3052, 1370, 42332], [5216, 18055, 1568]]}, {"prop": "capital", "subj": "Japan", "obj": "Tokyo", "pop": 702414, "ids": [1507, 8107, 1379, 5701, 1410, 22174, 72], "a": 5, "n": 1, "alts": ["tokyo", "t\u014dky\u014d", "t\u00f4ky\u00f4", "tokyo-to", "tokyo metropolitan prefecture", "t\u014dky\u014d-to", "t\u00f4ky\u00f4-to", "tokyo metropolis", "tokio", "tokyo prefecture"], "dis": [[43338], [54966], [17223]]}, {"prop": "capital", "subj": "Afghanistan", "obj": "Kabul", "pop": 683050, "ids": [1507, 8107, 1379, 28793, 1410, 1681, 25237, 72], "a": 5, "n": 2, "alts": ["kabul", "cabool", "caubul", "kabol", "cabul"], "dis": [[1408, 3276], [47139, 24014], [7276, 8119]]}, {"prop": "capital", "subj": "Pakistan", "obj": "Islamabad", "pop": 574520, "ids": [1507, 8107, 1379, 16221, 1410, 15213, 19274, 72], "a": 5, "n": 2, "alts": ["islamabad", "isl\u0101m\u0101b\u0101d"], "dis": [[17441, 2874], [6222, 1765], [9595, 8962]]}, {"prop": "capital", "subj": "Germany", "obj": "Berlin", "pop": 556493, "ids": [1507, 8107, 1379, 8810, 1410, 17223, 72], "a": 5, "n": 1, "alts": ["berlin", "berlin, germany", "berlin (germany)", "de-be"], "dis": [[29311], [54793], [46081]]}, {"prop": "capital", "subj": "Russia", "obj": "Moscow", "pop": 526066, "ids": [1507, 8107, 1379, 11114, 1410, 22403, 72], "a": 5, "n": 1, "alts": ["moscow", "moskva", "moscow, russia", "moskva federal city, russia", "moscow, ussr", "moskva, russia", "city of moscow", "moscow, russian federation", "moscow, soviet union", "moscow, russian sfsr", "muscovite", "moscovite"], "dis": [[38991], [58433], [52219]]}, {"prop": "capital", "subj": "Philippines", "obj": "Manila", "pop": 522202, "ids": [1507, 8107, 1379, 21962, 1410, 51987, 72], "a": 5, "n": 1, "alts": ["manila", "city of manila"], "dis": [[42188], [15466], [41534]]}, {"prop": "capital", "subj": "New Zealand", "obj": "Wellington", "pop": 508758, "ids": [1507, 8107, 1379, 2404, 10204, 1410, 43338, 72], "a": 6, "n": 1, "alts": ["wellington", "te whanga-nui-a-tara", "wellington, new zealand", "wellington, nz", "wellington metropolitan area"], "dis": [[52219], [42188], [55457]]}, {"prop": "capital", "subj": "South Korea", "obj": "Seoul", "pop": 492668, "ids": [1507, 8107, 1379, 4168, 14169, 1410, 47018, 72], "a": 6, "n": 1, "alts": ["seoul", "seoul teukbyeolsi", "s\u014ful t'\u016dkpy\u014flsi", "wiryeseong", "namgyeong", "hanseong", "hanyang", "keij\u014d", "keijou", "gyeongseong"], "dis": [[15670], [24650], [44063]]}, {"prop": "capital", "subj": "France", "obj": "Paris", "pop": 486947, "ids": [1507, 8107, 1379, 8360, 1410, 11225, 72], "a": 5, "n": 1, "alts": ["paris", "city of light", "paris, france"], "dis": [[16991], [31755], [24650]]}, {"prop": "capital", "subj": "Uttar Pradesh", "obj": "Lucknow", "pop": 467741, "ids": [1507, 8107, 1379, 45342, 26656, 1410, 33756, 4053, 72], "a": 6, "n": 2, "alts": ["lucknow", "the city of nawabs", "the golden city of india", "constantinople of east", "shiraz-i-hind"], "dis": [[26730, 1350], [38061, 4599], [33127, 11875]]}, {"prop": "capital", "subj": "Turkey", "obj": "Ankara", "pop": 454969, "ids": [1507, 8107, 1379, 17238, 1410, 2426, 59346, 5818, 72], "a": 5, "n": 3, "alts": ["ankara", "ancyra", "angora", "ancara"], "dis": [[15602, 1529, 1791], [18911, 5711, 59325], [54352, 1382, 1382]]}, {"prop": "capital", "subj": "United Arab Emirates", "obj": "Abu Dhabi", "pop": 448395, "ids": [1507, 8107, 1379, 3532, 12329, 49434, 1410, 36170, 1479, 8683, 59325, 72], "a": 7, "n": 4, "alts": ["abu dhabi", "ab\u016b dhabi", "abudhabi"], "dis": [[1408, 1585, 1375, 59337], [1417, 1685, 35461, 4599], [15367, 1398, 2642, 2874]]}, {"prop": "capital", "subj": "England", "obj": "London", "pop": 445438, "ids": [1507, 8107, 1379, 7610, 1410, 6098, 72], "a": 5, "n": 1, "alts": ["london", "london, uk", "london, united kingdom", "london, england", "modern babylon"], "dis": [[35853], [35187], [46197]]}, {"prop": "capital", "subj": "Namibia", "obj": "Windhoek", "pop": 425312, "ids": [1507, 8107, 1379, 26043, 1599, 1627, 1410, 24828, 7214, 2469, 72], "a": 7, "n": 3, "alts": ["windhoek"], "dis": [[1420, 2405, 1382], [6312, 3318, 3972], [1527, 3690, 1639]]}, {"prop": "capital", "subj": "Netherlands", "obj": "The Hague", "pop": 416350, "ids": [1507, 8107, 1379, 18442, 1410, 1507, 1491, 3966, 72], "a": 5, "n": 3, "alts": ["the hague", "den haag", "'s-gravenhage", "haag", "amsterdam", "mokum", "amsterdam, nl", "amsterdam, netherlands", "a'dam"], "dis": [[14643, 59325, 5818], [25144, 6897, 2420], [1469, 1398, 35634]]}, {"prop": "capital", "subj": "Soviet Union", "obj": "Moscow", "pop": 411538, "ids": [1507, 8107, 1379, 19954, 9033, 1410, 22403, 72], "a": 6, "n": 1, "alts": ["moscow", "moskva", "moscow, russia", "moskva federal city, russia", "moscow, ussr", "moskva, russia", "city of moscow", "moscow, russian federation", "moscow, soviet union", "moscow, russian sfsr", "muscovite", "moscovite"], "dis": [[21513], [11225], [42038]]}, {"prop": "capital", "subj": "Bangladesh", "obj": "Dhaka", "pop": 407891, "ids": [1507, 8107, 1379, 26753, 1410, 33127, 11875, 72], "a": 5, "n": 2, "alts": ["dhaka", "dacca", "dhaka city"], "dis": [[24447, 1655], [15088, 6511], [1408, 3276]]}, {"prop": "capital", "subj": "South Africa", "obj": "Pretoria", "pop": 407517, "ids": [1507, 8107, 1379, 4168, 8470, 1410, 50903, 29094, 72], "a": 6, "n": 2, "alts": ["bloemfontein", "pretoria", "pretoria, s. africa", "pretoria, s africa", "pretoria, gauteng, south africa", "pretoria, gauteng", "pretoria, south africa", "epitoli", "cape town", "kaapstad", "ikapa", "kaap", "cape", "cape town, south africa", "mother city", "tavern of the seas", "west side"], "dis": [[8080, 49388], [1445, 54762], [1408, 3276]]}, {"prop": "capital", "subj": "Italy", "obj": "Rome", "pop": 403188, "ids": [1507, 8107, 1379, 11840, 1410, 20592, 72], "a": 5, "n": 1, "alts": ["rome", "the eternal city", "roma", "rome italy"], "dis": [[58433], [50320], [53684]]}, {"prop": "capital", "subj": "Texas", "obj": "Austin", "pop": 400631, "ids": [1507, 8107, 1379, 8253, 1410, 21513, 72], "a": 5, "n": 1, "alts": ["austin", "austin, texas"], "dis": [[19594], [27118], [20603]]}, {"prop": "capital", "subj": "Indonesia", "obj": "Jakarta", "pop": 383128, "ids": [1507, 8107, 1379, 21189, 1410, 36911, 31889, 72], "a": 5, "n": 2, "alts": ["jakarta", "special capital region of jakarta", "jkt", "jakarta, indonesia", "djakarta", "jacarta"], "dis": [[50903, 29094], [7276, 8119], [14227, 7821]]}, {"prop": "capital", "subj": "Israel", "obj": "Jerusalem", "pop": 382095, "ids": [1507, 8107, 1379, 13463, 1410, 34838, 72], "a": 5, "n": 1, "alts": ["jerusalem", "yerushalayim", "j'lem", "aelia capitolina", "al-quds", "jerusalem, middle east"], "dis": [[32253], [26823], [19594]]}, {"prop": "capital", "subj": "Spain", "obj": "Madrid", "pop": 377325, "ids": [1507, 8107, 1379, 13834, 1410, 27118, 72], "a": 5, "n": 1, "alts": ["madrid", "city of madrid", "madrid, spain"], "dis": [[46614], [41534], [54781]]}, {"prop": "capital", "subj": "Hawaii", "obj": "Honolulu", "pop": 376099, "ids": [1507, 8107, 1379, 27411, 1410, 14643, 1404, 24605, 72], "a": 5, "n": 3, "alts": ["honolulu", "honolulu, hawaii", "urban honolulu", "honolulu cdp"], "dis": [[1452, 10292, 2004], [1413, 1855, 1352], [2426, 59346, 5818]]}, {"prop": "capital", "subj": "Saudi Arabia", "obj": "Riyadh", "pop": 368537, "ids": [1507, 8107, 1379, 24996, 30781, 1410, 1490, 11976, 43058, 72], "a": 6, "n": 3, "alts": ["riyadh"], "dis": [[6620, 1967, 2672], [1490, 2160, 1794], [2270, 3834, 59339]]}, {"prop": "capital", "subj": "Mexico", "obj": "Mexico City", "pop": 367408, "ids": [1507, 8107, 1379, 11198, 1410, 11198, 4599, 72], "a": 5, "n": 2, "alts": ["mexico city", "mexico d.f.", "ciudad de m\u00e9xico", "city of mexico", "mexico city, mexico", "cdmx", "mexico"], "dis": [[4407, 2304], [36911, 31889], [19201, 24855]]}, {"prop": "capital", "subj": "New York", "obj": "Albany", "pop": 362757, "ids": [1507, 8107, 1379, 2404, 4436, 1410, 52219, 72], "a": 6, "n": 1, "alts": ["albany", "albany, new york"], "dis": [[20592], [22403], [32253]]}, {"prop": "capital", "subj": "Poland", "obj": "Warsaw", "pop": 355720, "ids": [1507, 8107, 1379, 15373, 1410, 46102, 72], "a": 5, "n": 1, "alts": ["warsaw", "warszawa", "varshe", "warschau", "varshava", "var\u0161ava", "vars\u00f3", "varsavia", "varsovie", "varsovia", "var\u0161uva"], "dis": [[12532], [15670], [27789]]}, {"prop": "capital", "subj": "Switzerland", "obj": "Bern", "pop": 354820, "ids": [1507, 8107, 1379, 22388, 1410, 15466, 72], "a": 5, "n": 1, "alts": ["bern", "berne", "city of bern", "berna"], "dis": [[58774], [31755], [52172]]}, {"prop": "capital", "subj": "Nigeria", "obj": "Abuja", "pop": 336913, "ids": [1507, 8107, 1379, 21588, 1410, 36170, 8529, 72], "a": 5, "n": 2, "alts": ["abuja", "abv", "abj", "fct", "federal capital territory"], "dis": [[15088, 6511], [21109, 9266], [43578, 6428]]}, {"prop": "capital", "subj": "Alaska", "obj": "Juneau", "pop": 332194, "ids": [1507, 8107, 1379, 26453, 1410, 5195, 2874, 72], "a": 5, "n": 2, "alts": ["juneau", "juneau, alaska", "juneau, ak"], "dis": [[1495, 18956], [50903, 29094], [35043, 15903]]}, {"prop": "capital", "subj": "Ukraine", "obj": "Kyiv", "pop": 331690, "ids": [1507, 8107, 1379, 20404, 1410, 23069, 1498, 72], "a": 5, "n": 2, "alts": ["kyiv", "kiev", "kyjiv", "kyyiv", "kyjiw", "kyiw", "ky\u00efv", "kief", "kieff", "kiew", "kioff", "kiof", "kiowia", "kiovia", "kiow", "kiou", "ky\u0457v"], "dis": [[59143, 1512], [29099, 30595], [58320, 4599]]}, {"prop": "capital", "subj": "Florida", "obj": "Tallahassee", "pop": 330089, "ids": [1507, 8107, 1379, 9745, 1410, 39097, 2420, 1382, 6357, 72], "a": 5, "n": 4, "alts": ["tallahassee", "tallahassee, florida"], "dis": [[7276, 59326, 56955, 1739], [4592, 59346, 1424, 1613], [24379, 1398, 2004, 59326]]}, {"prop": "capital", "subj": "European Union", "obj": "City of Brussels", "pop": 326446, "ids": [1507, 8107, 1379, 6823, 9033, 1410, 4599, 1379, 41107, 72], "a": 6, "n": 3, "alts": ["city of brussels", "brussels city", "brussels, belgium", "bruxelles", "brussel", "stad brussel", "ville de bruxelles", "02", "bxl", "bru", "brussels", "strasbourg", "stra\u00dfburg", "strassburg", "schdroosburi", "strossburi", "strossburig", "strosburi"], "dis": [[1515, 31591, 7481], [1681, 1591, 6511], [4599, 1379, 6098]]}, {"prop": "capital", "subj": "North Korea", "obj": "Pyongyang", "pop": 325845, "ids": [1507, 8107, 1379, 4514, 14169, 1410, 12721, 1635, 33433, 72], "a": 6, "n": 3, "alts": ["pyongyang", "capital of revolution", "ryugyong"], "dis": [[17280, 1489, 59339], [2738, 1470, 8021], [1408, 9508, 1350]]}, {"prop": "capital", "subj": "Scotland", "obj": "Edinburgh", "pop": 322949, "ids": [1507, 8107, 1379, 15280, 1410, 26835, 72], "a": 5, "n": 1, "alts": ["edinburgh", "edinburg", "edinburgh, scotland", "city of edinburgh", "edina", "modern athens"], "dis": [[22174], [22403], [32253]]}, {"prop": "capital", "subj": "Sri Lanka", "obj": "Colombo", "pop": 319836, "ids": [1507, 8107, 1379, 21550, 28402, 1410, 2758, 1390, 5381, 72], "a": 6, "n": 3, "alts": ["colombo", "sri jayawardenepura kotte", "kotte", "colombo"], "dis": [[3594, 1448, 1403], [3684, 1405, 1382], [4391, 1677, 1356]]}, {"prop": "capital", "subj": "Iran", "obj": "Tehran", "pop": 317291, "ids": [1507, 8107, 1379, 11877, 1410, 54781, 72], "a": 5, "n": 1, "alts": ["tehran", "teheran"], "dis": [[17877], [36732], [53684]]}, {"prop": "capital", "subj": "Massachusetts", "obj": "Boston", "pop": 317076, "ids": [1507, 8107, 1379, 15632, 1410, 12923, 72], "a": 5, "n": 1, "alts": ["boston", "beantown", "the cradle of liberty", "the hub", "the cradle of modern america", "the athens of america", "the walking city", "the hub of the universe", "bostonia", "boston, massachusetts", "boston, ma", "boston, mass.", "puritan city"], "dis": [[15670], [16016], [32253]]}, {"prop": "capital", "subj": "Pennsylvania", "obj": "Harrisburg", "pop": 314264, "ids": [1507, 8107, 1379, 13748, 1410, 22845, 9629, 72], "a": 5, "n": 2, "alts": ["harrisburg", "capital of pennsylvania", "harrisburg, pennsylvania"], "dis": [[6312, 27604], [6060, 8521], [44826, 1360]]}, {"prop": "capital", "subj": "Sweden", "obj": "Stockholm", "pop": 310896, "ids": [1507, 8107, 1379, 18555, 1410, 41252, 72], "a": 5, "n": 1, "alts": ["stockholm", "sthlm", "sthlm"], "dis": [[32892], [37742], [41534]]}, {"prop": "capital", "subj": "Cyprus", "obj": "Nicosia", "pop": 302485, "ids": [1507, 8107, 1379, 45279, 1410, 16500, 1472, 1627, 72], "a": 5, "n": 3, "alts": ["nicosia", "lefkosia"], "dis": [[14643, 1404, 24605], [1491, 2619, 1359], [2758, 1390, 5381]]}, {"prop": "capital", "subj": "Norway", "obj": "Oslo", "pop": 293715, "ids": [1507, 8107, 1379, 20489, 1410, 48299, 72], "a": 5, "n": 1, "alts": ["oslo", "christiania", "kristiania", "noosl", "oslo, norway"], "dis": [[20592], [46614], [30233]]}, {"prop": "capital", "subj": "Virginia", "obj": "Richmond", "pop": 289975, "ids": [1507, 8107, 1379, 11365, 1410, 32438, 72], "a": 5, "n": 1, "alts": ["richmond", "richmond, virginia", "richmond, va", "richmond city"], "dis": [[16016], [35187], [24938]]}, {"prop": "capital", "subj": "Malaysia", "obj": "Kuala Lumpur", "pop": 288342, "ids": [1507, 8107, 1379, 22633, 1410, 1681, 1864, 59324, 1494, 3293, 1407, 72], "a": 5, "n": 6, "alts": ["kuala lumpur", "kl", "federal territory of kuala lumpur"], "dis": [[10541, 1440, 6444, 1408, 3198, 2435], [1408, 1904, 1855, 1416, 1364, 9931], [1788, 21120, 59328, 1375, 59346, 1381]]}, {"prop": "capital", "subj": "Papua New Guinea", "obj": "Port Moresby", "pop": 288000, "ids": [1507, 8107, 1379, 24086, 7553, 2404, 34486, 1410, 6551, 1445, 3888, 2645, 72], "a": 8, "n": 4, "alts": ["port moresby"], "dis": [[5616, 25024, 3761, 1378], [3271, 1363, 36096, 1422], [2548, 1405, 2549, 59334]]}, {"prop": "capital", "subj": "Georgia", "obj": "Atlanta", "pop": 285574, "ids": [1507, 8107, 1379, 14185, 1410, 22276, 72], "a": 5, "n": 1, "alts": ["atlanta", "terminus", "atlanta, georgia", "the big peach", "hotlanta", "the atl", "the a", "atlanta, ga"], "dis": [[43338], [20069], [19594]]}, {"prop": "capital", "subj": "Denmark", "obj": "Copenhagen", "pop": 282017, "ids": [1507, 8107, 1379, 25446, 1410, 43662, 72], "a": 5, "n": 1, "alts": ["copenhagen", "k\u00f8benhavn", "cph", "k\u00f6penhamn", "k\u00f6\u00f6penhamina", "kopenhagen", "kobenhavn", "k\u00f6benhavn", "hafnia", "hafn", "havn"], "dis": [[52172], [46971], [46614]]}, {"prop": "capital", "subj": "Illinois", "obj": "Springfield", "pop": 279201, "ids": [1507, 8107, 1379, 14087, 1410, 54966, 72], "a": 5, "n": 1, "alts": ["springfield", "springfield, illinois", "springfield, il", "calhoun"], "dis": [[36732], [20603], [46614]]}, {"prop": "capital", "subj": "Romania", "obj": "Bucharest", "pop": 278539, "ids": [1507, 8107, 1379, 30878, 1410, 38688, 8661, 72], "a": 5, "n": 2, "alts": ["bucharest", "little paris", "paris of the east", "bucure\u0219ti", "bucure\u015fti", "bucuresti", "\u0431\u0443\u0446\u0443\u0440\u0435\u0219\u0442\u0438"], "dis": [[3317, 5126], [35043, 15903], [25543, 4599]]}, {"prop": "capital", "subj": "New Jersey", "obj": "Trenton", "pop": 276308, "ids": [1507, 8107, 1379, 2404, 13781, 1410, 11748, 45126, 72], "a": 6, "n": 2, "alts": ["trenton", "western new jersey", "trenton city, mercer county, new jersey", "trenton township, mercer county, new jersey", "trenton city, new jersey", "trenton township, new jersey", "trenton, new jersey"], "dis": [[1494, 5203], [14227, 42566], [17441, 2874]]}, {"prop": "capital", "subj": "Michigan", "obj": "Lansing", "pop": 275040, "ids": [1507, 8107, 1379, 13172, 1410, 1494, 1611, 1371, 72], "a": 5, "n": 3, "alts": ["lansing", "lansing, michigan"], "dis": [[46252, 19079, 8962], [30617, 4493, 1352], [1494, 2619, 24806]]}, {"prop": "capital", "subj": "Ohio", "obj": "Columbus", "pop": 272695, "ids": [1507, 8107, 1379, 12850, 1410, 37102, 72], "a": 5, "n": 1, "alts": ["columbus", "columbus, ohio", "the discovery city", "arch city", "indie art capital", "cowtown", "the biggest small town in america", "cbus"], "dis": [[24650], [54781], [52219]]}, {"prop": "capital", "subj": "North Carolina", "obj": "Raleigh", "pop": 272064, "ids": [1507, 8107, 1379, 4514, 11383, 1410, 1490, 2160, 1794, 72], "a": 6, "n": 3, "alts": ["raleigh", "raleigh, north carolina", "raleigh, nc"], "dis": [[4599, 1379, 41107], [25191, 1738, 1444], [1469, 1511, 59324]]}, {"prop": "capital", "subj": "Egypt", "obj": "Cairo", "pop": 271405, "ids": [1507, 8107, 1379, 16750, 1410, 58433, 72], "a": 5, "n": 1, "alts": ["cairo", "cairo, egypt"], "dis": [[16689], [17223], [27118]]}, {"prop": "capital", "subj": "Czech Republic", "obj": "Prague", "pop": 271047, "ids": [1507, 8107, 1379, 19626, 7121, 1410, 42188, 72], "a": 6, "n": 1, "alts": ["prague", "praha", "hlavn\u00ed m\u011bsto praha", "city of prague"], "dis": [[28571], [16491], [38991]]}, {"prop": "capital", "subj": "Oman", "obj": "Muscat", "pop": 265129, "ids": [1507, 8107, 1379, 1549, 2450, 1410, 6060, 8521, 72], "a": 6, "n": 2, "alts": ["muscat"], "dis": [[1413, 1373], [3013, 35330], [23069, 1498]]}, {"prop": "capital", "subj": "Vietnam", "obj": "Hanoi", "pop": 262637, "ids": [1507, 8107, 1379, 22927, 1410, 1491, 7380, 59325, 72], "a": 5, "n": 3, "alts": ["hanoi", "thang long", "dong do", "tonkin", "h\u00e0 n\u1ed9i", "ha noi"], "dis": [[15547, 39156, 42135], [12150, 1568, 2611], [2758, 1390, 5381]]}, {"prop": "capital", "subj": "Arizona", "obj": "Phoenix", "pop": 261032, "ids": [1507, 8107, 1379, 17900, 1410, 28571, 72], "a": 5, "n": 1, "alts": ["phoenix", "phoenix, arizona", "city of phoenix"], "dis": [[46081], [47018], [52541]]}, {"prop": "capital", "subj": "Colorado", "obj": "Denver", "pop": 259487, "ids": [1507, 8107, 1379, 15947, 1410, 27789, 72], "a": 5, "n": 1, "alts": ["denver", "city and county of denver", "denver, colorado", "mile high city", "queen city of the plains", "queen city of the west"], "dis": [[58433], [28571], [43399]]}, {"prop": "capital", "subj": "New Mexico", "obj": "Santa Fe", "pop": 258119, "ids": [1507, 8107, 1379, 2404, 11198, 1410, 15329, 4063, 72], "a": 6, "n": 2, "alts": ["santa fe", "santa fe, new mexico", "royal city of the holy faith of st. francis of assisi", "o gah po'geh", "santa fe, n. m.", "santa fe n. m.", "santa fe, n.m.", "santa fe n.m.", "sfnm"], "dis": [[2404, 20603], [21191, 8962], [8705, 1405]]}, {"prop": "capital", "subj": "Minnesota", "obj": "Saint Paul", "pop": 257506, "ids": [1507, 8107, 1379, 16661, 1410, 14227, 7821, 72], "a": 5, "n": 2, "alts": ["saint paul", "st. paul", "saint paul, minnesota", "st. paul, minnesota", "st paul", "st paul, minnesota"], "dis": [[10473, 1734], [44826, 1360], [6312, 46971]]}, {"prop": "capital of", "subj": "New York City", "obj": "United States of America", "pop": 718380, "ids": [2404, 4436, 4599, 1410, 1358, 8107, 1379, 3532, 4025, 1379, 6263, 72], "a": 7, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[35260, 2079, 46276, 4677], [3594, 11976, 1657, 11331], [14330, 7121, 1379, 28793]]}, {"prop": "capital of", "subj": "London", "obj": "United Kingdom", "pop": 523018, "ids": [6098, 1410, 1358, 8107, 1379, 3532, 11690, 72], "a": 5, "n": 2, "alts": ["kingdom of essex", "kingdom of the east saxons", "essex", "united kingdom", "\ud83c\uddec\ud83c\udde7", "uk", "united kingdom of great britain and northern ireland", "u.k.", "gbr", "gb", "u. k.", "u k", "g.b.", "g. b.", "g b", "great britain", "g.b.r.", "g b r", "britain", "great britain and northern ireland", "great britain", "united kingdom of great britain", "uk", "gb", "ukgb", "kingdom of great britain", "united kingdom of great britain and ireland", "united kingdom", "uk", "gb", "ukgbi", "great britain and ireland", "the united kingdom", "britain", "kingdom of england", "england", "england", "british north america", "british america", "commonwealth of england", "english republic", "commonwealth", "interregnum", "protectorate", "commonwealth of england, scotland and ireland"], "dis": [[7992, 18747], [4814, 1622], [4168, 8470]]}, {"prop": "capital of", "subj": "Singapore", "obj": "Straits Settlements", "pop": 492490, "ids": [17827, 1410, 1358, 8107, 1379, 35260, 2079, 46276, 4677, 72], "a": 5, "n": 4, "alts": ["straits settlements"], "dis": [[40800, 1411, 1473, 3889], [3594, 11976, 1657, 11331], [1515, 2420, 16764, 12256]]}, {"prop": "capital of", "subj": "Los Angeles", "obj": "Los Angeles County", "pop": 379485, "ids": [11209, 12089, 1410, 1358, 8107, 1379, 11209, 12089, 3889, 72], "a": 6, "n": 3, "alts": ["los angeles county", "los angeles county, california", "county of los angeles, california", "la county"], "dis": [[13549, 54226, 1627], [11690, 1379, 11840], [4599, 1379, 37742]]}, {"prop": "capital of", "subj": "Washington, D.C.", "obj": "United States of America", "pop": 283465, "ids": [8225, 59342, 1479, 72, 59357, 72, 1410, 1358, 8107, 1379, 3532, 4025, 1379, 6263, 72], "a": 10, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[5308, 42730, 1927, 51259], [1419, 10047, 8057, 6561], [6512, 1364, 6745, 12256]]}, {"prop": "capital of", "subj": "Paris", "obj": "France", "pop": 265592, "ids": [11225, 1410, 1358, 8107, 1379, 8360, 72], "a": 5, "n": 1, "alts": ["france", "fr", "fr", "r\u00e9publique fran\u00e7aise", "la france", "republic of france", "french republic", "fra", "the hexagon"], "dis": [[58320], [11840], [11198]]}, {"prop": "capital of", "subj": "Dubai", "obj": "Emirate of Dubai", "pop": 264470, "ids": [31755, 1410, 1358, 8107, 1379, 6503, 1448, 1446, 1379, 31755, 72], "a": 5, "n": 5, "alts": ["emirate of dubai", "dubai"], "dis": [[4599, 1379, 12836, 16103, 14398], [30466, 1695, 5262, 59393, 44970], [11690, 1379, 7302, 2112, 1627]]}, {"prop": "capital of", "subj": "Mumbai", "obj": "Maharashtra", "pop": 263928, "ids": [30263, 1410, 1358, 8107, 1379, 47896, 72], "a": 5, "n": 1, "alts": ["maharashtra"], "dis": [[21962], [25841], [11803]]}, {"prop": "capital of", "subj": "Delhi", "obj": "Lodhi dynasty", "pop": 219227, "ids": [20603, 1410, 1358, 8107, 1379, 1494, 1438, 6920, 44970, 72], "a": 5, "n": 4, "alts": ["tughlaq dynasty", "lodhi dynasty", "\u0932\u094b\u0926\u0940 \u0935\u0902\u0936", "\u0932\u094b\u0926\u0940 \u0918\u0930\u093e\u0923\u0947", "delhi sultanate", "sultanate of delhi", "sur empire", "khalji dynasty"], "dis": [[1493, 19599, 1783, 10966], [53836, 1514, 2199, 3889], [1788, 3300, 1431, 12256]]}, {"prop": "capital of", "subj": "Toronto", "obj": "Ontario", "pop": 208499, "ids": [16491, 1410, 1358, 8107, 1379, 16471, 72], "a": 5, "n": 1, "alts": ["ontario", "ont", "on"], "dis": [[32163], [23302], [24569]]}, {"prop": "capital of", "subj": "Tokyo", "obj": "Japan", "pop": 205487, "ids": [22174, 1410, 1358, 8107, 1379, 5701, 72], "a": 5, "n": 1, "alts": ["japan", "state of japan", "land of the rising sun", "nihon", "nippon", "jp", "nippon-koku", "nihon-koku", "ja", "jpn", "jp", "jap", "ja", "jap", "empire of japan", "japanese empire", "imperial japan", "great empire of japan", "great japanese empire", "\u03b1\u03c5\u03c4\u03bf\u03ba\u03c1\u03b1\u03c4\u03bf\u03c1\u03af\u03b1 \u03c4\u03b7\u03c2 \u03b9\u03b1\u03c0\u03c9\u03bd\u03af\u03b1\u03c2", "japan", "occupation of japan", "japan", "allied occupation of japan", "occupied japan"], "dis": [[25841], [25543], [17900]]}, {"prop": "capital of", "subj": "Rome", "obj": "Roman Empire", "pop": 193323, "ids": [20592, 1410, 1358, 8107, 1379, 9486, 18747, 72], "a": 5, "n": 2, "alts": ["lazio", "province of rome", "provincia di roma", "papal states", "papal state", "church state", "state of the church", "stato della chiesa", "status ecclesiae", "ecclesiastical states", "roman republic", "res publica", "rome", "roman empire", "imperium romanum", "romanum imperium", "imperium romanum", "senatus populusque romanus", "italy", "italia", "italian republic", "it", "\ud83c\uddee\ud83c\uddf9", "ita"], "dis": [[45100, 3889], [10459, 3889], [56697, 3889]]}, {"prop": "capital of", "subj": "Boston", "obj": "Suffolk County", "pop": 170080, "ids": [12923, 1410, 1358, 8107, 1379, 56697, 3889, 72], "a": 5, "n": 2, "alts": ["suffolk county", "suffolk county, massachusetts", "massachusetts", "bay state", "commonwealth of massachusetts", "ma", "mass.", "us-ma"], "dis": [[10875, 3889], [3532, 11690], [27816, 3889]]}, {"prop": "capital of", "subj": "Berlin", "obj": "Germany", "pop": 163000, "ids": [17223, 1410, 1358, 8107, 1379, 8810, 72], "a": 5, "n": 1, "alts": ["germany", "frg", "brd", "bundesrepublik deutschland", "federal republic of germany", "de", "deutschland", "ger", "br deutschland", "de"], "dis": [[17900], [6991], [25841]]}, {"prop": "capital of", "subj": "Philadelphia", "obj": "United States of America", "pop": 159313, "ids": [19485, 1410, 1358, 8107, 1379, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[1690, 1841, 28801, 29712], [9033, 1379, 4168, 8470], [1788, 1607, 2509, 4251]]}, {"prop": "capital of", "subj": "Mexico City", "obj": "Mexico", "pop": 156078, "ids": [11198, 4599, 1410, 1358, 8107, 1379, 11198, 72], "a": 6, "n": 1, "alts": ["mexico", "mx", "mx", "united mexican states", "mexican republic", "mex", "\ud83c\uddf2\ud83c\uddfd"], "dis": [[22633], [42177], [16689]]}, {"prop": "capital of", "subj": "Atlanta", "obj": "Fulton County", "pop": 155804, "ids": [22276, 1410, 1358, 8107, 1379, 1493, 1559, 1350, 3889, 72], "a": 5, "n": 4, "alts": ["fulton county", "fulton county, georgia"], "dis": [[50131, 59331, 11033, 1407], [1479, 32896, 1379, 38168], [1491, 5377, 7662, 29712]]}, {"prop": "capital of", "subj": "Stockholm", "obj": "Union between Sweden and Norway", "pop": 153845, "ids": [41252, 1410, 1358, 8107, 1379, 9033, 2131, 18555, 1384, 20489, 72], "a": 5, "n": 5, "alts": ["stockholm county", "stockholms l\u00e4n", "sweden", "kingdom of sweden", "se", "se", "swe", "\ud83c\uddf8\ud83c\uddea", "stockholm municipality", "city of stockholm", "stockholms stad", "union between sweden and norway", "svensk-norska unionen", "den svensk-norske union", "sweden-norway"], "dis": [[1479, 2797, 59326, 36883, 3340], [1494, 6612, 5301, 16872, 5987], [5616, 1360, 42062, 16872, 5987]]}, {"prop": "capital of", "subj": "Chennai", "obj": "Tamil Nadu", "pop": 150369, "ids": [47543, 1410, 1358, 8107, 1379, 27949, 47991, 72], "a": 5, "n": 2, "alts": ["tamil nadu"], "dis": [[7566, 15273], [4168, 11383], [22845, 3889]]}, {"prop": "capital of", "subj": "Seattle", "obj": "King County", "pop": 149948, "ids": [22047, 1410, 1358, 8107, 1379, 6132, 3889, 72], "a": 5, "n": 2, "alts": ["king county", "king county, washington"], "dis": [[3532, 11690], [40087, 8991], [27816, 3889]]}, {"prop": "capital of", "subj": "Hyderabad", "obj": "Andhra Pradesh", "pop": 146978, "ids": [54793, 1410, 1358, 8107, 1379, 2738, 46442, 26656, 72], "a": 5, "n": 3, "alts": ["andhra pradesh", "a.p", "telangana"], "dis": [[2572, 3502, 3337], [2426, 1483, 6093], [26313, 2069, 29712]]}, {"prop": "capital of", "subj": "Babylon", "obj": "Achaemenid Empire", "pop": 145084, "ids": [26267, 28068, 1410, 1358, 8107, 1379, 1420, 12251, 26278, 1402, 18747, 72], "a": 6, "n": 5, "alts": ["achaemenid empire", "first persian empire", "persian empire", "persia", "achaemenid persian empire", "persian achaemenid empire", "achaemenid persia", "neo-babylonian empire"], "dis": [[30466, 1695, 5262, 59393, 44970], [11690, 1379, 1527, 45756, 54721], [51574, 1349, 4251, 14407, 5987]]}, {"prop": "capital of", "subj": "Jerusalem", "obj": "Jerusalem District", "pop": 136734, "ids": [34838, 1410, 1358, 8107, 1379, 34838, 5987, 72], "a": 5, "n": 2, "alts": ["jerusalem district", "mandatory palestine", "british mandate for palestine", "palestine", "british palestine", "mandate palestine", "state of palestine", "\ud83c\uddf5\ud83c\uddf8", "palestinian state", "palestine", "palestinian national authority", "palestinian authority", "judah", "kingdom of judah", "kingdom of judea", "southern kingdom", "kingdom of judah", "kingdom of jerusalem", "latin kingdom of jerusalem", "jerusalem", "israel", "state of israel", "il", "\ud83c\uddee\ud83c\uddf1", "holy land", "the jewish state", "isr"], "dis": [[31974, 12256], [54226, 1627], [45785, 3889]]}, {"prop": "capital of", "subj": "Budapest", "obj": "Hungary", "pop": 134197, "ids": [53684, 1410, 1358, 8107, 1379, 31355, 72], "a": 5, "n": 1, "alts": ["kingdom of hungary", "hungary", "transleithania", "hungary", "hu", "\ud83c\udded\ud83c\uddfa", "hun"], "dis": [[6991], [32163], [8360]]}, {"prop": "capital of", "subj": "Sydney", "obj": "New South Wales", "pop": 131219, "ids": [16991, 1410, 1358, 8107, 1379, 2404, 4168, 14170, 72], "a": 5, "n": 3, "alts": ["new south wales", "nsw", "n.s.w."], "dis": [[31310, 1363, 8960], [13836, 1778, 3889], [56240, 2450, 3889]]}, {"prop": "capital of", "subj": "Barcelona", "obj": "Catalonia", "pop": 127753, "ids": [28511, 1410, 1358, 8107, 1379, 28225, 11953, 72], "a": 5, "n": 2, "alts": ["catalonia", "catalunya", "catalu\u00f1a", "catalonha", "autonomous community of catalonia", "barcelona province", "province of barcelona", "provincia de barcelona"], "dis": [[4814, 1622], [36732, 3889], [9837, 8991]]}, {"prop": "capital of", "subj": "Austin", "obj": "Texas", "pop": 127467, "ids": [21513, 1410, 1358, 8107, 1379, 8253, 72], "a": 5, "n": 1, "alts": ["texas", "tx", "lone star state", "texas, united states", "state of texas", "us-tx", "tex."], "dis": [[16689], [31355], [47896]]}, {"prop": "capital of", "subj": "Sacramento", "obj": "California", "pop": 127391, "ids": [48617, 1410, 1358, 8107, 1379, 6293, 72], "a": 5, "n": 1, "alts": ["california", "ca", "state of california", "the golden state", "calif."], "dis": [[23484], [25361], [22633]]}, {"prop": "capital of", "subj": "Beijing", "obj": "Qing dynasty", "pop": 125585, "ids": [26823, 1410, 1358, 8107, 1379, 3052, 1371, 44970, 72], "a": 5, "n": 3, "alts": ["people's republic of china", "china", "cn", "pr china", "prc", "cn", "chn", "\ud83c\udde8\ud83c\uddf3", "china pr", "mainland china", "beiyang government", "taiwan", "republic of china", "chunghwa minkwo", "chunghwa minkuo", "formosa", "nationalist china", "free china", "free area of the republic of china", "republic of china (taiwan)", "chinese taipei", "\ud83c\uddf9\ud83c\uddfc", "tw", "roc", "taiwan roc", "china, republic", "twn", "china (formerly known as)", "qing dynasty", "empire of the great qing", "great qing", "qing empire", "manchu empire", "manchu dynasty", "qing china", "manchu china", "ch'ing dynasty", "ch'ing empire", "ch'ing china"], "dis": [[21987, 1627, 5987], [20693, 5546, 15502], [4255, 1557, 21766]]}, {"prop": "capital of", "subj": "Vienna", "obj": "Austria", "pop": 122060, "ids": [32253, 1410, 1358, 8107, 1379, 25650, 72], "a": 5, "n": 1, "alts": ["holy roman empire", "holy roman empire of the german nation", "first reich", "reich", "austrian empire", "austria", "habsburg monarchy", "habsburg empire", "austria-hungary", "austro-hungarian empire", "dual monarchy", "austria hungary", "austria", "\u00f6sterreich", "republic of austria", "republik \u00f6sterreich", "at", "at", "aut", "\ud83c\udde6\ud83c\uddf9", "aut", "lower austria", "nieder\u00f6sterreich"], "dis": [[6293], [5701], [13463]]}, {"prop": "capital of", "subj": "Melbourne", "obj": "Victoria", "pop": 115941, "ids": [20069, 1410, 1358, 8107, 1379, 16689, 72], "a": 5, "n": 1, "alts": ["victoria", "vic", "victoria, australia", "state of victoria", "state of vic", "vic."], "dis": [[17360], [25650], [32163]]}, {"prop": "capital of", "subj": "Houston", "obj": "Harris County", "pop": 115437, "ids": [17422, 1410, 1358, 8107, 1379, 22845, 3889, 72], "a": 5, "n": 2, "alts": ["harris county", "harris county, texas"], "dis": [[16747, 20182], [27816, 3889], [4168, 35442]]}, {"prop": "capital of", "subj": "Prague", "obj": "Czech Republic", "pop": 112627, "ids": [42188, 1410, 1358, 8107, 1379, 19626, 7121, 72], "a": 5, "n": 2, "alts": ["central bohemian region", "st\u0159edo\u010desk\u00fd kraj", "central bohemia", "central bohemia region", "czech socialist republic", "czech sr", "\u010desk\u00e1 socialistick\u00e1 republika", "czech republic", "czr", "cz", "\u010desko", "\u010desk\u00e1 republika", "\u010dr", "cze", "cze", "czechia", "czechoslovakia", "czecho-slovakia", "cs", "tch", "\u010deskoslovensko", "\u010dsfr", "federation of czechoslovakia", "people's republic of czechoslovakia", "csfr", "bohemia", "kingdom of bohemia", "\u010desk\u00e9 kr\u00e1lovstv\u00ed", "k\u00f6nigreich b\u00f6hmen", "regnum bohemiae", "czech kingdom", "bohemia"], "dis": [[45084, 3889], [16814, 3889], [22941, 3889]]}, {"prop": "capital of", "subj": "Milan", "obj": "province of Milan", "pop": 110644, "ids": [31881, 1410, 1358, 8107, 1379, 16185, 1379, 31881, 72], "a": 5, "n": 3, "alts": ["lombardy", "lombardia", "province of milan", "milan province", "milano province", "provincia di milano", "western roman empire (395-476 ad)", "roman empire", "golden ambrosian republic", "repubblica ambrosiana", "aurea repubblica ambrosiana"], "dis": [[57982, 18397, 1446], [11690, 1379, 22485], [11690, 1379, 7610]]}, {"prop": "capital of", "subj": "Dublin", "obj": "Ireland", "pop": 109536, "ids": [24650, 1410, 1358, 8107, 1379, 11803, 72], "a": 5, "n": 1, "alts": ["kingdom of ireland", "ireland", "\u00e9ire", "ie", "irl", "republic of ireland", "hibernia", "ireland, republic of", "ie", "ireland", "\ud83c\uddee\ud83c\uddea", "eire", "southern ireland"], "dis": [[22633], [7610], [13463]]}, {"prop": "capital of", "subj": "Madrid", "obj": "First Spanish Republic", "pop": 108266, "ids": [27118, 1410, 1358, 8107, 1379, 6212, 11937, 7121, 72], "a": 5, "n": 3, "alts": ["spain", "spain", "espa\u00f1a", "kingdom of spain", "es", "esp", "first spanish republic", "community of madrid", "madrid autonomous community", "madrid province", "comunidad de madrid", "autonomous community of madrid"], "dis": [[26313, 2069, 29712], [1549, 1909, 15502], [4255, 1557, 21766]]}, {"prop": "capital of", "subj": "Portland", "obj": "Multnomah County", "pop": 103539, "ids": [24786, 1410, 1358, 8107, 1379, 8532, 19147, 2420, 3889, 72], "a": 5, "n": 4, "alts": ["multnomah county", "multnomah county, oregon"], "dis": [[11690, 1379, 40146, 1810], [12256, 1379, 20741, 59325], [1469, 5912, 1472, 3889]]}, {"prop": "capital of", "subj": "Manila", "obj": "Philippines", "pop": 102270, "ids": [51987, 1410, 1358, 8107, 1379, 21962, 72], "a": 5, "n": 1, "alts": ["commonwealth of the philippines", "philippine commonwealth", "mancomunidad filipina", "spanish east indies", "spanish empire in asia", "philippines", "republic of the philippines", "ph", "ph", "the philippines", "philippine islands", "phl", "rp", "\ud83c\uddf5\ud83c\udded", "phi", "pinas"], "dis": [[6991], [25361], [23484]]}, {"prop": "capital of", "subj": "Athens", "obj": "Kingdom of Greece", "pop": 101288, "ids": [38168, 1410, 1358, 8107, 1379, 11690, 1379, 22485, 72], "a": 5, "n": 3, "alts": ["4th of august regime", "metaxas regime", "kingdom of greece", "greece", "kingdom of greece", "duchy of athens", "greece", "hellenic republic", "hellas", "gr", "el", "\ud83c\uddec\ud83c\uddf7", "greek republic", "gre", "ellada", "greek"], "dis": [[2404, 4168, 14170], [12903, 54226, 1627], [21987, 1627, 5987]]}, {"prop": "capital of", "subj": "Phoenix", "obj": "Arizona", "pop": 99292, "ids": [28571, 1410, 1358, 8107, 1379, 17900, 72], "a": 5, "n": 1, "alts": ["arizona", "az", "the grand canyon state", "state of arizona", "grand canyon state", "ariz.", "us-az"], "dis": [[8360], [11803], [21962]]}, {"prop": "capital of", "subj": "Pittsburgh", "obj": "Allegheny County", "pop": 95868, "ids": [28623, 1410, 1358, 8107, 1379, 59035, 1668, 59339, 3889, 72], "a": 5, "n": 4, "alts": ["allegheny county", "allegheny county, pennsylvania", "allegheny county, pa"], "dis": [[27153, 59338, 13313, 59337], [1417, 4102, 14770, 33167], [41724, 22860, 59321, 29712]]}, {"prop": "capital of", "subj": "Karachi", "obj": "West Pakistan", "pop": 95357, "ids": [12065, 29949, 1410, 1358, 8107, 1379, 5308, 16221, 72], "a": 6, "n": 2, "alts": ["dominion of pakistan", "pakistan", "west pakistan", "pakistan"], "dis": [[32892, 3889], [41106, 3889], [4168, 22927]]}, {"prop": "capital of", "subj": "Dallas", "obj": "Dallas County", "pop": 94573, "ids": [22941, 1410, 1358, 8107, 1379, 22941, 3889, 72], "a": 5, "n": 2, "alts": ["dallas county", "dallas county, texas"], "dis": [[6231, 6263], [41252, 29712], [8225, 3889]]}, {"prop": "capital of", "subj": "Denver", "obj": "Colorado", "pop": 94282, "ids": [27789, 1410, 1358, 8107, 1379, 15947, 72], "a": 5, "n": 1, "alts": ["colorado", "co", "state of colorado", "colo."], "dis": [[6293], [6991], [47896]]}, {"prop": "capital of", "subj": "City of London", "obj": "Kingdom of England", "pop": 86327, "ids": [4599, 1379, 6098, 1410, 1358, 8107, 1379, 11690, 1379, 7610, 72], "a": 7, "n": 3, "alts": ["kingdom of england", "england"], "dis": [[23773, 2572, 41374], [1549, 1909, 15502], [24086, 1364, 4025]]}, {"prop": "capital of", "subj": "The Hague", "obj": "South Holland", "pop": 85518, "ids": [1507, 1491, 3966, 1410, 1358, 8107, 1379, 4168, 35442, 72], "a": 7, "n": 2, "alts": ["south holland", "zuid-holland"], "dis": [[24815, 41147], [7341, 12657], [56040, 1839]]}, {"prop": "capital of", "subj": "Cape Town", "obj": "Union of South Africa", "pop": 84859, "ids": [21191, 8962, 1410, 1358, 8107, 1379, 9033, 1379, 4168, 8470, 72], "a": 6, "n": 4, "alts": ["western cape", "western cape province", "province of the western cape", "province of western cape", "union of south africa", "dominion of south africa", "usa", "south africa under apartheid", "south africa", "republic of south africa", "rsa", "sa", "za", "\ud83c\uddff\ud83c\udde6", "zaf"], "dis": [[12256, 1379, 53228, 53788], [3594, 2874, 5218, 3889], [23115, 16830, 59321, 12256]]}, {"prop": "capital of", "subj": "Ottawa", "obj": "Canada", "pop": 84101, "ids": [36732, 1410, 1358, 8107, 1379, 6991, 72], "a": 5, "n": 1, "alts": ["canada", "dominion of canada", "british north america", "can", "ca", "ca", "can", "can."], "dis": [[25543], [11321], [8253]]}, {"prop": "capital of", "subj": "Rio de Janeiro", "obj": "Rio de Janeiro", "pop": 82513, "ids": [25416, 1564, 56030, 1410, 1358, 8107, 1379, 25416, 1564, 56030, 72], "a": 7, "n": 3, "alts": ["state of brazil", "brazil", "federative republic of brazil", "br", "bra", "br", "\ud83c\udde7\ud83c\uddf7", "rio de janeiro", "rio de janeiro state", "rj", "rio de janeiro (state)"], "dis": [[22240, 9486, 18747], [41107, 19346, 15502], [12887, 2546, 3889]]}, {"prop": "capital of", "subj": "Florence", "obj": "Province of Florence", "pop": 78903, "ids": [38991, 1410, 1358, 8107, 1379, 12256, 1379, 38991, 72], "a": 5, "n": 3, "alts": ["tuscany", "toscana", "republic of florence", "repubblica fiorentina", "florentine republic", "repubblica di firenze", "province of florence", "florence", "florence province", "firenze province", "provincia di firenze", "firenze", "kingdom of italy", "regno d\u2019italia", "italy", "it"], "dis": [[25416, 1564, 56030], [26313, 2069, 29712], [9406, 6823, 9033]]}, {"prop": "capital of", "subj": "Brussels Capital Region", "obj": "European Economic Community", "pop": 76663, "ids": [41107, 19346, 15502, 1410, 1358, 8107, 1379, 6823, 23425, 10966, 72], "a": 7, "n": 3, "alts": ["burgundian netherlands", "western european union", "european economic community", "eec", "common market", "european community", "spanish netherlands", "austrian netherlands"], "dis": [[26516, 5308, 15502], [1527, 42685, 3889], [34926, 3013, 1639]]}, {"prop": "capital of", "subj": "Turin", "obj": "Kingdom of Italy", "pop": 75884, "ids": [9332, 1347, 1410, 1358, 8107, 1379, 11690, 1379, 11840, 72], "a": 6, "n": 3, "alts": ["piedmont", "province of turin", "provincia di torino", "turin province", "torino province", "turin", "provincia di torino", "kingdom of sardinia", "piedmont-sardinia", "kingdom of sardinia-piedmont", "kingdom of sardinia (1720-1861)", "kingdom of piedmont-sardinia", "kingdom of italy", "regno d\u2019italia", "italy", "it"], "dis": [[44420, 1448, 1363], [24086, 1364, 4025], [6485, 59324, 29712]]}, {"prop": "capital of", "subj": "Kuala Lumpur", "obj": "Malaya", "pop": 74837, "ids": [1681, 1864, 59324, 1494, 3293, 1407, 1410, 1358, 8107, 1379, 7341, 12657, 72], "a": 10, "n": 2, "alts": ["federated malay states", "malaya", "federation of malaya", "japanese occupation of malaya", "malaysia", "my", "federation of malaysia", "malaysia federation", "mas", "mys", "\ud83c\uddf2\ud83c\uddfe"], "dis": [[7992, 18747], [32892, 3889], [22845, 3889]]}, {"prop": "capital of", "subj": "Alexandria", "obj": "Alexandria Governorate", "pop": 74284, "ids": [57982, 1410, 1358, 8107, 1379, 57982, 18397, 1446, 72], "a": 5, "n": 3, "alts": ["alexandria governorate", "al-iskandariyya", "governorate of alexandria"], "dis": [[49320, 32313, 15502], [32165, 2450, 44970], [1527, 42685, 3889]]}, {"prop": "capital of", "subj": "Ho Chi Minh City", "obj": "South Vietnam", "pop": 69663, "ids": [19618, 28445, 4119, 59331, 4599, 1410, 1358, 8107, 1379, 4168, 22927, 72], "a": 9, "n": 2, "alts": ["state of vietnam", "associated state of vietnam", "state of vietnam", "state of viet-nam", "independent state of viet nam", "vietnam", "vi\u1ec7t nam", "south vietnam", "republic of vietnam", "cochinchina", "cochin-china"], "dis": [[46963, 5987], [9486, 18747], [4168, 8470]]}, {"prop": "capital of", "subj": "Perth", "obj": "Western Australia", "pop": 63356, "ids": [37818, 1410, 1358, 8107, 1379, 9406, 6690, 72], "a": 5, "n": 2, "alts": ["western australia", "wa", "west australia", "west oz", "w.a.", "state of western australia", "state of wa", "w. aust."], "dis": [[33908, 18442], [39614, 1627], [31646, 7121]]}, {"prop": "capital of", "subj": "Victoria", "obj": "Colony of Vancouver Island", "pop": 63056, "ids": [16689, 1410, 1358, 8107, 1379, 2758, 4774, 1379, 25485, 8991, 72], "a": 5, "n": 5, "alts": ["colony of vancouver island", "island of vancouver and its dependencies", "vancouver island"], "dis": [[5246, 3232, 1627, 18397, 1446], [30466, 1695, 5262, 59393, 44970], [11690, 1379, 1413, 1578, 55190]]}, {"prop": "capital of", "subj": "Brisbane", "obj": "City of Brisbane", "pop": 62779, "ids": [37497, 1410, 1358, 8107, 1379, 4599, 1379, 37497, 72], "a": 5, "n": 3, "alts": ["queensland", "qld", "qld.", "city of brisbane", "brisbane city council", "brisbane city"], "dis": [[24086, 1364, 4025], [1445, 1579, 28322], [1413, 15246, 33167]]}, {"prop": "capital of", "subj": "Oxford", "obj": "Oxfordshire", "pop": 60484, "ids": [17137, 1410, 1358, 8107, 1379, 17137, 9176, 72], "a": 5, "n": 2, "alts": ["oxfordshire", "oxon.", "county of oxford"], "dis": [[4168, 11383], [44846, 3889], [16747, 20182]]}, {"prop": "capital of", "subj": "Adelaide", "obj": "South Australia", "pop": 55362, "ids": [37711, 1410, 1358, 8107, 1379, 4168, 6690, 72], "a": 5, "n": 2, "alts": ["south australia", "sa", "s.a.", "s. aust."], "dis": [[4168, 8470], [46963, 5987], [58774, 29712]]}, {"prop": "capital of", "subj": "Nice", "obj": "Alpes-Maritimes", "pop": 54615, "ids": [30449, 1410, 1358, 8107, 1379, 30413, 1359, 63, 14904, 1362, 2338, 72], "a": 5, "n": 6, "alts": ["alpes-maritimes", "aups maritims"], "dis": [[11715, 63, 9635, 1564, 11766, 1604], [12256, 1379, 1452, 1655, 1356, 1639], [3594, 2494, 2371, 1379, 3227, 4270]]}, {"prop": "capital of", "subj": "Providence", "obj": "Rhode Island", "pop": 52960, "ids": [52282, 1410, 1358, 8107, 1379, 40087, 8991, 72], "a": 5, "n": 2, "alts": ["rhode island", "ri", "state of rhode island", "rhode island, united states", "little rhodie", "state of rhode island and providence plantations", "road island", "little rhody", "the ocean state", "rhode island and providence plantations"], "dis": [[45100, 3889], [48480, 3889], [34838, 5987]]}, {"prop": "capital of", "subj": "Halifax", "obj": "Nova Scotia", "pop": 51082, "ids": [19501, 47940, 1410, 1358, 8107, 1379, 30342, 41658, 72], "a": 6, "n": 2, "alts": ["nova scotia", "province of nova scotia", "ns"], "dis": [[45100, 3889], [22941, 3889], [7341, 12657]]}, {"prop": "capital of", "subj": "Cardiff", "obj": "Wales", "pop": 50234, "ids": [52541, 1410, 1358, 8107, 1379, 14170, 72], "a": 5, "n": 1, "alts": ["wales", "wal", "wls", "cymru", "cambria"], "dis": [[11803], [24569], [32163]]}, {"prop": "capital of", "subj": "Quebec City", "obj": "Quebec", "pop": 48989, "ids": [25543, 4599, 1410, 1358, 8107, 1379, 25543, 72], "a": 6, "n": 1, "alts": ["quebec", "qu\u00e9bec", "qc", "province of quebec", "quebec, canada", "quebec province"], "dis": [[23484], [8360], [8810]]}, {"prop": "capital of", "subj": "Kabul", "obj": "Democratic Republic of Afghanistan", "pop": 48855, "ids": [1681, 25237, 1410, 1358, 8107, 1379, 14330, 7121, 1379, 28793, 72], "a": 6, "n": 4, "alts": ["kingdom of afghanistan", "afghan kingdom", "afghanistan", "islamic state of afghanistan", "kabul", "kabul province", "islamic emirate of afghanistan", "transitional islamic state of afghanistan", "afghan transitional administration", "democratic republic of afghanistan", "republic of afghanistan", "dr afghanistan", "kabul shahi", "afghanistan", "af", "\ud83c\udde6\ud83c\uddeb", "afg"], "dis": [[27951, 1379, 1358, 21962], [1479, 32896, 1379, 38168], [1493, 2789, 2272, 18703]]}, {"prop": "capital of", "subj": "Chester", "obj": "Kingdom of Gwynedd", "pop": 48732, "ids": [40985, 1410, 1358, 8107, 1379, 11690, 1379, 1527, 45756, 54721, 72], "a": 5, "n": 5, "alts": ["kingdom of gwynedd", "gwynedd"], "dis": [[18407, 19458, 4940, 1770, 7121], [4220, 2549, 28161, 5987, 29712], [1420, 12251, 26278, 1402, 18747]]}, {"prop": "capital of", "subj": "Riga", "obj": "Latvia", "pop": 44327, "ids": [1490, 15008, 1410, 1358, 8107, 1379, 16747, 20182, 72], "a": 6, "n": 2, "alts": ["latvia", "republic of latvia", "latvian republic", "lv", "latvija", "\ud83c\uddf1\ud83c\uddfb", "lat", "lva"], "dis": [[40087, 8991], [36732, 3889], [35126, 3889]]}, {"prop": "capital of", "subj": "Madison", "obj": "Wisconsin", "pop": 43253, "ids": [32892, 1410, 1358, 8107, 1379, 15943, 72], "a": 5, "n": 1, "alts": ["wisconsin", "wi", "wisconsin, united states", "state of wisconsin", "badger state", "wis.", "dane county", "dane county, wisconsin"], "dis": [[6293], [25543], [17900]]}, {"prop": "capital of", "subj": "Lima", "obj": "Peru", "pop": 43122, "ids": [10259, 59324, 1410, 1358, 8107, 1379, 32163, 72], "a": 6, "n": 1, "alts": ["viceroyalty of per\u00fa", "viceroyalty of peru", "peru", "pe", "republic of peru", "rep\u00fablica del per\u00fa", "\ud83c\uddf5\ud83c\uddea", "republica del peru"], "dis": [[15947], [8360], [18555]]}, {"prop": "composer", "subj": "Halloween", "obj": "John Carpenter", "pop": 933638, "ids": [1507, 4264, 1379, 28920, 1523, 13980, 1520, 4801, 51589, 9744, 72], "a": 7, "n": 3, "alts": ["john carpenter", "john howard carpenter"], "dis": [[4801, 40348, 3814], [7524, 21367, 23376], [14972, 3003, 12917]]}, {"prop": "composer", "subj": "Doctor Who", "obj": "Murray Gold", "pop": 289775, "ids": [1507, 4264, 1379, 23158, 14253, 1523, 13980, 1520, 30226, 8250, 72], "a": 8, "n": 2, "alts": ["murray gold", "murray jonathan gold"], "dis": [[47606, 18278], [4801, 27789], [18273, 59398]]}, {"prop": "composer", "subj": "Hamilton", "obj": "Lin-Manuel Miranda", "pop": 254893, "ids": [1507, 4264, 1379, 17877, 1523, 13980, 1520, 8351, 63, 13032, 5403, 15367, 10524, 72], "a": 7, "n": 6, "alts": ["lin-manuel miranda", "lmm"], "dis": [[6312, 1440, 8992, 33963, 25829, 1363], [1494, 1355, 1347, 11374, 2215, 1391], [1408, 1497, 1405, 2419, 1622, 1407]]}, {"prop": "composer", "subj": "Neon Genesis Evangelion", "obj": "Shir\u014d Sagisu", "pop": 160245, "ids": [1507, 4264, 1379, 5732, 1350, 9840, 6984, 8860, 17300, 1373, 1523, 13980, 1520, 41181, 61843, 59243, 1360, 59334, 72], "a": 13, "n": 5, "alts": ["shir\u014d sagisu", "shiro sagisu", "shirou sagisu", "shiroh sagisu", "shirow sagisu", "sagisu shir\u014d", "sagisu shiro", "sagisu shirou", "sagisu shiroh"], "dis": [[1826, 59324, 1364, 2419, 1407], [11181, 1359, 1469, 1557, 1388], [1469, 1898, 59325, 55762, 23106]]}, {"prop": "composer", "subj": "Lost", "obj": "Michael Giacchino", "pop": 154447, "ids": [1507, 4264, 1379, 35739, 1523, 13980, 1520, 9114, 16260, 1405, 1439, 3972, 72], "a": 7, "n": 5, "alts": ["michael giacchino", "michael george giacchino"], "dis": [[30088, 11731, 1681, 1607, 11287], [1495, 1557, 1681, 11467, 44591], [1681, 2494, 1778, 4119, 28680]]}, {"prop": "composer", "subj": "Saw", "obj": "Charlie Clouser", "pop": 149597, "ids": [1507, 4264, 1379, 40373, 1523, 13980, 1520, 30542, 2534, 1679, 1349, 72], "a": 7, "n": 4, "alts": ["charlie clouser", "charles alexander clouser"], "dis": [[1408, 72, 59355, 72], [9114, 1408, 3894, 9747], [2548, 5117, 4220, 1370]]}, {"prop": "composer", "subj": "Star Trek: The Next Generation", "obj": "Dennis McCarthy", "pop": 130869, "ids": [1507, 4264, 1379, 10058, 47411, 59358, 1507, 13421, 30981, 1523, 13980, 1520, 37826, 5281, 57416, 72], "a": 12, "n": 3, "alts": ["dennis mccarthy", "dennis mc carthy"], "dis": [[50124, 2518, 25706], [1445, 15000, 23689], [11646, 16162, 12995]]}, {"prop": "composer", "subj": "Rent", "obj": "Jonathan Larson", "pop": 123783, "ids": [1507, 4264, 1379, 43715, 1523, 13980, 1520, 29151, 27976, 2162, 72], "a": 7, "n": 3, "alts": ["jonathan larson", "jonathan david larson"], "dis": [[14972, 8690, 2450], [37340, 7672, 4830], [1527, 22477, 24846]]}, {"prop": "composer", "subj": "The Social Network", "obj": "Trent Reznor", "pop": 122369, "ids": [1507, 4264, 1379, 1507, 10847, 8516, 1523, 13980, 1520, 11748, 1513, 2518, 59398, 7856, 72], "a": 9, "n": 5, "alts": ["trent reznor", "michael trent reznor"], "dis": [[30433, 8235, 5281, 59366, 3770], [1826, 59324, 1364, 2419, 1407], [1495, 1557, 1681, 11467, 44591]]}, {"prop": "composer", "subj": "Toy Story", "obj": "Randy Newman", "pop": 120670, "ids": [1507, 4264, 1379, 23706, 19650, 1523, 13980, 1520, 53055, 2404, 2450, 72], "a": 8, "n": 3, "alts": ["randy newman", "randall stuart newman", "randall s. newman"], "dis": [[9114, 29228, 2450], [20987, 27599, 6158], [7524, 2419, 1604]]}, {"prop": "composer", "subj": "Troy", "obj": "James Horner", "pop": 118889, "ids": [1507, 4264, 1379, 50634, 1523, 13980, 1520, 8266, 13993, 2120, 72], "a": 7, "n": 3, "alts": ["james horner", "james roy horner"], "dis": [[22430, 2861, 1429], [14881, 1716, 22875], [42897, 18322, 8650]]}, {"prop": "composer", "subj": "Frozen", "obj": "Christophe Beck", "pop": 116766, "ids": [1507, 4264, 1379, 38487, 9456, 1523, 13980, 1520, 4516, 1464, 1351, 32422, 72], "a": 8, "n": 4, "alts": ["christophe beck", "jean-christophe beck", "robert lopez", "robert joseph lopez", "kristen anderson-lopez", "kristen anderson", "kristen jane anderson"], "dis": [[13360, 2272, 22240, 3548], [3317, 14097, 6485, 6593], [1469, 5533, 1599, 1364]]}, {"prop": "composer", "subj": "The Last of Us", "obj": "Gustavo Santaolalla", "pop": 116087, "ids": [1507, 4264, 1379, 1507, 12342, 1379, 8114, 1523, 13980, 1520, 45220, 35526, 15329, 1404, 39873, 72], "a": 10, "n": 5, "alts": ["gustavo santaolalla", "gustavo alfredo santaolalla"], "dis": [[1681, 2494, 1778, 4119, 28680], [11181, 1359, 1469, 1557, 1388], [9114, 16260, 1405, 1439, 3972]]}, {"prop": "composer", "subj": "There Will Be Blood", "obj": "Jonny Greenwood", "pop": 109027, "ids": [1507, 4264, 1379, 2972, 9878, 3594, 22818, 1523, 13980, 1520, 14972, 3362, 7868, 6593, 72], "a": 10, "n": 4, "alts": ["jonny greenwood", "jonathan richard guy greenwood"], "dis": [[36056, 42371, 1469, 2207], [14943, 1681, 1460, 4341], [15039, 5884, 59324, 37207]]}, {"prop": "composer", "subj": "Happy Birthday to You", "obj": "Mildred J. Hill", "pop": 102323, "ids": [1507, 4264, 1379, 23616, 38085, 1385, 2292, 1523, 13980, 1520, 1445, 1735, 2327, 1609, 72, 10082, 72], "a": 10, "n": 6, "alts": ["mildred j. hill", "mildred jane hill", "mildred hill", "patty hill", "patty smith hill"], "dis": [[16260, 1405, 29441, 19077, 1665, 7323], [29481, 1886, 35908, 2968, 1656, 24017], [4652, 2776, 1408, 1472, 6964, 7323]]}, {"prop": "composer", "subj": "The Village", "obj": "James Newton Howard", "pop": 101240, "ids": [1507, 4264, 1379, 1507, 19387, 1523, 13980, 1520, 8266, 21712, 24987, 72], "a": 8, "n": 3, "alts": ["james newton howard", "jim howard"], "dis": [[9137, 3560, 27688], [30087, 5162, 1640], [4801, 54596, 1350]]}, {"prop": "composer", "subj": "The Fog", "obj": "John Carpenter", "pop": 100414, "ids": [1507, 4264, 1379, 1507, 1493, 1514, 1523, 13980, 1520, 4801, 51589, 9744, 72], "a": 9, "n": 3, "alts": ["john carpenter", "john howard carpenter"], "dis": [[7524, 2404, 2450], [25346, 11374, 1382], [28653, 18030, 22312]]}, {"prop": "composer", "subj": "The Prestige", "obj": "David Julyan", "pop": 99894, "ids": [1507, 4264, 1379, 1507, 36568, 17899, 1523, 13980, 1520, 7524, 5387, 1363, 72], "a": 9, "n": 3, "alts": ["david julyan"], "dis": [[12583, 27599, 6158], [9027, 24672, 14603], [15089, 1681, 4050]]}, {"prop": "composer", "subj": "The Omen", "obj": "Jerry Goldsmith", "pop": 99370, "ids": [1507, 4264, 1379, 1507, 1549, 8858, 1523, 13980, 1520, 32905, 8250, 38352, 72], "a": 9, "n": 3, "alts": ["jerry goldsmith", "jerrald king goldsmith"], "dis": [[8439, 1469, 10705], [7524, 21367, 23376], [9812, 33549, 2546]]}, {"prop": "composer", "subj": "They Live", "obj": "John Carpenter", "pop": 90266, "ids": [1507, 4264, 1379, 3330, 13588, 1523, 13980, 1520, 4801, 51589, 9744, 72], "a": 8, "n": 3, "alts": ["john carpenter", "john howard carpenter"], "dis": [[12980, 14061, 12714], [24052, 9812, 11676], [10277, 7456, 14348]]}, {"prop": "composer", "subj": "Arrested Development", "obj": "David Schwartz", "pop": 84794, "ids": [1507, 4264, 1379, 2548, 4939, 1367, 9138, 1523, 13980, 1520, 7524, 36111, 24260, 72], "a": 10, "n": 3, "alts": ["david schwartz"], "dis": [[10255, 4172, 17223], [50319, 31382, 18165], [9137, 3560, 27688]]}, {"prop": "composer", "subj": "Rocky", "obj": "Bill Conti", "pop": 83985, "ids": [1507, 4264, 1379, 47864, 1523, 13980, 1520, 10518, 3711, 59325, 72], "a": 7, "n": 3, "alts": ["bill conti", "william conti"], "dis": [[11646, 16162, 12995], [15039, 6066, 7310], [19255, 23757, 34398]]}, {"prop": "composer", "subj": "The Fly", "obj": "Howard Shore", "pop": 82626, "ids": [1507, 4264, 1379, 1507, 26728, 1523, 13980, 1520, 24987, 39901, 72], "a": 8, "n": 2, "alts": ["howard shore", "howard leslie shore"], "dis": [[45973, 34333], [4801, 46698], [9114, 10726]]}, {"prop": "composer", "subj": "Power", "obj": "50 Cent", "pop": 81738, "ids": [1507, 4264, 1379, 8878, 1523, 13980, 1520, 59320, 59367, 59344, 3444, 72], "a": 7, "n": 4, "alts": ["50 cent", "curtis jackson", "curtis james jackson", "curtis james jackson iii"], "dis": [[1445, 15279, 1419, 50173], [23794, 43727, 1684, 13953], [1681, 2186, 59321, 5308]]}, {"prop": "composer", "subj": "True Blood", "obj": "Nathan Barr", "pop": 81457, "ids": [1507, 4264, 1379, 5619, 22818, 1523, 13980, 1520, 45973, 34333, 72], "a": 8, "n": 2, "alts": ["nathan barr"], "dis": [[48382, 57701], [7821, 14603], [9114, 10726]]}, {"prop": "composer", "subj": "Up", "obj": "Michael Giacchino", "pop": 80458, "ids": [1507, 4264, 1379, 5816, 1523, 13980, 1520, 9114, 16260, 1405, 1439, 3972, 72], "a": 7, "n": 5, "alts": ["michael giacchino", "michael george giacchino"], "dis": [[14953, 1493, 1514, 1391, 7662], [6859, 72, 1549, 1557, 59326], [25340, 1469, 72, 15616, 28885]]}, {"prop": "composer", "subj": "The Graduate", "obj": "Dave Grusin", "pop": 79381, "ids": [1507, 4264, 1379, 1507, 34712, 1523, 13980, 1520, 21758, 3509, 1406, 1347, 72], "a": 8, "n": 4, "alts": ["dave grusin", "robert david grusin"], "dis": [[27128, 1788, 1465, 47483], [4450, 72, 1479, 1352], [27229, 13993, 1986, 2645]]}, {"prop": "composer", "subj": "Inside Out", "obj": "Michael Giacchino", "pop": 74976, "ids": [1507, 4264, 1379, 33044, 6249, 1523, 13980, 1520, 9114, 16260, 1405, 1439, 3972, 72], "a": 8, "n": 5, "alts": ["michael giacchino", "michael george giacchino"], "dis": [[6859, 72, 1549, 1557, 59326], [34455, 12920, 1408, 1656, 59325], [11181, 1359, 1469, 1557, 1388]]}, {"prop": "composer", "subj": "Cloud Atlas", "obj": "Tom Tykwer", "pop": 74050, "ids": [1507, 4264, 1379, 14746, 46093, 1523, 13980, 1520, 9549, 22345, 59346, 2598, 72], "a": 8, "n": 4, "alts": ["tom tykwer", "reinhold heil", "reinhold e. heil"], "dis": [[1690, 51393, 6512, 4774], [12583, 1527, 13449, 1438], [30433, 8235, 20362, 55330]]}, {"prop": "composer", "subj": "Unbreakable", "obj": "James Newton Howard", "pop": 70121, "ids": [1507, 4264, 1379, 2023, 6733, 1597, 1523, 13980, 1520, 8266, 21712, 24987, 72], "a": 9, "n": 3, "alts": ["james newton howard", "jim howard"], "dis": [[32905, 10943, 1371], [14972, 59325, 32174], [32905, 19596, 7721]]}, {"prop": "composer", "subj": "The O.C.", "obj": "Christopher Tyng", "pop": 69564, "ids": [1507, 4264, 1379, 1507, 1549, 72, 59357, 72, 1523, 13980, 1520, 27971, 22345, 1853, 72], "a": 11, "n": 3, "alts": ["christopher tyng"], "dis": [[25214, 1690, 8077], [4801, 1419, 1533], [15039, 6066, 7310]]}, {"prop": "composer", "subj": "Cats", "obj": "Andrew Lloyd Webber", "pop": 68737, "ids": [1507, 4264, 1379, 1419, 2790, 1523, 13980, 1520, 15054, 37787, 6987, 1640, 72], "a": 8, "n": 4, "alts": ["andrew lloyd webber", "baron lloyd-webber", "sir andrew lloyd webber", "andrew lloyd webber, baron lloyd-webber", "the rt hon. the lord lloyd-webber", "the lord lloyd-webber", "lord lloyd-webber"], "dis": [[25191, 1763, 4498, 54083], [1413, 72, 1632, 20365], [27128, 1788, 1465, 47483]]}, {"prop": "composer", "subj": "The Room", "obj": "Mladen Milicevic", "pop": 64610, "ids": [1507, 4264, 1379, 1507, 15980, 1523, 13980, 1520, 1445, 19835, 1356, 10564, 1613, 27521, 72], "a": 8, "n": 6, "alts": ["mladen milicevic"], "dis": [[43190, 1494, 59392, 3300, 59338, 2261], [4652, 2776, 1408, 1472, 6964, 7323], [9826, 3362, 1445, 72, 59366, 72]]}, {"prop": "composer", "subj": "Full House", "obj": "Jesse Frederick", "pop": 63441, "ids": [1507, 4264, 1379, 13597, 5562, 1523, 13980, 1520, 50319, 37340, 72], "a": 8, "n": 2, "alts": ["jesse frederick", "frederick james conway", "jesse frederick james conaway"], "dis": [[11646, 16745], [19488, 43953], [14972, 13827]]}, {"prop": "composer", "subj": "The Master", "obj": "Jonny Greenwood", "pop": 62076, "ids": [1507, 4264, 1379, 1507, 12419, 1523, 13980, 1520, 14972, 3362, 7868, 6593, 72], "a": 8, "n": 4, "alts": ["jonny greenwood", "jonathan richard guy greenwood"], "dis": [[35888, 2292, 59335, 1611], [2419, 24750, 21078, 59339], [7821, 2870, 1407, 7147]]}, {"prop": "composer", "subj": "Signs", "obj": "James Newton Howard", "pop": 61883, "ids": [1507, 4264, 1379, 10347, 59328, 1523, 13980, 1520, 8266, 21712, 24987, 72], "a": 8, "n": 3, "alts": ["james newton howard", "jim howard"], "dis": [[59147, 53213, 41832], [7524, 2404, 2450], [10804, 2548, 1861]]}, {"prop": "composer", "subj": "We Are the World", "obj": "Michael Jackson", "pop": 61279, "ids": [1507, 4264, 1379, 1874, 6726, 1358, 4178, 1523, 13980, 1520, 9114, 15670, 72], "a": 10, "n": 2, "alts": ["michael jackson", "mj", "the king of pop", "michael joe jackson", "michael joseph jackson", "m.j.", "m. j.", "king of pop", "king of music", "the king"], "dis": [[8710, 14318], [33038, 58965], [36311, 59339]]}, {"prop": "composer", "subj": "The Host", "obj": "Lee Byung-woo", "pop": 55711, "ids": [1507, 4264, 1379, 1507, 21853, 1523, 13980, 1520, 12917, 5082, 3300, 63, 1824, 59326, 72], "a": 8, "n": 6, "alts": ["lee byung-woo", "lee byeong-u", "lee byeong-woo"], "dis": [[38741, 59325, 2078, 59346, 2382, 21007], [1490, 9723, 1788, 1405, 1375, 59340], [9595, 2861, 2255, 1374, 10875, 1391]]}, {"prop": "composer", "subj": "Gravity", "obj": "Steven Price", "pop": 52804, "ids": [1507, 4264, 1379, 3509, 11876, 1523, 13980, 1520, 28204, 16814, 72], "a": 8, "n": 2, "alts": ["steven price"], "dis": [[19711, 54487], [10277, 12489], [20864, 43354]]}, {"prop": "composer", "subj": "Gran Torino", "obj": "Kyle Eastwood", "pop": 51441, "ids": [1507, 4264, 1379, 26704, 9429, 3972, 1523, 13980, 1520, 43036, 6953, 6593, 72], "a": 9, "n": 3, "alts": ["michael stevens", "kyle eastwood", "kyle eastwood"], "dis": [[19378, 20989, 1509], [9114, 1527, 1512], [10277, 7456, 14348]]}, {"prop": "composer", "subj": "Super Metroid", "obj": "Kenji Yamamoto", "pop": 50651, "ids": [1507, 4264, 1379, 8883, 6975, 3705, 1523, 13980, 1520, 12344, 11140, 32467, 58326, 72], "a": 9, "n": 4, "alts": ["kenji yamamoto"], "dis": [[30922, 58283, 30815, 33715], [19488, 1413, 2251, 9094], [49395, 1413, 7193, 11458]]}, {"prop": "composer", "subj": "Leverage", "obj": "Joseph LoDuca", "pop": 48315, "ids": [1507, 4264, 1379, 1494, 2310, 1533, 1523, 13980, 1520, 15089, 11766, 59371, 45882, 72], "a": 9, "n": 4, "alts": ["joseph loduca"], "dis": [[4450, 72, 1479, 1352], [20178, 53869, 1351, 59325], [33491, 37312, 1364, 1804]]}, {"prop": "composer", "subj": "The Piano", "obj": "Michael Nyman", "pop": 47100, "ids": [1507, 4264, 1379, 1507, 53328, 1523, 13980, 1520, 9114, 1515, 32647, 72], "a": 8, "n": 3, "alts": ["michael nyman", "michael laurence nyman"], "dis": [[32905, 10943, 1371], [32905, 19596, 7721], [5279, 2232, 4801]]}, {"prop": "composer", "subj": "Annie", "obj": "Charles Strouse", "pop": 46479, "ids": [1507, 4264, 1379, 55617, 1523, 13980, 1520, 12583, 6066, 2441, 72], "a": 7, "n": 3, "alts": ["charles strouse"], "dis": [[22430, 12065, 4140], [50319, 31382, 18165], [9595, 1716, 4443]]}, {"prop": "composer", "subj": "Miami Vice", "obj": "Jan Hammer", "pop": 45805, "ids": [1507, 4264, 1379, 20847, 21626, 1523, 13980, 1520, 3868, 54423, 72], "a": 8, "n": 2, "alts": ["jan hammer", "jan hammer jr."], "dis": [[15039, 56756], [1788, 38500], [48382, 57701]]}, {"prop": "composer", "subj": "Crash", "obj": "Howard Shore", "pop": 42821, "ids": [1507, 4264, 1379, 5758, 2116, 1523, 13980, 1520, 24987, 39901, 72], "a": 8, "n": 2, "alts": ["howard shore", "howard leslie shore"], "dis": [[23978, 39339], [39451, 55104], [25340, 46330]]}, {"prop": "composer", "subj": "The Interview", "obj": "Henry Jackman", "pop": 42217, "ids": [1507, 4264, 1379, 1507, 42757, 1523, 13980, 1520, 14943, 8710, 2450, 72], "a": 8, "n": 3, "alts": ["henry jackman", "henry pryce jackman"], "dis": [[1445, 15000, 23689], [9753, 10518, 1798], [11748, 34749, 31611]]}, {"prop": "composer", "subj": "Holes", "obj": "Joel McNeely", "pop": 41620, "ids": [1507, 4264, 1379, 1491, 5305, 1523, 13980, 1520, 46319, 5281, 14385, 1769, 72], "a": 8, "n": 4, "alts": ["joel mcneely"], "dis": [[2534, 21947, 21861, 38670], [15054, 37787, 6987, 1640], [57086, 1407, 16949, 1363]]}, {"prop": "composer", "subj": "Metroid", "obj": "Hirokazu Tanaka", "pop": 40382, "ids": [1507, 4264, 1379, 6975, 3705, 1523, 13980, 1520, 57305, 59346, 2215, 59334, 29207, 11875, 72], "a": 8, "n": 6, "alts": ["hirokazu tanaka", "chip tanaka"], "dis": [[24395, 1460, 15263, 1420, 1503, 1347], [1490, 9723, 1788, 1405, 1375, 59340], [9553, 59325, 21550, 3327, 1382, 1422]]}, {"prop": "composer", "subj": "Hair", "obj": "Galt MacDermot", "pop": 40292, "ids": [1507, 4264, 1379, 35079, 1523, 13980, 1520, 1527, 4605, 6905, 59371, 1869, 1411, 72], "a": 7, "n": 6, "alts": ["galt macdermot", "arthur terence galt macdermot"], "dis": [[38741, 59339, 1681, 2847, 1607, 34629], [11238, 4638, 2978, 1347, 1424, 39024], [1445, 1735, 2327, 1609, 72, 10082]]}, {"prop": "composer", "subj": "The Firm", "obj": "Dave Grusin", "pop": 39704, "ids": [1507, 4264, 1379, 1507, 41976, 1523, 13980, 1520, 21758, 3509, 1406, 1347, 72], "a": 8, "n": 4, "alts": ["dave grusin", "robert david grusin"], "dis": [[44421, 13389, 19501, 1356], [9549, 22345, 59346, 2598], [57086, 1407, 16949, 1363]]}, {"prop": "composer", "subj": "Chicago", "obj": "John Kander", "pop": 39601, "ids": [1507, 4264, 1379, 11241, 1523, 13980, 1520, 4801, 1681, 9266, 72], "a": 7, "n": 3, "alts": ["john kander", "john harold kander"], "dis": [[54596, 2092, 33386], [18223, 29244, 13327], [10804, 13539, 30408]]}, {"prop": "composer", "subj": "Chariots of Fire", "obj": "Vangelis", "pop": 39450, "ids": [1507, 4264, 1379, 1793, 1920, 2974, 1379, 10192, 1523, 13980, 1520, 1690, 17300, 1360, 72], "a": 11, "n": 3, "alts": ["vangelis", "evangelos odysseas papathanassiou", "vangelis o. papathanassiou"], "dis": [[32905, 10943, 1371], [1507, 26534, 1888], [7524, 13836, 1778]]}, {"prop": "composer", "subj": "Into the Woods", "obj": "Stephen Sondheim", "pop": 39224, "ids": [1507, 4264, 1379, 50324, 1358, 37501, 1523, 13980, 1520, 19378, 1413, 1810, 22901, 72], "a": 9, "n": 4, "alts": ["stephen sondheim", "stephen joshua sondheim"], "dis": [[13738, 1493, 22424, 1360], [1490, 36002, 2797, 25747], [1408, 72, 59355, 72]]}, {"prop": "composer", "subj": "King Kong", "obj": "Max Steiner", "pop": 38013, "ids": [1507, 4264, 1379, 6132, 15672, 1523, 13980, 1520, 10277, 7456, 14348, 72], "a": 8, "n": 3, "alts": ["max steiner", "maximilian raoul steiner"], "dis": [[10674, 9924, 54486], [17856, 9321, 50344], [25214, 1690, 8077]]}, {"prop": "composer", "subj": "Happy Feet", "obj": "John Powell", "pop": 37705, "ids": [1507, 4264, 1379, 23616, 4063, 1388, 1523, 13980, 1520, 4801, 46698, 72], "a": 9, "n": 2, "alts": ["john powell"], "dis": [[34173, 22845], [49530, 29228], [19711, 54487]]}, {"prop": "composer", "subj": "Tron", "obj": "Wendy Carlos", "pop": 37495, "ids": [1507, 4264, 1379, 2278, 1350, 1523, 13980, 1520, 44218, 59339, 31264, 72], "a": 8, "n": 3, "alts": ["wendy carlos", "walter carlos"], "dis": [[20231, 1527, 15246], [7370, 1493, 12706], [50124, 2518, 25706]]}, {"prop": "composer", "subj": "WarGames", "obj": "Arthur B. Rubinstein", "pop": 37446, "ids": [1507, 4264, 1379, 5231, 52095, 1523, 13980, 1520, 25340, 1469, 72, 15616, 28885, 72], "a": 8, "n": 5, "alts": ["arthur b. rubinstein", "arthur benjamin rubinstein", "arthur rubinstein"], "dis": [[3318, 13417, 1491, 9664, 30560], [10665, 34668, 1527, 4940, 59398], [5929, 36782, 5758, 1360, 12904]]}, {"prop": "composer", "subj": "The Island", "obj": "Steve Jablonsky", "pop": 36020, "ids": [1507, 4264, 1379, 1507, 8991, 1523, 13980, 1520, 15816, 1609, 39156, 2072, 7625, 72], "a": 8, "n": 5, "alts": ["steve jablonsky"], "dis": [[58311, 1494, 1402, 59340, 1390], [41181, 61843, 59243, 1360, 59334], [1609, 2393, 1716, 59339, 1861]]}, {"prop": "composer", "subj": "The Raid", "obj": "Mike Shinoda", "pop": 35826, "ids": [1507, 4264, 1379, 1507, 47618, 1523, 13980, 1520, 13738, 47952, 16791, 72], "a": 8, "n": 3, "alts": ["mike shinoda", "michael kenji shinoda", "michael \"mike\" shinoda"], "dis": [[30542, 23405, 4140], [5174, 59325, 53036], [1788, 8992, 7156]]}, {"prop": "composer", "subj": "Imagine", "obj": "John Lennon", "pop": 34168, "ids": [1507, 4264, 1379, 46886, 1523, 13980, 1520, 4801, 54596, 1350, 72], "a": 7, "n": 3, "alts": ["john lennon", "john winston ono lennon", "john winston lennon", "lennon john", "john ono lennon"], "dis": [[4801, 40348, 3814], [57941, 36893, 1352], [10677, 13337, 44452]]}, {"prop": "composer", "subj": "Saw IV", "obj": "Charlie Clouser", "pop": 33595, "ids": [1507, 4264, 1379, 40373, 13461, 1523, 13980, 1520, 30542, 2534, 1679, 1349, 72], "a": 8, "n": 4, "alts": ["charlie clouser", "charles alexander clouser"], "dis": [[15089, 11766, 59371, 45882], [52361, 5101, 1356, 26912], [19378, 1413, 1810, 22901]]}, {"prop": "composer", "subj": "Glory", "obj": "James Horner", "pop": 33329, "ids": [1507, 4264, 1379, 8176, 1693, 1523, 13980, 1520, 8266, 13993, 2120, 72], "a": 8, "n": 3, "alts": ["james horner", "james roy horner"], "dis": [[7524, 5387, 1363], [25244, 12180, 5546], [10518, 1778, 34015]]}, {"prop": "composer", "subj": "Paprika", "obj": "Susumu Hirasawa", "pop": 30431, "ids": [1507, 4264, 1379, 24086, 51338, 1523, 13980, 1520, 15526, 1451, 59334, 45609, 1382, 13109, 72], "a": 8, "n": 6, "alts": ["susumu hirasawa", "hirasawa susumu"], "dis": [[1491, 1472, 1435, 1347, 28784, 22094], [1417, 11679, 59339, 6066, 1422, 4140], [9224, 59326, 15602, 1364, 4432, 59325]]}, {"prop": "composer", "subj": "31", "obj": "Rob Zombie", "pop": 30163, "ids": [1507, 4264, 1379, 59320, 59362, 59345, 1523, 13980, 1520, 9027, 2572, 41871, 72], "a": 9, "n": 3, "alts": ["rob zombie", "robert bartleh cummings", "rob straker"], "dis": [[11646, 16162, 12995], [11215, 59339, 37826], [24987, 23034, 1350]]}, {"prop": "composer", "subj": "To Die For", "obj": "Danny Elfman", "pop": 30036, "ids": [1507, 4264, 1379, 3015, 13479, 2522, 1523, 13980, 1520, 39421, 5279, 59337, 2450, 72], "a": 9, "n": 4, "alts": ["danny elfman", "daniel robert elfman"], "dis": [[3509, 3142, 13807, 1729], [49395, 1413, 7193, 11458], [9114, 1479, 5590, 18358]]}, {"prop": "composer", "subj": "Get Back", "obj": "Paul McCartney", "pop": 29775, "ids": [1507, 4264, 1379, 4979, 10203, 1523, 13980, 1520, 7821, 17537, 1522, 3814, 72], "a": 8, "n": 4, "alts": ["paul mccartney", "sir paul mccartney", "macca", "james paul mccartney"], "dis": [[52361, 5101, 1356, 26912], [2426, 14965, 2989, 41960], [25340, 15319, 59343, 1487]]}, {"prop": "composer", "subj": "The King and I", "obj": "Richard Rodgers", "pop": 29613, "ids": [1507, 4264, 1379, 1507, 6132, 1384, 1417, 1523, 13980, 1520, 11646, 16162, 12995, 72], "a": 10, "n": 3, "alts": ["richard rodgers", "richard charles rodgers"], "dis": [[32905, 13827, 1363], [26750, 13337, 1349], [18223, 29244, 13327]]}, {"prop": "composer", "subj": "The Sting", "obj": "Marvin Hamlisch", "pop": 29042, "ids": [1507, 4264, 1379, 1507, 1716, 1371, 1523, 13980, 1520, 3318, 13417, 1491, 9664, 30560, 72], "a": 9, "n": 5, "alts": ["marvin hamlisch", "marvin frederick hamlisch"], "dis": [[57305, 7374, 1681, 2371, 31111], [5929, 36782, 5758, 1360, 12904], [6859, 72, 1549, 1557, 59326]]}, {"prop": "composer", "subj": "The Other Woman", "obj": "Aaron Zigman", "pop": 28895, "ids": [1507, 4264, 1379, 1507, 8647, 29427, 1523, 13980, 1520, 35037, 2572, 1416, 2450, 72], "a": 9, "n": 4, "alts": ["aaron zigman"], "dis": [[29647, 1419, 14649, 1627], [21716, 2419, 4137, 1363], [4286, 3357, 59339, 13523]]}, {"prop": "country", "subj": "BBC", "obj": "United Kingdom", "pop": 246420, "ids": [18783, 1410, 4520, 1377, 3532, 11690, 72], "a": 4, "n": 2, "alts": ["united kingdom", "\ud83c\uddec\ud83c\udde7", "uk", "united kingdom of great britain and northern ireland", "u.k.", "gbr", "gb", "u. k.", "u k", "g.b.", "g. b.", "g b", "great britain", "g.b.r.", "g b r", "britain", "great britain and northern ireland"], "dis": [[10473, 7604], [7370, 1347], [52034, 18703]]}, {"prop": "country", "subj": "Snapchat", "obj": "United States of America", "pop": 221772, "ids": [35665, 14576, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[24086, 7553, 2404, 34486], [11690, 1379, 1358, 18442], [11690, 1379, 36408, 4774]]}, {"prop": "country", "subj": "Sinaloa Cartel", "obj": "Mexico", "pop": 126977, "ids": [1413, 2022, 13149, 18926, 1391, 1410, 4520, 1377, 11198, 72], "a": 8, "n": 1, "alts": ["mexico", "mx", "mx", "united mexican states", "mexican republic", "mex", "\ud83c\uddf2\ud83c\uddfd"], "dis": [[17238], [26753], [31842]]}, {"prop": "country", "subj": "Greek", "obj": "Turkey", "pop": 117605, "ids": [15066, 1410, 4520, 1377, 17238, 72], "a": 4, "n": 1, "alts": ["united kingdom", "\ud83c\uddec\ud83c\udde7", "uk", "united kingdom of great britain and northern ireland", "u.k.", "gbr", "gb", "u. k.", "u k", "g.b.", "g. b.", "g b", "great britain", "g.b.r.", "g b r", "britain", "great britain and northern ireland", "russia", "rossiya", "rossija", "ru", "ru", "rossijskaja federatsija", "russian federation", "rossiyskaya federatsiya", "rus", "rus", "rf", "canada", "dominion of canada", "british north america", "can", "ca", "ca", "can", "can.", "germany", "frg", "brd", "bundesrepublik deutschland", "federal republic of germany", "de", "deutschland", "ger", "br deutschland", "de", "ukraine", "ua", "ukr", "ua", "ukrainia", "\ud83c\uddfa\ud83c\udde6", "ukr.", "ukraina", "albania", "republic of albania", "al", "republika e shqip\u00ebris\u00eb", "shqip\u00ebris\u00eb", "\ud83c\udde6\ud83c\uddf1", "alb", "cyprus", "cy", "\ud83c\udde8\ud83c\uddfe", "greek cypriot state", "republic of cyprus", "greek administration of southern cyprus", "kazakhstan", "republic of kazakhstan", "kz", "qazaqstan", "kaz", "\ud83c\uddf0\ud83c\uddff", "abkhazia", "abkhasia", "abkazia", "abchasia", "abkhaziya", "united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica", "sweden", "kingdom of sweden", "se", "se", "swe", "\ud83c\uddf8\ud83c\uddea", "italy", "italia", "italian republic", "it", "\ud83c\uddee\ud83c\uddf9", "ita", "armenia", "republic of armenia", "\ud83c\udde6\ud83c\uddf2", "arm", "am", "australia", "commonwealth of australia", "au", "aus", "au", "british colony of australia", "\ud83c\udde6\ud83c\uddfa", "straya", "aussieland", "greece", "hellenic republic", "hellas", "gr", "el", "\ud83c\uddec\ud83c\uddf7", "greek republic", "gre", "ellada", "greek", "turkey", "republic of turkey", "\ud83c\uddf9\ud83c\uddf7", "tur", "tr", "egypt", "republic of egypt", "eg", "egy", "\ud83c\uddea\ud83c\uddec", "arab republic of egypt", "arab rep. egypt", "rep. egypt", "israel", "state of israel", "il", "\ud83c\uddee\ud83c\uddf1", "holy land", "the jewish state", "isr"], "dis": [[50547], [41211], [22485]]}, {"prop": "country", "subj": "Royal Navy", "obj": "United Kingdom", "pop": 106956, "ids": [10297, 15258, 1410, 4520, 1377, 3532, 11690, 72], "a": 5, "n": 2, "alts": ["united kingdom", "\ud83c\uddec\ud83c\udde7", "uk", "united kingdom of great britain and northern ireland", "u.k.", "gbr", "gb", "u. k.", "u k", "g.b.", "g. b.", "g b", "great britain", "g.b.r.", "g b r", "britain", "great britain and northern ireland"], "dis": [[40016, 12375], [57728, 10524], [44063, 1627]]}, {"prop": "country", "subj": "Fox News", "obj": "United States of America", "pop": 106131, "ids": [16032, 8355, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 36408, 4774], [11690, 1379, 1358, 18442], [1612, 32439, 2469, 10035]]}, {"prop": "country", "subj": "USS Constitution", "obj": "United States of America", "pop": 94586, "ids": [3542, 59352, 27764, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[24086, 7553, 2404, 34486], [11690, 1379, 1358, 18442], [1612, 32439, 2469, 10035]]}, {"prop": "country", "subj": "Ferrari", "obj": "Italy", "pop": 85860, "ids": [29620, 1920, 1410, 4520, 1377, 11840, 72], "a": 5, "n": 1, "alts": ["italy", "italia", "italian republic", "it", "\ud83c\uddee\ud83c\uddf9", "ita"], "dis": [[54634], [11114], [22485]]}, {"prop": "country", "subj": "Brown University", "obj": "United States of America", "pop": 85476, "ids": [11676, 3063, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[1612, 32439, 2469, 10035], [11690, 1379, 1358, 18442], [11690, 1379, 36408, 4774]]}, {"prop": "country", "subj": "Old English", "obj": "Wales", "pop": 82471, "ids": [11524, 5519, 1410, 4520, 1377, 14170, 72], "a": 5, "n": 1, "alts": ["england", "scotland", "alba", "scotland, united kingdom", "sct", "caledonia", "scot", "wales", "wal", "wls", "cymru", "cambria"], "dis": [[37313], [5995], [59101]]}, {"prop": "country", "subj": "Charlotte Hornets", "obj": "United States of America", "pop": 78030, "ids": [29995, 31611, 2379, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[1612, 32439, 2469, 10035], [11690, 1379, 36408, 4774], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Victoria", "obj": "Canada", "pop": 63056, "ids": [16689, 1410, 4520, 1377, 6991, 72], "a": 4, "n": 1, "alts": ["canada", "dominion of canada", "british north america", "can", "ca", "ca", "can", "can."], "dis": [[37313], [5701], [32163]]}, {"prop": "country", "subj": "Civic Democratic Party", "obj": "Czech Republic", "pop": 58738, "ids": [58434, 14330, 8301, 1410, 4520, 1377, 19626, 7121, 72], "a": 6, "n": 2, "alts": ["czech republic", "czr", "cz", "\u010desko", "\u010desk\u00e1 republika", "\u010dr", "cze", "cze", "czechia"], "dis": [[4168, 8470], [4514, 14169], [3532, 11690]]}, {"prop": "country", "subj": "Cambridge", "obj": "United Kingdom", "pop": 58473, "ids": [17814, 1410, 4520, 1377, 3532, 11690, 72], "a": 4, "n": 2, "alts": ["united kingdom", "\ud83c\uddec\ud83c\udde7", "uk", "united kingdom of great britain and northern ireland", "u.k.", "gbr", "gb", "u. k.", "u k", "g.b.", "g. b.", "g b", "great britain", "g.b.r.", "g b r", "britain", "great britain and northern ireland"], "dis": [[19626, 7121], [21550, 28402], [17280, 1511]]}, {"prop": "country", "subj": "Chrysler", "obj": "United States of America", "pop": 55586, "ids": [57344, 3061, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 1358, 18442], [11690, 1379, 36408, 4774], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Billboard", "obj": "United States of America", "pop": 51269, "ids": [31131, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 4, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 1358, 18442], [1612, 32439, 2469, 10035], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Malay", "obj": "Indonesia", "pop": 47480, "ids": [51666, 1410, 4520, 1377, 21189, 72], "a": 4, "n": 1, "alts": ["indonesia", "republic of indonesia", "id", "id", "ina", "idn"], "dis": [[49150], [31236], [37313]]}, {"prop": "country", "subj": "KLM", "obj": "Netherlands", "pop": 44656, "ids": [1681, 33079, 1410, 4520, 1377, 18442, 72], "a": 5, "n": 1, "alts": ["netherlands", "holland", "the netherlands", "nl", "ned", "nederland", "nl", "\ud83c\uddf3\ud83c\uddf1", "netherlands (after 1945)"], "dis": [[20404], [30878], [41211]]}, {"prop": "country", "subj": "South China Sea", "obj": "Malaysia", "pop": 43175, "ids": [4168, 6213, 14776, 1410, 4520, 1377, 22633, 72], "a": 6, "n": 1, "alts": ["people's republic of china", "china", "cn", "pr china", "prc", "cn", "chn", "\ud83c\udde8\ud83c\uddf3", "china pr", "mainland china", "cambodia", "kingdom of cambodia", "kh", "\ud83c\uddf0\ud83c\udded", "kampuchea", "camboya", "campuchia", "kamboja", "kambodzha", "malaysia", "my", "federation of malaysia", "malaysia federation", "mas", "mys", "\ud83c\uddf2\ud83c\uddfe", "taiwan", "republic of china", "chunghwa minkwo", "chunghwa minkuo", "formosa", "nationalist china", "free china", "free area of the republic of china", "republic of china (taiwan)", "chinese taipei", "\ud83c\uddf9\ud83c\uddfc", "tw", "roc", "taiwan roc", "china, republic", "twn", "china (formerly known as)", "vietnam", "socialist republic of vietnam", "vn", "vie", "\ud83c\uddfb\ud83c\uddf3", "vi\u1ec7t nam", "viet nam", "srv", "vnm", "brunei", "nation of brunei, the abode of peace", "bn", "brunei darussalam", "\ud83c\udde7\ud83c\uddf3", "brn", "bru", "negara brunei darussalam", "philippines", "republic of the philippines", "ph", "ph", "the philippines", "philippine islands", "phl", "rp", "\ud83c\uddf5\ud83c\udded", "phi", "pinas"], "dis": [[46849], [17238], [48727]]}, {"prop": "country", "subj": "Australian Labor Party", "obj": "Australia", "pop": 42258, "ids": [8853, 14542, 8301, 1410, 4520, 1377, 6690, 72], "a": 6, "n": 1, "alts": ["australia", "commonwealth of australia", "au", "aus", "au", "british colony of australia", "\ud83c\udde6\ud83c\uddfa", "straya", "aussieland"], "dis": [[18442], [59101], [28793]]}, {"prop": "country", "subj": "Epic Records", "obj": "United States of America", "pop": 41519, "ids": [54230, 11540, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 1358, 18442], [1612, 32439, 2469, 10035], [11690, 1379, 36408, 4774]]}, {"prop": "country", "subj": "Air Canada", "obj": "Canada", "pop": 38458, "ids": [6147, 6991, 1410, 4520, 1377, 6991, 72], "a": 5, "n": 1, "alts": ["canada", "dominion of canada", "british north america", "can", "ca", "ca", "can", "can."], "dis": [[48274], [31355], [36733]]}, {"prop": "country", "subj": "Devon", "obj": "United Kingdom", "pop": 38425, "ids": [44153, 1410, 4520, 1377, 3532, 11690, 72], "a": 4, "n": 2, "alts": ["united kingdom", "\ud83c\uddec\ud83c\udde7", "uk", "united kingdom of great britain and northern ireland", "u.k.", "gbr", "gb", "u. k.", "u k", "g.b.", "g. b.", "g b", "great britain", "g.b.r.", "g b r", "britain", "great britain and northern ireland"], "dis": [[38391, 25686], [40016, 12375], [30731, 18703]]}, {"prop": "country", "subj": "Laika", "obj": "United States of America", "pop": 37668, "ids": [5616, 15000, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 1358, 18442], [1612, 32439, 2469, 10035], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "US Open", "obj": "United States of America", "pop": 35087, "ids": [3542, 6404, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 1358, 18442], [11690, 1379, 36408, 4774], [1612, 32439, 2469, 10035]]}, {"prop": "country", "subj": "Radio Free Asia", "obj": "United States of America", "pop": 34691, "ids": [14180, 6717, 12160, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[24086, 7553, 2404, 34486], [1612, 32439, 2469, 10035], [11690, 1379, 36408, 4774]]}, {"prop": "country", "subj": "Bouvet Island", "obj": "Norway", "pop": 31555, "ids": [26664, 20315, 8991, 1410, 4520, 1377, 20489, 72], "a": 6, "n": 1, "alts": ["norway", "kingdom of norway", "norge", "norv\u00e8ge\u200f", "no", "nor", "no", "noreg", "norwegen\u200f", "\ud83c\uddf3\ud83c\uddf4"], "dis": [[11840], [21189], [32163]]}, {"prop": "country", "subj": "BT Group", "obj": "United Kingdom", "pop": 30636, "ids": [38057, 5548, 1410, 4520, 1377, 3532, 11690, 72], "a": 5, "n": 2, "alts": ["united kingdom", "\ud83c\uddec\ud83c\udde7", "uk", "united kingdom of great britain and northern ireland", "u.k.", "gbr", "gb", "u. k.", "u k", "g.b.", "g. b.", "g b", "great britain", "g.b.r.", "g b r", "britain", "great britain and northern ireland"], "dis": [[4168, 14169], [10473, 7604], [17280, 1511]]}, {"prop": "country", "subj": "New School", "obj": "United States of America", "pop": 27081, "ids": [2404, 4234, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 36408, 4774], [24086, 7553, 2404, 34486], [1612, 32439, 2469, 10035]]}, {"prop": "country", "subj": "British American Tobacco", "obj": "United Kingdom", "pop": 26492, "ids": [6231, 3295, 27599, 19438, 1410, 4520, 1377, 3532, 11690, 72], "a": 7, "n": 2, "alts": ["united kingdom", "\ud83c\uddec\ud83c\udde7", "uk", "united kingdom of great britain and northern ireland", "u.k.", "gbr", "gb", "u. k.", "u k", "g.b.", "g. b.", "g b", "great britain", "g.b.r.", "g b r", "britain", "great britain and northern ireland"], "dis": [[44063, 1627], [21550, 28402], [28324, 44041]]}, {"prop": "country", "subj": "Netherlands Antilles", "obj": "Kingdom of the Netherlands", "pop": 24207, "ids": [18442, 6512, 29681, 1410, 4520, 1377, 11690, 1379, 1358, 18442, 72], "a": 6, "n": 4, "alts": ["kingdom of the netherlands", "the netherlands", "netherlands", "nl", "\ud83c\uddf3\ud83c\uddf1"], "dis": [[24086, 7553, 2404, 34486], [3532, 4025, 1379, 6263], [1612, 32439, 2469, 10035]]}, {"prop": "country", "subj": "People's Party", "obj": "Spain", "pop": 23965, "ids": [11038, 59361, 59328, 8301, 1410, 4520, 1377, 13834, 72], "a": 7, "n": 1, "alts": ["spain", "espa\u00f1a", "kingdom of spain", "es", "esp"], "dis": [[58501], [46849], [49017]]}, {"prop": "country", "subj": "Chicago Fire FC", "obj": "United States of America", "pop": 22948, "ids": [11241, 10192, 13837, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[24086, 7553, 2404, 34486], [11690, 1379, 1358, 18442], [11690, 1379, 36408, 4774]]}, {"prop": "country", "subj": "Appalachian State University", "obj": "United States of America", "pop": 21981, "ids": [5216, 1364, 1684, 1770, 4200, 3063, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 9, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 1358, 18442], [24086, 7553, 2404, 34486], [11690, 1379, 36408, 4774]]}, {"prop": "country", "subj": "Western Hockey League", "obj": "Canada", "pop": 17829, "ids": [9406, 25254, 5668, 1410, 4520, 1377, 6991, 72], "a": 6, "n": 1, "alts": ["canada", "dominion of canada", "british north america", "can", "ca", "ca", "can", "can."], "dis": [[11114], [11198], [20169]]}, {"prop": "country", "subj": "Space Shuttle program", "obj": "United States of America", "pop": 16613, "ids": [14289, 2419, 58443, 2513, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 7, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 36408, 4774], [1612, 32439, 2469, 10035], [11690, 1379, 1358, 18442]]}, {"prop": "country", "subj": "Asiana Airlines", "obj": "South Korea", "pop": 15692, "ids": [2685, 11299, 36522, 1410, 4520, 1377, 4168, 14169, 72], "a": 6, "n": 2, "alts": ["south korea", "republic of korea", "rok", "kr", "rep. korea", "s. korea", "korea republic", "\ud83c\uddf0\ud83c\uddf7", "kor"], "dis": [[57728, 10524], [3532, 11690], [7868, 2272]]}, {"prop": "country", "subj": "DIC Entertainment", "obj": "United States of America", "pop": 14978, "ids": [1479, 2677, 22733, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[1612, 32439, 2469, 10035], [11690, 1379, 1358, 18442], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Eastern United States", "obj": "United States of America", "pop": 14208, "ids": [16225, 3532, 4025, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[24086, 7553, 2404, 34486], [11690, 1379, 36408, 4774], [1612, 32439, 2469, 10035]]}, {"prop": "country", "subj": "T-33", "obj": "United States of America", "pop": 13157, "ids": [1408, 63, 59362, 59362, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 7, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[24086, 7553, 2404, 34486], [1612, 32439, 2469, 10035], [11690, 1379, 1358, 18442]]}, {"prop": "country", "subj": "Traverse City", "obj": "United States of America", "pop": 13083, "ids": [18565, 4219, 4599, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[1612, 32439, 2469, 10035], [11690, 1379, 36408, 4774], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Antique", "obj": "Philippines", "pop": 11961, "ids": [6512, 3024, 1410, 4520, 1377, 21962, 72], "a": 5, "n": 1, "alts": ["philippines", "republic of the philippines", "ph", "ph", "the philippines", "philippine islands", "phl", "rp", "\ud83c\uddf5\ud83c\udded", "phi", "pinas"], "dis": [[28793], [29136], [49017]]}, {"prop": "country", "subj": "Port Authority Trans-Hudson", "obj": "United States of America", "pop": 11522, "ids": [6551, 23891, 5285, 63, 59389, 1528, 2162, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 10, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 36408, 4774], [11690, 1379, 1358, 18442], [1612, 32439, 2469, 10035]]}, {"prop": "country", "subj": "Rusyn", "obj": "Serbia", "pop": 11488, "ids": [46816, 2447, 1410, 4520, 1377, 37313, 72], "a": 5, "n": 1, "alts": ["ukraine", "ua", "ukr", "ua", "ukrainia", "\ud83c\uddfa\ud83c\udde6", "ukr.", "ukraina", "slovakia", "slovak republic", "sk", "\ud83c\uddf8\ud83c\uddf0", "svk", "romania", "roumania", "rumania", "rom\u00e2nia", "ro", "\ud83c\uddf7\ud83c\uddf4", "poland", "pol", "republic of poland", "pl", "polska", "serbia", "\ud83c\uddf7\ud83c\uddf8", "republic of serbia", "republika srbija", "rs", "srbija", "srb", "rs"], "dis": [[58501], [8810], [21189]]}, {"prop": "country", "subj": "Pierre", "obj": "United States of America", "pop": 11306, "ids": [35187, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 4, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 36408, 4774], [1612, 32439, 2469, 10035], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Illinois State University", "obj": "United States of America", "pop": 10846, "ids": [14087, 4200, 3063, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 36408, 4774], [1612, 32439, 2469, 10035], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Middle East Airlines", "obj": "Lebanon", "pop": 10477, "ids": [14507, 6953, 36522, 1410, 4520, 1377, 41492, 72], "a": 6, "n": 1, "alts": ["lebanon", "\ud83c\uddf1\ud83c\udde7", "lbn", "lebanese republic", "lb", "republic of lebanon"], "dis": [[23379], [25720], [26753]]}, {"prop": "country", "subj": "Japan\u2013South Korea relations", "obj": "Japan", "pop": 10394, "ids": [5701, 59586, 33857, 14169, 3875, 1410, 4520, 1377, 5701, 72], "a": 8, "n": 1, "alts": ["japan", "state of japan", "land of the rising sun", "nihon", "nippon", "jp", "nippon-koku", "nihon-koku", "ja", "jpn", "jp", "jap", "ja", "jap"], "dis": [[58501], [36733], [24098]]}, {"prop": "country", "subj": "Rong", "obj": "Norway", "pop": 10212, "ids": [1490, 1635, 1410, 4520, 1377, 20489, 72], "a": 5, "n": 1, "alts": ["norway", "kingdom of norway", "norge", "norv\u00e8ge\u200f", "no", "nor", "no", "noreg", "norwegen\u200f", "\ud83c\uddf3\ud83c\uddf4"], "dis": [[52044], [31355], [23263]]}, {"prop": "country", "subj": "Williamson County", "obj": "United States of America", "pop": 9836, "ids": [9753, 2162, 3889, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[1612, 32439, 2469, 10035], [11690, 1379, 1358, 18442], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Abkhaz", "obj": "Jordan", "pop": 9764, "ids": [4589, 22032, 2215, 1410, 4520, 1377, 20169, 72], "a": 6, "n": 1, "alts": ["russia", "rossiya", "rossija", "ru", "ru", "rossijskaja federatsija", "russian federation", "rossiyskaya federatsiya", "rus", "rus", "rf", "abkhazia", "abkhasia", "abkazia", "abchasia", "abkhaziya", "turkey", "republic of turkey", "\ud83c\uddf9\ud83c\uddf7", "tur", "tr", "jordan", "hashemite kingdom of jordan", "jo", "jor", "jo", "kingdom of jordan", "\ud83c\uddef\ud83c\uddf4", "syria", "sy", "\ud83c\uddf8\ud83c\uddfe", "syrian arab republic", "syr"], "dis": [[5995], [49017], [54634]]}, {"prop": "country", "subj": "Old Persian", "obj": "Iran", "pop": 9700, "ids": [11524, 49120, 1410, 4520, 1377, 11877, 72], "a": 5, "n": 1, "alts": ["iran", "islamic republic of iran", "persia", "ir", "islamic rep. iran", "\ud83c\uddee\ud83c\uddf7"], "dis": [[48274], [11114], [40275]]}, {"prop": "country", "subj": "Colorado College", "obj": "United States of America", "pop": 9019, "ids": [15947, 5846, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[24086, 7553, 2404, 34486], [11690, 1379, 1358, 18442], [11690, 1379, 36408, 4774]]}, {"prop": "country", "subj": "history of Ghana", "obj": "Ghana", "pop": 8839, "ids": [4537, 1379, 29136, 1410, 4520, 1377, 29136, 72], "a": 6, "n": 1, "alts": ["ghana", "republic of ghana", "gh", "gha"], "dis": [[15373], [11840], [22927]]}, {"prop": "country", "subj": "Bank of Korea", "obj": "South Korea", "pop": 8813, "ids": [9348, 1379, 14169, 1410, 4520, 1377, 4168, 14169, 72], "a": 6, "n": 2, "alts": ["south korea", "republic of korea", "rok", "kr", "rep. korea", "s. korea", "korea republic", "\ud83c\uddf0\ud83c\uddf7", "kor"], "dis": [[7370, 1347], [7868, 2272], [38391, 25686]]}, {"prop": "country", "subj": "The Goldfinch", "obj": "Netherlands", "pop": 8809, "ids": [1507, 8250, 10935, 1439, 1410, 4520, 1377, 18442, 72], "a": 7, "n": 1, "alts": ["netherlands", "holland", "the netherlands", "nl", "ned", "nederland", "nl", "\ud83c\uddf3\ud83c\uddf1", "netherlands (after 1945)"], "dis": [[22485], [48274], [11803]]}, {"prop": "country", "subj": "Vale", "obj": "Brazil", "pop": 8583, "ids": [58302, 1410, 4520, 1377, 11321, 72], "a": 4, "n": 1, "alts": ["brazil", "federative republic of brazil", "br", "bra", "br", "\ud83c\udde7\ud83c\uddf7"], "dis": [[41492], [37313], [8810]]}, {"prop": "country", "subj": "Valparaiso University", "obj": "United States of America", "pop": 8242, "ids": [6620, 1984, 12230, 59326, 3063, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 8, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 1358, 18442], [1612, 32439, 2469, 10035], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Curtin University", "obj": "Australia", "pop": 7644, "ids": [32523, 1347, 3063, 1410, 4520, 1377, 6690, 72], "a": 6, "n": 1, "alts": ["australia", "commonwealth of australia", "au", "aus", "au", "british colony of australia", "\ud83c\udde6\ud83c\uddfa", "straya", "aussieland"], "dis": [[22633], [16750], [25446]]}, {"prop": "country", "subj": "Jam Filled Entertainment", "obj": "Canada", "pop": 7577, "ids": [21670, 1493, 4955, 22733, 1410, 4520, 1377, 6991, 72], "a": 7, "n": 1, "alts": ["canada", "dominion of canada", "british north america", "can", "ca", "ca", "can", "can."], "dis": [[31842], [28793], [46849]]}, {"prop": "country", "subj": "Asian Highway Network", "obj": "Uzbekistan", "pop": 7405, "ids": [12015, 20560, 8516, 1410, 4520, 1377, 1612, 32439, 2469, 10035, 72], "a": 6, "n": 4, "alts": ["people's republic of china", "china", "cn", "pr china", "prc", "cn", "chn", "\ud83c\udde8\ud83c\uddf3", "china pr", "mainland china", "russia", "rossiya", "rossija", "ru", "ru", "rossijskaja federatsija", "russian federation", "rossiyskaya federatsiya", "rus", "rus", "rf", "japan", "state of japan", "land of the rising sun", "nihon", "nippon", "jp", "nippon-koku", "nihon-koku", "ja", "jpn", "jp", "jap", "ja", "jap", "azerbaijan", "republic of azerbaijan", "az", "\ud83c\udde6\ud83c\uddff", "aze", "kazakhstan", "republic of kazakhstan", "kz", "qazaqstan", "kaz", "\ud83c\uddf0\ud83c\uddff", "indonesia", "republic of indonesia", "id", "id", "ina", "idn", "uzbekistan", "republic of uzbekistan", "uz", "\ud83c\uddfa\ud83c\uddff", "uzb", "singapore", "republic of singapore", "sg", "sg", "\ud83c\uddf8\ud83c\uddec", "singapore city", "sgp", "lion city", "garden city", "armenia", "republic of armenia", "\ud83c\udde6\ud83c\uddf2", "arm", "am", "cambodia", "kingdom of cambodia", "kh", "\ud83c\uddf0\ud83c\udded", "kampuchea", "camboya", "campuchia", "kamboja", "kambodzha", "turkey", "republic of turkey", "\ud83c\uddf9\ud83c\uddf7", "tur", "tr", "india", "bharat", "hindustan", "bharatvarsh", "in", "in", "republic of india", "\ud83c\uddee\ud83c\uddf3", "ind", "aryavratt", "iran", "islamic republic of iran", "persia", "ir", "islamic rep. iran", "\ud83c\uddee\ud83c\uddf7", "kyrgyzstan", "kyrgyz republic", "kg", "kirgizia", "\ud83c\uddf0\ud83c\uddec", "kgz", "laos", "lao people's democratic republic", "la", "lao pdr", "lao", "\ud83c\uddf1\ud83c\udde6", "malaysia", "my", "federation of malaysia", "malaysia federation", "mas", "mys", "\ud83c\uddf2\ud83c\uddfe", "pakistan", "\ud83c\uddf5\ud83c\uddf0", "pk", "islamic republic of pakistan", "pak", "sri lanka", "democratic socialist republic of sri lanka", "ceylon", "taprobane", "serendib", "ceylan", "\ud83c\uddf1\ud83c\uddf0", "lk", "lanka, sri", "sri", "lke", "ce", "srilanka", "sri", "\u0633\u0631\u0646\u062f\u064a\u0628", "\u0633\u064a\u0644\u0627\u0646", "tajikistan", "republic of tajikistan", "tj", "\ud83c\uddf9\ud83c\uddef", "tjk", "thailand", "kingdom of thailand", "th", "siam", "\ud83c\uddf9\ud83c\udded", "land of smiles", "tha", "turkmenistan", "turkmenia", "tm", "tkm", "\ud83c\uddf9\ud83c\uddf2", "vietnam", "socialist republic of vietnam", "vn", "vie", "\ud83c\uddfb\ud83c\uddf3", "vi\u1ec7t nam", "viet nam", "srv", "vnm", "south korea", "republic of korea", "rok", "kr", "rep. korea", "s. korea", "korea republic", "\ud83c\uddf0\ud83c\uddf7", "kor", "afghanistan", "af", "\ud83c\udde6\ud83c\uddeb", "afg", "bangladesh", "bd", "\ud83c\udde7\ud83c\udde9", "ban", "people's republic of bangladesh", "bhutan", "kingdom of bhutan", "bt", "bhu", "\ud83c\udde7\ud83c\uddf9"], "dis": [[3532, 4025, 1379, 6263], [11690, 1379, 36408, 4774], [11690, 1379, 1358, 18442]]}, {"prop": "country", "subj": "Japanese passport", "obj": "Japan", "pop": 7272, "ids": [9765, 41979, 1410, 4520, 1377, 5701, 72], "a": 5, "n": 1, "alts": ["japan", "state of japan", "land of the rising sun", "nihon", "nippon", "jp", "nippon-koku", "nihon-koku", "ja", "jpn", "jp", "jap", "ja", "jap"], "dis": [[21962], [25720], [48274]]}, {"prop": "country", "subj": "Directorate of Military Intelligence", "obj": "United Kingdom", "pop": 7169, "ids": [10317, 1446, 1379, 25603, 28344, 1410, 4520, 1377, 3532, 11690, 72], "a": 8, "n": 2, "alts": ["united kingdom", "\ud83c\uddec\ud83c\udde7", "uk", "united kingdom of great britain and northern ireland", "u.k.", "gbr", "gb", "u. k.", "u k", "g.b.", "g. b.", "g b", "great britain", "g.b.r.", "g b r", "britain", "great britain and northern ireland"], "dis": [[44063, 1627], [52034, 18703], [38956, 1627]]}, {"prop": "country", "subj": "hundred", "obj": "Denmark", "pop": 6794, "ids": [8614, 1410, 4520, 1377, 25446, 72], "a": 4, "n": 1, "alts": ["norway", "kingdom of norway", "norge", "norv\u00e8ge\u200f", "no", "nor", "no", "noreg", "norwegen\u200f", "\ud83c\uddf3\ud83c\uddf4", "finland", "republic of finland", "finnia", "land of thousand lakes", "fi", "suomi", "suomen tasavalta", "republiken finland", "\ud83c\uddeb\ud83c\uddee", "fin", "sweden", "kingdom of sweden", "se", "se", "swe", "\ud83c\uddf8\ud83c\uddea", "denmark", "dk", "danmark", "dnk", "dek", "dk", "denmark proper", "metropolitan denmark", "\ud83c\udde9\ud83c\uddf0", "den", "tan", "realm of denmark", "kingdom of denmark"], "dis": [[24098], [25361], [22633]]}, {"prop": "country", "subj": "Mumbai Monorail", "obj": "India", "pop": 6557, "ids": [30263, 5246, 1355, 1630, 1410, 4520, 1377, 5995, 72], "a": 7, "n": 1, "alts": ["india", "bharat", "hindustan", "bharatvarsh", "in", "in", "republic of india", "\ud83c\uddee\ud83c\uddf3", "ind", "aryavratt"], "dis": [[6991], [16221], [13834]]}, {"prop": "country", "subj": "Katmai National Park and Preserve", "obj": "United States of America", "pop": 6479, "ids": [19151, 2728, 59325, 4012, 5433, 1384, 5563, 6106, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 11, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 1358, 18442], [11690, 1379, 36408, 4774], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Pitzer College", "obj": "United States of America", "pop": 6306, "ids": [19677, 6926, 5846, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 1358, 18442], [1612, 32439, 2469, 10035], [24086, 7553, 2404, 34486]]}, {"prop": "country", "subj": "Bethel", "obj": "United States of America", "pop": 6135, "ids": [11251, 3761, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[1612, 32439, 2469, 10035], [11690, 1379, 36408, 4774], [11690, 1379, 1358, 18442]]}, {"prop": "country", "subj": "Belavia", "obj": "Belarus", "pop": 5839, "ids": [7276, 42471, 1410, 4520, 1377, 46849, 72], "a": 5, "n": 1, "alts": ["belarus", "\ud83c\udde7\ud83c\uddfe", "white russia", "white ruthenia", "republic of belarus", "by", "byeloruss", "blr", "bielorussia", "belorussia", "byelorussia"], "dis": [[11114], [15373], [35161]]}, {"prop": "country", "subj": "Millennium Biltmore Hotel", "obj": "United States of America", "pop": 5838, "ids": [15693, 47331, 1469, 3860, 5938, 14040, 1410, 4520, 1377, 3532, 4025, 1379, 6263, 72], "a": 9, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[11690, 1379, 36408, 4774], [1612, 32439, 2469, 10035], [11690, 1379, 1358, 18442]]}, {"prop": "director", "subj": "The Matrix", "obj": "The Wachowskis", "pop": 291114, "ids": [1507, 23560, 1523, 8779, 1520, 1507, 1495, 1684, 2681, 59346, 1360, 72], "a": 5, "n": 6, "alts": ["the wachowskis", "wachowski brothers", "the wachowski brothers", "the wachowski starship", "wachowski sisters"], "dis": [[4801, 1445, 4623, 59339, 1440, 39282], [9114, 1549, 59361, 38242, 2816, 6119], [4220, 1967, 59326, 7799, 1869, 59325]]}, {"prop": "director", "subj": "Star Wars", "obj": "George Lucas", "pop": 288624, "ids": [10058, 19016, 1523, 8779, 1520, 9595, 39282, 72], "a": 5, "n": 2, "alts": ["george lucas", "george walton lucas jr."], "dis": [[15670, 23978], [8710, 8250], [39451, 20989]]}, {"prop": "director", "subj": "Saturday Night Live", "obj": "Beth McCarthy-Miller", "pop": 285161, "ids": [8675, 15387, 13588, 1523, 8779, 1520, 34362, 5281, 57416, 63, 59365, 9573, 72], "a": 6, "n": 6, "alts": ["beth mccarthy-miller", "don roy king", "donald roy king"], "dis": [[9753, 8458, 1487, 1564, 59365, 5203], [12065, 1440, 2371, 5541, 4102, 2549], [4801, 1445, 4623, 59339, 1440, 39282]]}, {"prop": "director", "subj": "Breaking Bad", "obj": "Michelle MacLaren", "pop": 274263, "ids": [55653, 15602, 1523, 8779, 1520, 33382, 6905, 59376, 16673, 72], "a": 5, "n": 4, "alts": ["john shiban", "michelle maclaren", "michelle maxwell maclaren", "charles haid", "charles maurice haid iii", "bryan cranston", "bryan lee cranston", "vince gilligan", "george vincent gilligan, jr.", "george mastras", "peter gould"], "dis": [[14811, 1839, 48193, 7513], [37332, 1788, 1599, 3243], [23978, 2685, 1462, 1429]]}, {"prop": "director", "subj": "Avatar", "obj": "James Cameron", "pop": 237293, "ids": [9383, 12361, 1523, 8779, 1520, 8266, 36890, 72], "a": 5, "n": 2, "alts": ["james cameron", "james francis cameron", "jim cameron"], "dis": [[19501, 25321], [43738, 34614], [12489, 10714]]}, {"prop": "director", "subj": "Halloween II", "obj": "Rob Zombie", "pop": 215400, "ids": [28920, 4401, 1523, 8779, 1520, 9027, 2572, 41871, 72], "a": 5, "n": 3, "alts": ["rob zombie", "robert bartleh cummings", "rob straker"], "dis": [[7524, 13519, 1391], [45853, 10665, 55244], [1469, 5203, 5445]]}, {"prop": "director", "subj": "The Terminal", "obj": "Steven Spielberg", "pop": 156195, "ids": [1507, 40975, 1523, 8779, 1520, 28204, 2870, 20090, 7662, 72], "a": 5, "n": 4, "alts": ["steven spielberg", "steven allan spielberg", "spielberg"], "dis": [[31382, 1534, 22834, 1639], [27128, 1490, 3198, 43063], [12209, 1394, 2748, 7845]]}, {"prop": "director", "subj": "Lost", "obj": "J. J. Abrams", "pop": 154447, "ids": [35739, 1523, 8779, 1520, 1609, 72, 1609, 72, 46657, 2552, 72], "a": 4, "n": 6, "alts": ["j. j. abrams", "jeffrey jacob abrams", "j.j. abrams", "j j abrams", "jj abrams"], "dis": [[16323, 11010, 1681, 2776, 59328, 13109], [1420, 1539, 57518, 3509, 3321, 7931], [1609, 72, 1413, 1382, 2371, 24400]]}, {"prop": "director", "subj": "Back to the Future", "obj": "Robert Zemeckis", "pop": 150710, "ids": [10203, 1385, 1358, 19186, 1523, 8779, 1520, 9812, 2572, 1454, 9725, 1360, 72], "a": 7, "n": 5, "alts": ["robert zemeckis", "robert l. zemeckis", "robert lee zemeckis"], "dis": [[23638, 36933, 2142, 1673, 7323], [4220, 1472, 35900, 1378, 59346], [5162, 14699, 5967, 1381, 23028]]}, {"prop": "director", "subj": "L\u00e9on: The Professional", "obj": "Luc Besson", "pop": 146310, "ids": [1494, 60025, 1350, 59358, 1507, 19567, 1523, 8779, 1520, 14811, 1469, 1481, 1350, 72], "a": 9, "n": 4, "alts": ["luc besson", "luc paul maurice besson"], "dis": [[27604, 19564, 8313, 32760], [1788, 1355, 1398, 46233], [4801, 8250, 20579, 40958]]}, {"prop": "director", "subj": "Seven", "obj": "David Fincher", "pop": 136347, "ids": [28625, 1523, 8779, 1520, 7524, 6727, 12156, 72], "a": 4, "n": 3, "alts": ["david fincher", "david andrew leo fincher"], "dis": [[41181, 3389, 45215], [9114, 10726, 1350], [43261, 8749, 4238]]}, {"prop": "director", "subj": "Alien", "obj": "Ridley Scott", "pop": 132756, "ids": [2270, 7607, 1523, 8779, 1520, 49926, 3389, 9610, 72], "a": 5, "n": 3, "alts": ["ridley scott", "sir ridley scott"], "dis": [[56984, 26162, 2199], [42060, 2870, 16973], [19255, 29388, 1657]]}, {"prop": "director", "subj": "Citizen Kane", "obj": "Orson Welles", "pop": 127989, "ids": [19864, 34543, 1681, 2611, 1523, 8779, 1520, 3271, 2162, 9608, 1359, 72], "a": 7, "n": 4, "alts": ["orson welles", "george orson welles"], "dis": [[32932, 1493, 1481, 57955], [50909, 16125, 1486, 1509], [9114, 1632, 2355, 25773]]}, {"prop": "director", "subj": "Grease", "obj": "Randal Kleiser", "pop": 127970, "ids": [26094, 1582, 1523, 8779, 1520, 1490, 26920, 29244, 14545, 72], "a": 5, "n": 4, "alts": ["randal kleiser", "john randal kleiser"], "dis": [[9595, 1413, 3014, 5542], [57941, 1452, 1546, 1353], [4801, 5967, 1888, 8650]]}, {"prop": "director", "subj": "Cars", "obj": "Joe Ranft", "pop": 107210, "ids": [44243, 1523, 8779, 1520, 15039, 33888, 1725, 72], "a": 4, "n": 3, "alts": ["john lasseter", "john alan lasseter", "john a. lasseter", "joe ranft", "joseph henry ranft"], "dis": [[47139, 14796, 1349], [9383, 59325, 19739], [40743, 1681, 22625]]}, {"prop": "director", "subj": "Iron Man", "obj": "Jon Favreau", "pop": 104680, "ids": [22415, 5174, 1523, 8779, 1520, 14972, 1493, 1591, 1352, 2874, 72], "a": 5, "n": 5, "alts": ["jon favreau", "jonathan favreau", "jonathan kolia favreau"], "dis": [[22313, 1469, 1894, 1381, 1725], [18223, 1479, 3560, 4998, 59346], [2785, 44753, 56477, 59322, 15903]]}, {"prop": "director", "subj": "The Craft", "obj": "Andrew Fleming", "pop": 103205, "ids": [1507, 31254, 1523, 8779, 1520, 15054, 36120, 4764, 72], "a": 5, "n": 3, "alts": ["andrew fleming"], "dis": [[27863, 51138, 9161], [19378, 5758, 20233], [55720, 10565, 1363]]}, {"prop": "director", "subj": "The Holiday", "obj": "Nancy Meyers", "pop": 101218, "ids": [1507, 34015, 1523, 8779, 1520, 37160, 1445, 2825, 1465, 72], "a": 5, "n": 4, "alts": ["nancy meyers", "nancy jane meyers"], "dis": [[3969, 3362, 6512, 4774], [36545, 2518, 1362, 2450], [1469, 6533, 7846, 19299]]}, {"prop": "director", "subj": "The Omen", "obj": "Richard Donner", "pop": 99370, "ids": [1507, 1549, 8858, 1523, 8779, 1520, 11646, 7063, 2120, 72], "a": 6, "n": 3, "alts": ["richard donner", "richard donald schwartzberg"], "dis": [[20222, 5198, 10804], [25214, 11988, 1382], [12583, 53910, 1355]]}, {"prop": "director", "subj": "E.T. the Extra-Terrestrial", "obj": "Steven Spielberg", "pop": 97322, "ids": [1529, 72, 59351, 72, 1358, 33897, 63, 36864, 44231, 1523, 8779, 1520, 28204, 2870, 20090, 7662, 72], "a": 12, "n": 4, "alts": ["steven spielberg", "steven allan spielberg", "spielberg"], "dis": [[25287, 9735, 1568, 5419], [21861, 18419, 1363, 13523], [11033, 13389, 5583, 2405]]}, {"prop": "director", "subj": "In Time", "obj": "Andrew Niccol", "pop": 94721, "ids": [1646, 7555, 1523, 8779, 1520, 15054, 16500, 2391, 72], "a": 5, "n": 3, "alts": ["andrew niccol", "andrew m. niccol"], "dis": [[1469, 1461, 59334], [9114, 33158, 1434], [21872, 10102, 1373]]}, {"prop": "director", "subj": "Ghost", "obj": "Jerry Zucker", "pop": 88655, "ids": [40220, 1523, 8779, 1520, 32905, 2572, 28907, 72], "a": 4, "n": 3, "alts": ["jerry zucker"], "dis": [[28872, 57514, 1347], [8213, 10389, 59326], [29151, 2861, 2751]]}, {"prop": "director", "subj": "Training Day", "obj": "Antoine Fuqua", "pop": 86325, "ids": [16181, 6666, 1523, 8779, 1520, 6512, 59326, 1534, 34986, 57001, 72], "a": 5, "n": 5, "alts": ["antoine fuqua"], "dis": [[52006, 6875, 2382, 59333, 1834], [45853, 24859, 1880, 15852, 1439], [14642, 28881, 63, 1625, 59346]]}, {"prop": "director", "subj": "Into the Wild", "obj": "Sean Penn", "pop": 83157, "ids": [50324, 1358, 13337, 1523, 8779, 1520, 32392, 34970, 72], "a": 6, "n": 2, "alts": ["sean penn", "sean justin penn", "pappy pariah"], "dis": [[9114, 8031], [15816, 33038], [10709, 25747]]}, {"prop": "director", "subj": "Carrie", "obj": "Kimberly Peirce", "pop": 80170, "ids": [3684, 9477, 1523, 8779, 1520, 56860, 1434, 9014, 1448, 1437, 72], "a": 5, "n": 5, "alts": ["kimberly peirce", "kimberly ane peirce"], "dis": [[24336, 1355, 59334, 14458, 41918], [6274, 59337, 7884, 4098, 50989], [14642, 28881, 63, 11612, 1853]]}, {"prop": "director", "subj": "The Birds", "obj": "Alfred Hitchcock", "pop": 79684, "ids": [1507, 57172, 1523, 8779, 1520, 34683, 1491, 7212, 47402, 72], "a": 5, "n": 4, "alts": ["alfred hitchcock", "alfred joseph hitchcock", "sir alfred hitchcock"], "dis": [[10665, 2419, 59327, 14348], [15039, 8599, 3493, 1349], [1527, 1406, 13389, 21109]]}, {"prop": "director", "subj": "Inside Out", "obj": "Ronnie del Carmen", "pop": 74976, "ids": [33044, 6249, 1523, 8779, 1520, 23604, 21646, 2664, 31382, 1356, 72], "a": 5, "n": 5, "alts": ["pete docter", "peter docter", "peter hans docter", "ronnie del carmen", "ronaldo del carmen", "ronaldo del carmen"], "dis": [[7370, 1413, 6714, 14685, 1522], [1493, 2072, 1490, 4941, 7873], [14642, 28881, 63, 1625, 59346]]}, {"prop": "director", "subj": "Blade", "obj": "Stephen Norrington", "pop": 73446, "ids": [3474, 1958, 1523, 8779, 1520, 19378, 6485, 4952, 2232, 72], "a": 5, "n": 4, "alts": ["stephen norrington"], "dis": [[56182, 1495, 1356, 2816], [9753, 7196, 4493, 1378], [12583, 50951, 40791, 1541]]}, {"prop": "director", "subj": "Her", "obj": "Spike Jonze", "pop": 70875, "ids": [6555, 1523, 8779, 1520, 2870, 4601, 14972, 3846, 72], "a": 4, "n": 4, "alts": ["spike jonze", "adam spiegel"], "dis": [[19562, 2748, 7799, 2728], [44761, 16500, 1411, 3856], [56182, 1495, 1356, 2816]]}, {"prop": "director", "subj": "Crash", "obj": "Paul Haggis", "pop": 69270, "ids": [5758, 2116, 1523, 8779, 1520, 7821, 1491, 20233, 1360, 72], "a": 5, "n": 4, "alts": ["paul haggis", "paul edward haggis"], "dis": [[43028, 11748, 59343, 15271], [42968, 2522, 4939, 1349], [9549, 1490, 6443, 2225]]}, {"prop": "director", "subj": "The Fast and the Furious", "obj": "Rob Cohen", "pop": 68781, "ids": [1507, 20602, 1384, 1358, 1493, 21251, 1523, 8779, 1520, 9027, 37050, 72], "a": 9, "n": 2, "alts": ["rob cohen", "robert cohen"], "dis": [[16067, 12019], [24052, 27674], [12344, 41106]]}, {"prop": "director", "subj": "Scary Movie 3", "obj": "David Zucker", "pop": 67560, "ids": [2732, 1629, 25517, 59320, 59362, 1523, 8779, 1520, 7524, 2572, 28907, 72], "a": 8, "n": 3, "alts": ["david zucker"], "dis": [[10714, 59339, 22536], [1469, 7362, 40103], [12583, 25690, 1350]]}, {"prop": "director", "subj": "Independence Day", "obj": "Roland Emmerich", "pop": 66502, "ids": [41717, 6666, 1523, 8779, 1520, 23020, 1483, 6503, 1805, 1620, 72], "a": 5, "n": 5, "alts": ["roland emmerich"], "dis": [[1527, 1600, 1350, 1490, 3152], [4016, 41414, 63, 1656, 59326], [17657, 63, 54440, 6274, 1578]]}, {"prop": "director", "subj": "2012", "obj": "Roland Emmerich", "pop": 65303, "ids": [59320, 59349, 59344, 59345, 59349, 1523, 8779, 1520, 23020, 1483, 6503, 1805, 1620, 72], "a": 8, "n": 5, "alts": ["roland emmerich"], "dis": [[1452, 1497, 59325, 9383, 11561], [51222, 34150, 59331, 10564, 5419], [1527, 2576, 50989, 29620, 18602]]}, {"prop": "director", "subj": "Taken", "obj": "Pierre Morel", "pop": 64711, "ids": [45090, 1523, 8779, 1520, 35187, 5547, 59329, 72], "a": 4, "n": 3, "alts": ["pierre morel"], "dis": [[1507, 27037, 19924], [9595, 2419, 9161], [3013, 18891, 55481]]}, {"prop": "director", "subj": "The Room", "obj": "Tommy Wiseau", "pop": 64610, "ids": [1507, 15980, 1523, 8779, 1520, 44135, 1495, 1998, 2874, 72], "a": 5, "n": 4, "alts": ["tommy wiseau"], "dis": [[1419, 2777, 7291, 3128], [9486, 4592, 1611, 7374], [41266, 3474, 4765, 59338]]}, {"prop": "director", "subj": "The Mask", "obj": "Chuck Russell", "pop": 63555, "ids": [1507, 53742, 1523, 8779, 1520, 45254, 28123, 72], "a": 5, "n": 2, "alts": ["chuck russell", "charles russell"], "dis": [[22430, 7821], [8710, 39861], [8266, 36890]]}, {"prop": "director", "subj": "Death Becomes Her", "obj": "Robert Zemeckis", "pop": 63335, "ids": [25410, 26991, 10310, 6555, 1523, 8779, 1520, 9812, 2572, 1454, 9725, 1360, 72], "a": 7, "n": 5, "alts": ["robert zemeckis", "robert l. zemeckis", "robert lee zemeckis"], "dis": [[17615, 42786, 3327, 1456, 11731], [28230, 1417, 1439, 2371, 13109], [2944, 5419, 33306, 1451, 32081]]}, {"prop": "director", "subj": "The Town", "obj": "Ben Affleck", "pop": 63230, "ids": [1507, 8962, 1523, 8779, 1520, 7370, 11965, 1378, 2108, 72], "a": 5, "n": 4, "alts": ["ben affleck", "benjami g\u00e9za affleck", "benjamin g\u00e9za affleck-boldt", "benjami geza affleck", "benjamin geza affleck-boldt"], "dis": [[9114, 2572, 1347, 7662], [20987, 21640, 3360, 59398], [19378, 1479, 15392, 1687]]}, {"prop": "director", "subj": "Hugo", "obj": "Martin Scorsese", "pop": 61534, "ids": [52288, 1523, 8779, 1520, 12489, 2732, 1698, 4768, 72], "a": 4, "n": 4, "alts": ["martin scorsese", "martin skorseze", "martin charles scorsese"], "dis": [[5174, 1966, 19093, 5560], [13457, 45418, 59322, 2190], [11225, 5162, 1551, 1431]]}, {"prop": "director", "subj": "Zombieland", "obj": "Ruben Fleischer", "pop": 61502, "ids": [2572, 6714, 20090, 1483, 1523, 8779, 1520, 15616, 1356, 36120, 36943, 72], "a": 7, "n": 4, "alts": ["ruben fleischer", "ruben samuel fleischer"], "dis": [[7524, 1420, 72, 28653], [41266, 2063, 1360, 3061], [12339, 49690, 2491, 50924]]}, {"prop": "director", "subj": "Casino", "obj": "Martin Scorsese", "pop": 59995, "ids": [35411, 1523, 8779, 1520, 12489, 2732, 1698, 4768, 72], "a": 4, "n": 4, "alts": ["martin scorsese", "martin skorseze", "martin charles scorsese"], "dis": [[56367, 1515, 2004, 1437], [16250, 54421, 34929, 30865], [15039, 8599, 3493, 1349]]}, {"prop": "director", "subj": "Silence", "obj": "Martin Scorsese", "pop": 58729, "ids": [9555, 1647, 1523, 8779, 1520, 12489, 2732, 1698, 4768, 72], "a": 5, "n": 4, "alts": ["martin scorsese", "martin skorseze", "martin charles scorsese"], "dis": [[1417, 1485, 28505, 3389], [16250, 54421, 34929, 30865], [9826, 1394, 1479, 38546]]}, {"prop": "director", "subj": "Total Recall", "obj": "Paul Verhoeven", "pop": 57353, "ids": [17241, 4407, 1541, 1523, 8779, 1520, 7821, 9519, 7214, 10431, 72], "a": 6, "n": 4, "alts": ["paul verhoeven"], "dis": [[29388, 4618, 1491, 1808], [29151, 1515, 2984, 3420], [58175, 1378, 1494, 53086]]}, {"prop": "director", "subj": "Room", "obj": "Lenny Abrahamson", "pop": 56453, "ids": [15980, 1523, 8779, 1520, 32311, 3362, 46753, 2162, 72], "a": 4, "n": 4, "alts": ["lenny abrahamson", "leonard ian abrahamson", "leonard abrahamson"], "dis": [[49395, 9114, 1413, 1880], [52346, 1598, 1448, 28537], [22328, 21711, 1690, 4343]]}, {"prop": "director", "subj": "Brothers", "obj": "Jim Sheridan", "pop": 55925, "ids": [31339, 1523, 8779, 1520, 12980, 19357, 1402, 1363, 72], "a": 4, "n": 4, "alts": ["jim sheridan"], "dis": [[1490, 10524, 1491, 42685], [32311, 3362, 46753, 2162], [4589, 1391, 1527, 1659]]}, {"prop": "director", "subj": "My Girl", "obj": "Howard Zieff", "pop": 55131, "ids": [4162, 13191, 1523, 8779, 1520, 24987, 2572, 59325, 3492, 72], "a": 5, "n": 4, "alts": ["howard zieff"], "dis": [[16234, 22834, 2420, 1349], [15089, 1494, 1721, 59339], [7341, 52465, 1646, 2206]]}, {"prop": "director", "subj": "A.I. Artificial Intelligence", "obj": "Steven Spielberg", "pop": 54344, "ids": [1420, 72, 59355, 72, 46715, 28344, 1523, 8779, 1520, 28204, 2870, 20090, 7662, 72], "a": 9, "n": 4, "alts": ["steven spielberg", "steven allan spielberg", "spielberg"], "dis": [[1408, 5046, 20362, 5031], [26750, 1419, 1589, 1439], [4801, 10709, 1479, 41625]]}, {"prop": "director", "subj": "Lucy", "obj": "Luc Besson", "pop": 53411, "ids": [51354, 1523, 8779, 1520, 14811, 1469, 1481, 1350, 72], "a": 4, "n": 4, "alts": ["luc besson", "luc paul maurice besson"], "dis": [[37230, 1522, 23884, 15119], [43464, 2120, 14061, 12714], [27816, 34670, 1347, 7945]]}, {"prop": "director", "subj": "Lincoln", "obj": "Steven Spielberg", "pop": 53410, "ids": [23423, 1523, 8779, 1520, 28204, 2870, 20090, 7662, 72], "a": 4, "n": 4, "alts": ["steven spielberg", "steven allan spielberg", "spielberg"], "dis": [[57941, 1452, 1546, 1353], [51578, 1497, 10039, 1422], [5279, 1839, 8996, 1460]]}, {"prop": "director", "subj": "Brave", "obj": "Steve Purcell", "pop": 53071, "ids": [3317, 1543, 1523, 8779, 1520, 15816, 16965, 5690, 72], "a": 5, "n": 3, "alts": ["steve purcell", "steve ross purcell", "steven ross purcell", "brenda chapman"], "dis": [[4801, 23986, 16065], [11238, 43464, 2120], [9549, 9519, 3945]]}, {"prop": "director", "subj": "Falling Down", "obj": "Joel Schumacher", "pop": 51068, "ids": [18834, 1371, 16442, 1523, 8779, 1520, 46319, 5967, 1451, 16211, 72], "a": 6, "n": 4, "alts": ["joel schumacher", "joel t. schumacher"], "dis": [[7524, 1420, 72, 28653], [9753, 3594, 7839, 1534], [1452, 1517, 11371, 27467]]}, {"prop": "director", "subj": "Monster", "obj": "Patty Jenkins", "pop": 50709, "ids": [48575, 1523, 8779, 1520, 5583, 1961, 42279, 72], "a": 4, "n": 3, "alts": ["patty jenkins", "patricia lea \"patty\" jenkins", "patricia lea jenkins"], "dis": [[45431, 14339, 59321], [54864, 8420, 59398], [8266, 27599, 1558]]}, {"prop": "director", "subj": "Elf", "obj": "Jon Favreau", "pop": 50490, "ids": [5279, 59337, 1523, 8779, 1520, 14972, 1493, 1591, 1352, 2874, 72], "a": 5, "n": 5, "alts": ["jon favreau", "jonathan favreau", "jonathan kolia favreau"], "dis": [[4801, 5281, 16828, 7116, 1350], [1612, 8197, 4588, 4391, 2230], [28860, 1557, 1839, 54153, 19116]]}, {"prop": "director", "subj": "Fantastic Four", "obj": "Josh Trank", "pop": 50266, "ids": [29374, 4214, 12891, 1523, 8779, 1520, 24846, 2278, 2231, 72], "a": 6, "n": 3, "alts": ["josh trank", "joshua benjamin \"josh\" trank"], "dis": [[58311, 17678, 59325], [52361, 4498, 43786], [25214, 18869, 5203]]}, {"prop": "director", "subj": "The Intern", "obj": "Nancy Meyers", "pop": 48727, "ids": [1507, 4387, 1523, 8779, 1520, 37160, 1445, 2825, 1465, 72], "a": 5, "n": 4, "alts": ["nancy meyers", "nancy jane meyers"], "dis": [[3327, 1543, 1356, 55316], [37230, 1522, 10909, 3389], [37332, 1788, 1599, 3243]]}, {"prop": "director", "subj": "RED", "obj": "Robert Schwentke", "pop": 47253, "ids": [48141, 1523, 8779, 1520, 9812, 36111, 1386, 1509, 72], "a": 4, "n": 4, "alts": ["robert schwentke"], "dis": [[22313, 1452, 3761, 1363], [52378, 7681, 24859, 57766], [5378, 1445, 2494, 1438]]}, {"prop": "director", "subj": "The Beach", "obj": "Danny Boyle", "pop": 46958, "ids": [1507, 12532, 1523, 8779, 1520, 39421, 16433, 1378, 72], "a": 5, "n": 3, "alts": ["danny boyle"], "dis": [[26500, 12065, 2888], [25340, 39332, 1347], [2758, 26036, 34375]]}, {"prop": "director", "subj": "Saw II", "obj": "Darren Lynn Bouseman", "pop": 46937, "ids": [40373, 4401, 1523, 8779, 1520, 15319, 2261, 42768, 1469, 2441, 2450, 72], "a": 5, "n": 6, "alts": ["darren lynn bouseman", "darren lynn bousman"], "dis": [[6248, 3389, 1419, 1607, 1391, 1677], [14195, 1684, 1400, 55250, 1472, 1435], [16949, 1793, 1622, 30695, 1459, 59339]]}, {"prop": "director", "subj": "Annie", "obj": "John Huston", "pop": 46479, "ids": [55617, 1523, 8779, 1520, 4801, 1491, 1569, 1350, 72], "a": 4, "n": 4, "alts": ["john huston"], "dis": [[1469, 6533, 7846, 19299], [37332, 1788, 1400, 1375], [1479, 42974, 46351, 2450]]}, {"prop": "director", "subj": "Hook", "obj": "Steven Spielberg", "pop": 46255, "ids": [40570, 1523, 8779, 1520, 28204, 2870, 20090, 7662, 72], "a": 4, "n": 4, "alts": ["steven spielberg", "steven allan spielberg", "spielberg"], "dis": [[26159, 36120, 3560, 1853], [2534, 46611, 2572, 22632], [13187, 11817, 1456, 1363]]}, {"prop": "director", "subj": "Legion", "obj": "Scott Stewart", "pop": 43805, "ids": [50530, 1523, 8779, 1520, 9610, 28653, 72], "a": 4, "n": 2, "alts": ["scott stewart"], "dis": [[19711, 58851], [11238, 15670], [33596, 34778]]}, {"prop": "director", "subj": "This Is the End", "obj": "Seth Rogen", "pop": 43461, "ids": [1900, 2899, 1358, 9946, 1523, 8779, 1520, 58896, 1490, 4072, 72], "a": 7, "n": 3, "alts": ["seth rogen", "seth aaron rogen", "evan goldberg"], "dis": [[9549, 1527, 3417], [19488, 11891, 1861], [12489, 1490, 2349]]}, {"prop": "director", "subj": "The Great Wall", "obj": "Zhang Yimou", "pop": 42900, "ids": [1507, 7566, 11796, 1523, 8779, 1520, 37332, 1788, 1400, 1375, 72], "a": 6, "n": 4, "alts": ["zhang yimou"], "dis": [[7063, 2732, 1578, 3972], [58175, 1378, 1494, 53086], [16067, 1413, 1558, 22901]]}, {"prop": "director", "subj": "The Interview", "obj": "Evan Goldberg", "pop": 42217, "ids": [1507, 42757, 1523, 8779, 1520, 55751, 8250, 7662, 72], "a": 5, "n": 3, "alts": ["seth rogen", "seth aaron rogen", "evan goldberg"], "dis": [[11374, 12911, 30764], [9812, 29739, 2450], [44761, 1515, 2797]]}, {"prop": "director", "subj": "The Fly", "obj": "Kurt Neumann", "pop": 41809, "ids": [1507, 26728, 1523, 8779, 1520, 49699, 5732, 40605, 72], "a": 5, "n": 3, "alts": ["kurt neumann"], "dis": [[19255, 1716, 2190], [14628, 4915, 2082], [1469, 5203, 5445]]}, {"prop": "director", "subj": "Chef", "obj": "Jon Favreau", "pop": 41346, "ids": [40104, 1523, 8779, 1520, 14972, 1493, 1591, 1352, 2874, 72], "a": 4, "n": 5, "alts": ["jon favreau", "jonathan favreau", "jonathan kolia favreau"], "dis": [[14195, 2825, 52570, 1404, 2672], [11817, 1684, 21400, 12180, 1350], [39135, 1378, 1420, 9683, 25829]]}, {"prop": "director", "subj": "Sunshine", "obj": "Danny Boyle", "pop": 40724, "ids": [9826, 29164, 1523, 8779, 1520, 39421, 16433, 1378, 72], "a": 5, "n": 3, "alts": ["danny boyle"], "dis": [[2270, 12399, 3972], [42060, 2870, 16973], [20987, 19121, 1391]]}, {"prop": "director", "subj": "Brazil", "obj": "Terry Gilliam", "pop": 40686, "ids": [11321, 1523, 8779, 1520, 35219, 1527, 24507, 72], "a": 4, "n": 3, "alts": ["terry gilliam", "terrence vance gilliam"], "dis": [[22536, 1493, 1635], [21716, 48819, 2968], [9812, 7868, 54941]]}, {"prop": "director", "subj": "PK", "obj": "Rajkumar Hirani", "pop": 40668, "ids": [21358, 1523, 8779, 1520, 16949, 59346, 24400, 45609, 5419, 72], "a": 4, "n": 5, "alts": ["rajkumar hirani"], "dis": [[1493, 2072, 1490, 4941, 7873], [34721, 3814, 3308, 1540, 1350], [23638, 36933, 2142, 1673, 7323]]}, {"prop": "father", "subj": "Donald Trump", "obj": "Fred Trump", "pop": 15101521, "ids": [1507, 8279, 1379, 16234, 7562, 1523, 22430, 7562, 72], "a": 6, "n": 2, "alts": ["fred trump", "frederick christ trump", "fred christ trump", "frederick trump", "frederick christ trump, jr.", "frederick trump, jr.", "fred christ trump, jr."], "dis": [[9014, 3646], [32392, 34970], [27145, 4417]]}, {"prop": "father", "subj": "Brandon Lee", "obj": "Bruce Lee", "pop": 1452516, "ids": [1507, 8279, 1379, 43953, 12917, 1523, 27229, 12917, 72], "a": 6, "n": 2, "alts": ["bruce lee", "lee jun-fan"], "dis": [[47864, 14318], [2419, 15653], [1495, 15337]]}, {"prop": "father", "subj": "Elizabeth II", "obj": "George VI", "pop": 1291432, "ids": [1507, 8279, 1379, 19594, 4401, 1523, 9595, 31650, 72], "a": 6, "n": 2, "alts": ["george vi", "albert frederick arthur george", "king george vi", "albert windsor", "albert frederick arthur george windsor", "george windsor", "albert", "bertie", "prince albert, duke of york"], "dis": [[19378, 9114], [1527, 10292], [16294, 49525]]}, {"prop": "father", "subj": "Michael Jackson", "obj": "Joe Jackson", "pop": 753723, "ids": [1507, 8279, 1379, 9114, 15670, 1523, 15039, 15670, 72], "a": 6, "n": 2, "alts": ["joe jackson", "joseph walter jackson"], "dis": [[22345, 43135], [14943, 13523], [57699, 1406]]}, {"prop": "father", "subj": "Kim Kardashian", "obj": "Robert Kardashian", "pop": 707111, "ids": [1507, 8279, 1379, 14642, 1681, 1578, 2116, 1770, 1523, 9812, 1681, 1578, 2116, 1770, 72], "a": 9, "n": 5, "alts": ["robert kardashian", "robert george kardashian", "rob kardashian"], "dis": [[45220, 2978, 1690, 1379, 18555], [9812, 1417, 1379, 9735, 7553], [16067, 1527, 2729, 1356, 22901]]}, {"prop": "father", "subj": "Dwayne Johnson", "obj": "Rocky Johnson", "pop": 636668, "ids": [1507, 8279, 1379, 1479, 2413, 1861, 14318, 1523, 47864, 14318, 72], "a": 8, "n": 2, "alts": ["rocky johnson", "wayde douglas bowles", "sweet ebony diamond"], "dis": [[16162, 28653], [14943, 13523], [6132, 48211]]}, {"prop": "father", "subj": "Barack Obama", "obj": "Barack Obama Sr.", "pop": 592684, "ids": [1507, 8279, 1379, 45072, 15469, 1523, 45072, 15469, 31821, 72, 72], "a": 6, "n": 4, "alts": ["barack obama sr.", "barack hussein obama, sr.", "barack hussein obama"], "dis": [[14943, 47049, 34974, 1411], [9595, 1417, 1379, 22485], [34683, 1529, 1434, 12532]]}, {"prop": "father", "subj": "George VI", "obj": "George V", "pop": 420204, "ids": [1507, 8279, 1379, 9595, 31650, 1523, 9595, 1690, 72], "a": 6, "n": 2, "alts": ["george v", "george frederick ernest albert", "george windsor", "george frederick ernest albert windsor", "george saxe-coburg-gotha", "george frederick ernest albert saxe-coburg-gotha", "king george v", "king of great britain george v", "v, king of great britain george", "george v of the united kingdom"], "dis": [[16162, 28653], [27229, 12917], [10804, 23423]]}, {"prop": "father", "subj": "Alexander the Great", "obj": "Philip II of Macedon", "pop": 394950, "ids": [1507, 8279, 1379, 19488, 1358, 7566, 1523, 25214, 4401, 1379, 44214, 1350, 72], "a": 7, "n": 5, "alts": ["philip ii of macedon", "king of macedonia filippo ii", "king of macedonia philippos ii", "king of macedonia philip ii", "king of macedonia filip ii", "king of macedonia philippus ii", "philip of macedon"], "dis": [[3227, 13746, 1445, 3865, 1359], [9753, 14943, 11988, 1430, 17031], [1413, 7839, 1379, 24996, 30781]]}, {"prop": "father", "subj": "Abraham Lincoln", "obj": "Thomas Lincoln", "pop": 394926, "ids": [1507, 8279, 1379, 46753, 23423, 1523, 10804, 23423, 72], "a": 6, "n": 2, "alts": ["thomas lincoln"], "dis": [[2419, 15653], [10518, 20355], [27145, 4417]]}, {"prop": "father", "subj": "Jesus", "obj": "Joseph", "pop": 353596, "ids": [1507, 8279, 1379, 13855, 1523, 15089, 72], "a": 5, "n": 1, "alts": ["joseph", "saint joseph", "st joseph", "josef", "st. joseph", "god the father", "the father", "father"], "dis": [[47588], [20362], [48457]]}, {"prop": "father", "subj": "George Washington", "obj": "Augustine Washington", "pop": 350454, "ids": [1507, 8279, 1379, 9595, 8225, 1523, 5445, 1534, 8225, 72], "a": 6, "n": 3, "alts": ["augustine washington", "augustine washington sr."], "dis": [[8439, 6875, 18764], [8710, 27475, 29897], [14196, 3232, 59334]]}, {"prop": "father", "subj": "George W. Bush", "obj": "George H. W. Bush", "pop": 311254, "ids": [1507, 8279, 1379, 9595, 1495, 72, 24666, 1523, 9595, 1491, 72, 1495, 72, 24666, 72], "a": 8, "n": 6, "alts": ["george h. w. bush", "george bush sr.", "ghw bush", "george bush", "george h.w.", "g.h.w. bush", "ghwb", "george h.w.bush", "george h. w", "bush i", "george h w", "george herbert bush", "george h. w.", "h w bush", "george bush snr.", "george h. bush", "vice president bush", "h. w. bush", "g. h. w. bush", "herbert walker bush", "george h bush", "bush sr.", "g h w bush", "president bush", "george herbert walker bush", "president george h. w. bush", "george h w bush", "george bush sr", "h.w. bush", "president george h.w. bush", "bush senior", "george bush senior", "bush 41", "george h.w. bush", "george w. bush senior", "george bush, sr."], "dis": [[53399, 1528, 2146, 59324, 57305, 47475], [10389, 1527, 3016, 2900, 28862, 1861], [36545, 2419, 1783, 2450, 1379, 40275]]}, {"prop": "father", "subj": "Mary, Queen of Scots", "obj": "James V of Scotland", "pop": 304392, "ids": [1507, 8279, 1379, 8959, 59342, 16039, 1379, 2732, 2974, 1523, 8266, 1690, 1379, 15280, 72], "a": 10, "n": 4, "alts": ["james v of scotland", "james stewart", "james v", "goodman of ballengeich"], "dis": [[11238, 1495, 14437, 1809], [2270, 1370, 1374, 4401], [1408, 1516, 15666, 59321]]}, {"prop": "father", "subj": "Selena Quintanilla", "obj": "Abraham Quintanilla,", "pop": 234014, "ids": [1507, 8279, 1379, 25550, 9321, 4286, 1565, 1363, 8034, 1523, 46753, 4286, 1565, 1363, 8034, 59342, 72], "a": 10, "n": 6, "alts": ["abraham quintanilla,", "abraham isaac quintanilla"], "dis": [[53399, 1528, 2146, 59324, 57305, 47475], [1529, 2371, 28146, 1788, 2215, 13109], [28230, 43833, 17871, 4401, 1379, 22485]]}, {"prop": "father", "subj": "Melora Hardin", "obj": "Jerry Hardin", "pop": 215189, "ids": [1507, 8279, 1379, 10073, 8390, 13794, 1347, 1523, 32905, 13794, 1347, 72], "a": 8, "n": 3, "alts": ["jerry hardin"], "dis": [[1420, 3458, 1355], [12209, 1394, 1417], [1793, 30824, 1350]]}, {"prop": "father", "subj": "Wayne Gretzky", "obj": "Walter Gretzky", "pop": 210751, "ids": [1507, 8279, 1379, 29776, 1527, 1626, 59398, 7625, 1523, 28143, 1527, 1626, 59398, 7625, 72], "a": 9, "n": 5, "alts": ["walter gretzky"], "dis": [[9812, 1417, 1379, 9735, 7553], [1413, 7839, 1379, 24996, 30781], [29388, 3300, 63, 23622, 12917]]}, {"prop": "father", "subj": "John, King of England", "obj": "Henry II of England", "pop": 189191, "ids": [1507, 8279, 1379, 4801, 59342, 6132, 1379, 7610, 1523, 14943, 4401, 1379, 7610, 72], "a": 9, "n": 4, "alts": ["henry ii of england", "henry curtmantle", "henry fitzempress", "henry plantagenet", "henry ii", "king of england henry ii", "henry of anjou", "king henry ii of england", "king henry ii"], "dis": [[51578, 3458, 1379, 14185], [22430, 1419, 72, 50593], [37340, 1690, 1379, 25446]]}, {"prop": "father", "subj": "Joan of Arc", "obj": "Jacques d'Arc", "pop": 181930, "ids": [1507, 8279, 1379, 47651, 1379, 27372, 1523, 52361, 1383, 59361, 43364, 72], "a": 7, "n": 4, "alts": ["jacques d'arc"], "dis": [[21932, 26338, 2225, 7625], [29103, 21711, 7799, 2728], [4801, 13461, 1379, 25720]]}, {"prop": "father", "subj": "King Arthur", "obj": "Uther Pendragon", "pop": 176470, "ids": [1507, 8279, 1379, 6132, 25340, 1523, 1612, 1682, 1452, 1517, 59327, 7443, 72], "a": 6, "n": 6, "alts": ["uther pendragon"], "dis": [[46753, 4286, 1565, 1363, 8034, 59342], [37340, 4401, 1379, 8654, 59322, 3945], [1681, 1543, 59331, 35048, 1447, 1363]]}, {"prop": "father", "subj": "Krishna", "obj": "Vasudeva", "pop": 164554, "ids": [1507, 8279, 1379, 46003, 3003, 1523, 35296, 1528, 27966, 72], "a": 6, "n": 3, "alts": ["vasudeva"], "dis": [[1494, 1423, 1684], [16339, 3317, 2903], [11374, 1408, 2118]]}, {"prop": "father", "subj": "Charles II of England", "obj": "Charles I of England", "pop": 163843, "ids": [1507, 8279, 1379, 12583, 4401, 1379, 7610, 1523, 12583, 1417, 1379, 7610, 72], "a": 8, "n": 4, "alts": ["charles i of england", "charles i", "king of england charles i", "charles ist", "king charles the martyr"], "dis": [[7341, 52465, 1681, 1969], [37529, 3116, 18569, 59325], [2270, 1738, 3770, 1406]]}, {"prop": "father", "subj": "Mark Twain", "obj": "John Marshall Clemens", "pop": 146014, "ids": [1507, 8279, 1379, 5378, 12239, 1485, 1523, 4801, 30731, 9525, 59335, 1643, 72], "a": 7, "n": 5, "alts": ["john marshall clemens"], "dis": [[49196, 2447, 1417, 1379, 25446], [4220, 1738, 1362, 1687, 1350], [1527, 21867, 47260, 29865, 1417]]}, {"prop": "father", "subj": "Tiger Woods", "obj": "Earl Woods", "pop": 137469, "ids": [1507, 8279, 1379, 41141, 37501, 1523, 31677, 37501, 72], "a": 6, "n": 2, "alts": ["earl woods", "earl dennison woods"], "dis": [[10674, 31650], [6132, 48211], [49047, 26576]]}, {"prop": "father", "subj": "Moses", "obj": "Amram", "pop": 131794, "ids": [1507, 8279, 1379, 1445, 8569, 1523, 4220, 4629, 72], "a": 6, "n": 2, "alts": ["amram", "imran", "amran"], "dis": [[31677, 37501], [34478, 22536], [19378, 9114]]}, {"prop": "father", "subj": "Apollo", "obj": "Zeus", "pop": 130900, "ids": [1507, 8279, 1379, 47588, 1523, 28817, 1406, 72], "a": 5, "n": 2, "alts": ["zeus"], "dis": [[22430, 7562], [20452, 11140], [1420, 3000]]}, {"prop": "father", "subj": "George S. Patton", "obj": "George S. Patton", "pop": 130744, "ids": [1507, 8279, 1379, 9595, 1413, 72, 5583, 2232, 1523, 9595, 1413, 72, 5583, 2232, 72], "a": 9, "n": 5, "alts": ["george s. patton", "george smith patton", "george smith patton jr.", "george smith patton ii"], "dis": [[22045, 2785, 1770, 1379, 45900], [9486, 1472, 13461, 7196, 50905], [9114, 1417, 3224, 2978, 59325]]}, {"prop": "father", "subj": "Aung San Suu Kyi", "obj": "Aung San", "pop": 127882, "ids": [1507, 8279, 1379, 1420, 3300, 6312, 24051, 59334, 23069, 59325, 1523, 1420, 3300, 6312, 72], "a": 11, "n": 3, "alts": ["aung san", "general aung san", "bogyoke aung san"], "dis": [[16335, 41171, 40960], [16335, 20838, 11140], [17661, 1430, 27920]]}, {"prop": "father", "subj": "Al Gore", "obj": "Albert Arnold Gore", "pop": 121399, "ids": [1507, 8279, 1379, 2270, 1527, 1512, 1523, 25287, 39861, 1527, 1512, 72], "a": 7, "n": 4, "alts": ["albert arnold gore", "albert arnold gore, sr.", "al gore", "albert arnold \"al\" gore, sr.", "albert a. gore, sr.", "albert gore, sr."], "dis": [[8266, 1690, 1379, 15280], [9595, 1681, 7212, 1347], [2572, 3215, 1367, 2199]]}, {"prop": "father", "subj": "Johann Sebastian Bach", "obj": "Johann Ambrosius Bach", "pop": 120538, "ids": [1507, 8279, 1379, 59147, 53213, 41832, 1523, 59147, 19458, 4940, 4830, 41832, 72], "a": 7, "n": 5, "alts": ["johann ambrosius bach"], "dis": [[22045, 2785, 1770, 1379, 45900], [2572, 17222, 57669, 1394, 1635], [1413, 1412, 1416, 63061, 1448]]}, {"prop": "father", "subj": "David", "obj": "Isa\u00ed", "pop": 111751, "ids": [1507, 8279, 1379, 7524, 1523, 2899, 59324, 60577, 72], "a": 5, "n": 3, "alts": ["isa\u00ed", "yishai", "isai", "issai"], "dis": [[52373, 6119, 59326], [15089, 4088, 5031], [2572, 20419, 1391]]}, {"prop": "father", "subj": "Chelsea Clinton", "obj": "Bill Clinton", "pop": 100666, "ids": [1507, 8279, 1379, 33812, 20355, 1523, 10518, 20355, 72], "a": 6, "n": 2, "alts": ["bill clinton", "william jefferson clinton", "william jefferson blythe iii", "william jefferson blythe", "william j. clinton", "clinton", "william jefferson \"bill\" clinton", "william clinton", "president clinton", "president bill clinton", "kelindun", "william jefferson, iv blythe", "bill klinton", "wjc"], "dis": [[31677, 37501], [28817, 1406], [40195, 19247]]}, {"prop": "father", "subj": "Lily Allen", "obj": "Keith Allen", "pop": 98451, "ids": [1507, 8279, 1379, 58372, 22536, 1523, 34478, 22536, 72], "a": 6, "n": 2, "alts": ["keith allen", "keith philip george allen", "keith howell charles allen"], "dis": [[1527, 10292], [32392, 34970], [15039, 24987]]}, {"prop": "father", "subj": "Rand Paul", "obj": "Ron Paul", "pop": 98029, "ids": [1507, 8279, 1379, 30385, 7821, 1523, 23604, 7821, 72], "a": 6, "n": 2, "alts": ["ron paul", "ronald ernest \"ron\" paul", "ronald ernest paul", "paul, ronald ernest", "paul, ron"], "dis": [[1495, 15337], [28817, 1406], [57699, 1406]]}, {"prop": "father", "subj": "Alexander Graham Bell", "obj": "Alexander Melville Bell", "pop": 94860, "ids": [1507, 8279, 1379, 19488, 27954, 17353, 1523, 19488, 10073, 7009, 17353, 72], "a": 7, "n": 4, "alts": ["alexander melville bell"], "dis": [[2270, 1370, 1374, 1417], [22910, 3089, 1379, 48382], [2572, 3215, 1367, 2199]]}, {"prop": "father", "subj": "Rama", "obj": "Dasharatha", "pop": 93777, "ids": [1507, 8279, 1379, 1490, 4251, 1523, 1479, 2116, 1370, 46601, 72], "a": 6, "n": 4, "alts": ["dasharatha"], "dis": [[19121, 4493, 55581, 15852], [9833, 59321, 8912, 11223], [23628, 1469, 5826, 12836]]}, {"prop": "father", "subj": "Beau Biden", "obj": "Joe Biden", "pop": 93669, "ids": [1507, 8279, 1379, 3594, 2874, 39926, 1523, 15039, 39926, 72], "a": 7, "n": 2, "alts": ["joe biden", "joseph biden", "joseph r. biden", "joseph r. biden jr.", "joseph r. biden, jr.", "biden", "joey biden", "jrb", "potus 46", "joe r. biden jr.", "joseph robinette biden"], "dis": [[47864, 14318], [1495, 15337], [16162, 28653]]}, {"prop": "father", "subj": "James Madison", "obj": "James Madison, Sr.", "pop": 93306, "ids": [1507, 8279, 1379, 8266, 32892, 1523, 8266, 32892, 59342, 31821, 72, 72], "a": 6, "n": 5, "alts": ["james madison, sr.", "james madison sr."], "dis": [[8049, 1402, 1398, 1406, 4401], [9595, 1413, 72, 5583, 2232], [4544, 44623, 5546, 31821, 72]]}, {"prop": "father", "subj": "David Cassidy", "obj": "Jack Cassidy", "pop": 88800, "ids": [1507, 8279, 1379, 7524, 27475, 29897, 1523, 8710, 27475, 29897, 72], "a": 7, "n": 3, "alts": ["jack cassidy"], "dis": [[11374, 1408, 2118], [1494, 7434, 32232], [20119, 4814, 1411]]}, {"prop": "father", "subj": "Thor", "obj": "Odin", "pop": 87051, "ids": [1507, 8279, 1379, 35689, 1523, 26313, 1347, 72], "a": 5, "n": 2, "alts": ["odin", "othin", "wodin"], "dis": [[37441, 1350], [15039, 39926], [54199, 19199]]}, {"prop": "father", "subj": "Helen of Troy", "obj": "Zeus", "pop": 86909, "ids": [1507, 8279, 1379, 40297, 1379, 50634, 1523, 28817, 1406, 72], "a": 7, "n": 2, "alts": ["zeus"], "dis": [[5231, 57682], [13187, 55864], [10804, 23423]]}, {"prop": "father", "subj": "Ganesha", "obj": "Shiva", "pop": 85302, "ids": [1507, 8279, 1379, 1527, 11431, 5031, 1523, 2419, 15653, 72], "a": 7, "n": 2, "alts": ["shiva", "rudra", "bhairav", "achintya", "akul", "eeshan", "ekaling", "gaganesh", "gaurinath", "girish", "girik", "girindra", "harendra", "hariaksa", "hemakesh", "jatin", "jayant", "jogesh", "joginder", "jograj", "jwalia", "kashinath", "kedar", "madesh", "madhavan", "mahamani", "mahesh", "mangesh", "mrigasya", "mukesh", "nabhya", "nagesh", "omkarnath", "panchal", "panshul", "purahan", "purajit", "chandra sekharudu", "parameshwara", "pinaki", "bhola", "mahadev", "mahadeva", "niranjan", "siva", "mah\u00e2d\u00eava"], "dis": [[37332, 8662], [3449, 44960], [34362, 5403]]}, {"prop": "father", "subj": "Charles Lindbergh", "obj": "Charles August Lindbergh", "pop": 81718, "ids": [1507, 8279, 1379, 12583, 18857, 1640, 1695, 1523, 12583, 5445, 18857, 1640, 1695, 72], "a": 8, "n": 5, "alts": ["charles august lindbergh", "charles a. lindbergh"], "dis": [[17627, 2978, 6875, 8840, 28040], [9273, 1591, 59322, 60577, 59346], [8266, 32892, 59342, 31821, 72]]}, {"prop": "father", "subj": "Trace Cyrus", "obj": "Billy Ray Cyrus", "pop": 80252, "ids": [1507, 8279, 1379, 48129, 1419, 50173, 1523, 33596, 14628, 1419, 50173, 72], "a": 7, "n": 4, "alts": ["billy ray cyrus", "william ray cyrus"], "dis": [[11238, 1417, 1379, 37313], [46637, 19488, 32058, 1827], [26750, 1490, 25509, 59339]]}, {"prop": "father", "subj": "Elizabeth Cheney", "obj": "Dick Cheney", "pop": 79365, "ids": [1507, 8279, 1379, 19594, 21193, 2825, 1523, 25346, 21193, 2825, 72], "a": 7, "n": 3, "alts": ["dick cheney", "richard bruce cheney", "richard cheney", "richard bruce \"dick\" cheney"], "dis": [[9536, 1382, 43007], [23347, 3013, 59324], [8548, 29390, 1541]]}, {"prop": "father", "subj": "Anil Kapoor", "obj": "Surinder Kapoor", "pop": 79033, "ids": [1507, 8279, 1379, 2426, 1394, 27588, 4039, 1523, 7992, 8357, 27588, 4039, 72], "a": 8, "n": 4, "alts": ["surinder kapoor"], "dis": [[1495, 41815, 8458, 1487], [8266, 1690, 1379, 15280], [1445, 2770, 29374, 4214]]}, {"prop": "father", "subj": "Arjun Sarja", "obj": "Shakti Prasad", "pop": 78970, "ids": [1507, 8279, 1379, 2548, 29193, 13133, 8529, 1523, 2419, 1622, 12401, 3327, 1382, 1422, 72], "a": 8, "n": 6, "alts": ["shakti prasad"], "dis": [[15236, 1350, 1656, 13461, 1379, 25720], [1413, 1430, 7968, 1417, 1379, 18555], [49047, 2583, 1660, 1417, 1358, 11524]]}, {"prop": "father", "subj": "Ronan Farrow", "obj": "Woody Allen", "pop": 76944, "ids": [1507, 8279, 1379, 23604, 1363, 1493, 4033, 1523, 10714, 59339, 22536, 72], "a": 8, "n": 3, "alts": ["woody allen", "allan stewart konigsberg", "allen stewart konigsberg", "heywood allen"], "dis": [[5616, 1822, 1359], [4068, 59332, 3061], [9826, 4286, 1363]]}, {"prop": "father", "subj": "Jacob", "obj": "Isaac", "pop": 76739, "ids": [1507, 8279, 1379, 22200, 1523, 48382, 72], "a": 5, "n": 1, "alts": ["isaac", "yitzchak", "yitzhak", "yizhak", "yitzchaq", "yizhaq", "yitskhak", "yitzchok", "isaac the armenian", "isak"], "dis": [[47588], [20362], [34659]]}, {"prop": "father", "subj": "T.\u00a0S. Eliot", "obj": "Henry Ware Eliot", "pop": 76305, "ids": [1507, 8279, 1379, 1408, 72, 1283, 1249, 59352, 72, 34974, 1411, 1523, 14943, 47049, 34974, 1411, 72], "a": 12, "n": 4, "alts": ["henry ware eliot"], "dis": [[19121, 4493, 55581, 15852], [22857, 1391, 20797, 2445], [10674, 1609, 2092, 59321]]}, {"prop": "father", "subj": "Mary, Princess Royal and Countess of Harewood", "obj": "George V", "pop": 76000, "ids": [1507, 8279, 1379, 8959, 59342, 31849, 10297, 1384, 3449, 1481, 1379, 1491, 1579, 6593, 1523, 9595, 1690, 72], "a": 15, "n": 2, "alts": ["george v", "george frederick ernest albert", "george windsor", "george frederick ernest albert windsor", "george saxe-coburg-gotha", "george frederick ernest albert saxe-coburg-gotha", "king george v", "king of great britain george v", "v, king of great britain george", "george v of the united kingdom"], "dis": [[1527, 10292], [40195, 19247], [27229, 12917]]}, {"prop": "father", "subj": "Hera", "obj": "Cronus", "pop": 75520, "ids": [1507, 8279, 1379, 1491, 4343, 1523, 57699, 1406, 72], "a": 6, "n": 2, "alts": ["cronus", "kronos"], "dis": [[9595, 1690], [19378, 9114], [23604, 7821]]}, {"prop": "father", "subj": "Anubis", "obj": "Osiris", "pop": 75012, "ids": [1507, 8279, 1379, 2426, 1484, 1360, 1523, 16294, 49525, 72], "a": 7, "n": 2, "alts": ["osiris"], "dis": [[1527, 3215], [31808, 19618], [42369, 34970]]}, {"prop": "father", "subj": "Dick Grayson", "obj": "Batman", "pop": 74659, "ids": [1507, 8279, 1379, 25346, 3509, 1829, 1350, 1523, 48457, 72], "a": 8, "n": 1, "alts": ["batman", "matches malone", "sir hemingford grey", "mordecai wayne", "wayne, bruce", "bruce wayne", "the bat-man", "the caped crusader", "the dark knight", "the world's greatest detective", "the insider", "the batman", "the bat", "bat-man", "caped crusader"], "dis": [[19378], [7524], [47588]]}, {"prop": "father", "subj": "Talia Balsam", "obj": "Martin Balsam", "pop": 73982, "ids": [1507, 8279, 1379, 19045, 1627, 1469, 1950, 1398, 1523, 12489, 1469, 1950, 1398, 72], "a": 9, "n": 4, "alts": ["martin balsam", "martin henry balsam"], "dis": [[15054, 1417, 1379, 31355], [44799, 4401, 1379, 11321], [9812, 4401, 1379, 8360]]}, {"prop": "father", "subj": "Odysseus", "obj": "Laertes", "pop": 73587, "ids": [1507, 8279, 1379, 26313, 1729, 1435, 1406, 1523, 5616, 1822, 1359, 72], "a": 8, "n": 3, "alts": ["laertes"], "dis": [[9595, 13836, 2450], [1413, 10292, 2350], [1494, 1423, 1684]]}, {"prop": "father", "subj": "Horus", "obj": "Osiris", "pop": 71001, "ids": [1507, 8279, 1379, 13993, 1406, 1523, 16294, 49525, 72], "a": 6, "n": 2, "alts": ["osiris"], "dis": [[13187, 55864], [6132, 48211], [42369, 34970]]}, {"prop": "father", "subj": "Chris Penn", "obj": "Leo Penn", "pop": 70096, "ids": [1507, 8279, 1379, 13765, 34970, 1523, 42369, 34970, 72], "a": 6, "n": 2, "alts": ["leo penn", "leonard francis penn"], "dis": [[1646, 51100], [23085, 3136], [1527, 3215]]}, {"prop": "father", "subj": "Isis", "obj": "Geb", "pop": 67359, "ids": [1507, 8279, 1379, 2899, 1360, 1523, 1527, 3215, 72], "a": 6, "n": 2, "alts": ["geb", "seb", "keb"], "dis": [[23604, 7821], [31808, 19618], [9595, 31650]]}, {"prop": "father", "subj": "Demeter", "obj": "Cronus", "pop": 61181, "ids": [1507, 8279, 1379, 6572, 4493, 1523, 57699, 1406, 72], "a": 6, "n": 2, "alts": ["cronus", "kronos"], "dis": [[19378, 9114], [32392, 34970], [10058, 11690]]}, {"prop": "father", "subj": "Nebuchadnezzar II", "obj": "Nabopolassar", "pop": 60231, "ids": [1507, 8279, 1379, 1515, 3215, 2337, 1422, 1861, 11679, 1370, 4401, 1523, 1515, 1456, 29299, 1537, 1370, 72], "a": 12, "n": 5, "alts": ["nabopolassar"], "dis": [[3227, 13746, 1445, 3865, 1359], [17753, 34232, 59322, 54199, 8683], [9753, 2572, 9725, 13313, 59337]]}, {"prop": "father", "subj": "Paul I of Russia", "obj": "Peter III of Russia", "pop": 57543, "ids": [1507, 8279, 1379, 7821, 1417, 1379, 11114, 1523, 11238, 11223, 1379, 11114, 72], "a": 8, "n": 4, "alts": ["peter iii of russia", "charles peter ulrich of holstein-gottorp", "karl peter ulrich"], "dis": [[13344, 2341, 26728, 3582], [1479, 2116, 1370, 46601], [2270, 1370, 1374, 1417]]}, {"prop": "father", "subj": "Linus Torvalds", "obj": "Nils Torvalds", "pop": 56679, "ids": [1507, 8279, 1379, 8351, 1406, 9429, 2260, 5745, 1523, 1515, 7933, 9429, 2260, 5745, 72], "a": 9, "n": 5, "alts": ["nils torvalds"], "dis": [[2426, 1660, 5816, 2453, 2160], [2572, 17222, 57669, 1394, 1635], [29388, 3300, 63, 23622, 12917]]}, {"prop": "father", "subj": "Edward the Elder", "obj": "Alfred the Great", "pop": 55823, "ids": [1507, 8279, 1379, 18223, 1358, 34875, 1349, 1523, 34683, 1358, 7566, 72], "a": 8, "n": 3, "alts": ["alfred the great", "king of wessex aelfred", "king of england alfred", "alfred the wise", "king of england aelfred", "king of england alfred the great", "alfred the great of wessex", "alfred", "alfred king", "king alfred", "alfred den store"], "dis": [[35296, 1528, 27966], [28143, 10457, 1528], [41266, 58442, 1443]]}, {"prop": "father", "subj": "Alexander I of Russia", "obj": "Paul I of Russia", "pop": 55331, "ids": [1507, 8279, 1379, 19488, 1417, 1379, 11114, 1523, 7821, 1417, 1379, 11114, 72], "a": 8, "n": 4, "alts": ["paul i of russia", "pavel i", "emperor of russia paul i", "zar di russia pavel i", "zar paul i. russland", "emperor of russia paul", "empereur de russie paul i", "pablo primero", "tsaar van rusland pavel i", "emperor of russia pavel petrovich", "paul the first of russia", "emperor of russia pavel i", "emperor of russia paul petrovitch", "paul i"], "dis": [[28082, 1408, 1371, 28574], [25214, 13461, 1379, 8360], [1452, 3856, 59398, 11223]]}, {"prop": "father", "subj": "Stephen of England", "obj": "Stephen", "pop": 52061, "ids": [1507, 8279, 1379, 19378, 1379, 7610, 1523, 19378, 72], "a": 7, "n": 1, "alts": ["stephen", "stephen ii henry", "stephen ii, count of blois", "\u00e9tienne henri", "etienne henri", "stephen ii"], "dis": [[7524], [48382], [34659]]}, {"prop": "father", "subj": "Steve Kerr", "obj": "Malcolm Kerr", "pop": 51795, "ids": [1507, 8279, 1379, 15816, 1681, 1969, 1523, 7341, 52465, 1681, 1969, 72], "a": 7, "n": 4, "alts": ["malcolm kerr", "malcolm h. kerr", "malcolm hooper kerr"], "dis": [[23020, 1483, 2685, 1439], [26750, 1490, 25509, 59339], [23628, 1469, 5826, 12836]]}, {"prop": "father", "subj": "Goldust", "obj": "Dusty Rhodes", "pop": 51174, "ids": [1507, 8279, 1379, 8250, 1569, 1523, 1479, 56407, 54226, 72], "a": 6, "n": 3, "alts": ["dusty rhodes", "the american dream", "virgil riley runnels", "virgil runnels"], "dis": [[27739, 21890, 12941], [9826, 4286, 1363], [3868, 9391, 16294]]}, {"prop": "father", "subj": "Arjuna", "obj": "Indra", "pop": 50033, "ids": [1507, 8279, 1379, 2548, 59392, 12406, 1523, 2764, 1765, 72], "a": 7, "n": 2, "alts": ["indra", "ahih\u00e1n", "indran", "ahihan", "indiran", "meghavahana", "vritrahan", "sakra", "sakka", "vajrapani", "vasava", "\u0907\u0928\u094d\u0926\u094d\u0930"], "dis": [[27145, 4417], [4220, 4629], [13187, 49740]]}, {"prop": "father", "subj": "Joseph", "obj": "Jacob", "pop": 48496, "ids": [1507, 8279, 1379, 15089, 1523, 22200, 72], "a": 5, "n": 1, "alts": ["jacob", "israel", "ya'akov", "ya'aqov", "yaakov", "yisrael"], "dis": [[34659], [15089], [19378]]}, {"prop": "genre", "subj": "The Rolling Stones", "obj": "rock music", "pop": 427893, "ids": [1507, 51882, 1716, 3783, 21933, 1385, 1358, 25979, 1379, 7653, 4264, 72], "a": 9, "n": 2, "alts": ["rock music", "rock and roll", "rock", "rock", "blues rock", "blues rock", "psychedelic rock", "acid rock", "psy rock", "psyrock", "psychedelia", "rhythm and blues", "r&b", "rnb", "rnb", "r and b", "r & b", "country rock", "country rock", "rock and roll", "rock 'n roll", "rock 'n' roll", "rock n' roll", "rock & roll", "rock&roll", "hard rock", "country music", "country and western", "country & western", "country", "nashville sound", "blues", "blues music"], "dis": [[11312, 8252], [8322, 4451], [8322, 8252]]}, {"prop": "genre", "subj": "Prince", "obj": "rock music", "pop": 380230, "ids": [18967, 21933, 1385, 1358, 25979, 1379, 7653, 4264, 72], "a": 6, "n": 2, "alts": ["rock music", "rock and roll", "rock", "rock", "funk", "funk", "pop music", "pop", "pop", "rhythm and blues", "r&b", "rnb", "rnb", "r and b", "r & b", "minneapolis sound"], "dis": [[33149, 4841], [26844, 7653], [4537, 14780]]}, {"prop": "genre", "subj": "Will Smith", "obj": "hip hop music", "pop": 345519, "ids": [9878, 10459, 21933, 1385, 1358, 25979, 1379, 17506, 7083, 4264, 72], "a": 7, "n": 3, "alts": ["hip hop music", "hip-hop music"], "dis": [[19478, 2326, 3512], [56093, 1349, 4101], [2385, 63, 39250]]}, {"prop": "genre", "subj": "Paris Hilton", "obj": "pop music", "pop": 296247, "ids": [11225, 56168, 21933, 1385, 1358, 25979, 1379, 2744, 4264, 72], "a": 7, "n": 2, "alts": ["pop music", "pop", "pop"], "dis": [[2197, 29563], [54615, 32779], [5678, 4114]]}, {"prop": "genre", "subj": "The Matrix", "obj": "science fiction film", "pop": 291114, "ids": [1507, 23560, 21933, 1385, 1358, 25979, 1379, 7517, 20307, 4101, 72], "a": 7, "n": 3, "alts": ["cyberpunk", "action film", "action movie", "film action", "movie action", "action", "science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[44388, 1454, 10222], [1414, 39755, 8252], [9406, 13186, 4264]]}, {"prop": "genre", "subj": "Doctor Who", "obj": "science fiction television series", "pop": 289775, "ids": [23158, 14253, 21933, 1385, 1358, 25979, 1379, 7517, 20307, 7463, 3512, 72], "a": 7, "n": 4, "alts": ["science fiction", "sf", "scifi", "sci fi", "sci-fi", "science-fiction", "sci fi", "sciencefiction", "science fiction television series", "science fiction tv", "science fiction series", "science fiction television", "science fiction on television"], "dis": [[1933, 63, 10061, 45726], [2385, 63, 15138, 4378], [18372, 1958, 3299, 6806]]}, {"prop": "genre", "subj": "Star Wars", "obj": "space opera", "pop": 288624, "ids": [10058, 19016, 21933, 1385, 1358, 25979, 1379, 3597, 29686, 72], "a": 7, "n": 2, "alts": ["science fiction", "sf", "scifi", "sci fi", "sci-fi", "science-fiction", "sci fi", "sciencefiction", "space opera", "space operas"], "dis": [[26422, 3512], [7663, 4264], [26844, 7653]]}, {"prop": "genre", "subj": "Alien", "obj": "horror film", "pop": 273178, "ids": [2270, 7607, 21933, 1385, 1358, 25979, 1379, 27871, 4101, 72], "a": 7, "n": 2, "alts": ["horror film", "horror movie", "science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[33149, 4841], [4537, 14780], [11519, 7463]]}, {"prop": "genre", "subj": "Nineteen Eighty-Four", "obj": "political fiction", "pop": 237868, "ids": [1515, 29973, 13727, 28185, 59339, 63, 42700, 21933, 1385, 1358, 25979, 1379, 7173, 20307, 72], "a": 12, "n": 2, "alts": ["political fiction"], "dis": [[4114, 4264], [26874, 8589], [45287, 19538]]}, {"prop": "genre", "subj": "Avatar", "obj": "science fiction film", "pop": 237293, "ids": [9383, 12361, 21933, 1385, 1358, 25979, 1379, 7517, 20307, 4101, 72], "a": 7, "n": 3, "alts": ["science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[1441, 59340, 1464], [51275, 1466, 9406], [1544, 1466, 4101]]}, {"prop": "genre", "subj": "Yoko Ono", "obj": "experimental music", "pop": 201934, "ids": [1788, 35024, 3154, 59326, 21933, 1385, 1358, 25979, 1379, 7663, 4264, 72], "a": 9, "n": 2, "alts": ["fluxus", "neo-dada", "fluxus collective", "rock music", "rock and roll", "rock", "rock", "experimental music", "experimental", "pop music", "pop", "pop"], "dis": [[33149, 4101], [12903, 48514], [1357, 51843]]}, {"prop": "genre", "subj": "The Thing", "obj": "horror film", "pop": 195702, "ids": [1507, 47849, 21933, 1385, 1358, 25979, 1379, 27871, 4101, 72], "a": 7, "n": 2, "alts": ["horror film", "horror movie", "science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[3072, 1547], [3538, 4132], [19538, 4264]]}, {"prop": "genre", "subj": "Billy Joel", "obj": "rock music", "pop": 190041, "ids": [33596, 46319, 21933, 1385, 1358, 25979, 1379, 7653, 4264, 72], "a": 7, "n": 2, "alts": ["rock music", "rock and roll", "rock", "rock"], "dis": [[13914, 2744], [45680, 44726], [25979, 14780]]}, {"prop": "genre", "subj": "Lost", "obj": "science fiction television series", "pop": 154447, "ids": [35739, 21933, 1385, 1358, 25979, 1379, 7517, 20307, 7463, 3512, 72], "a": 6, "n": 4, "alts": ["science fiction television series", "science fiction tv", "science fiction series", "science fiction television", "science fiction on television", "mystery fiction", "mysteries", "mystery", "mystery books", "mystery literature", "detective and mystery fiction"], "dis": [[3810, 1738, 6956, 8252], [4487, 63, 10061, 45726], [10316, 31190, 3987, 8252]]}, {"prop": "genre", "subj": "Suits", "obj": "legal drama", "pop": 153467, "ids": [1413, 11283, 21933, 1385, 1358, 25979, 1379, 7440, 14001, 72], "a": 7, "n": 2, "alts": ["legal drama", "courtroom drama", "comedy-drama", "dramedy", "comedic drama", "tragicomedy", "seriocomedy", "comedy drama", "dramatic comedy", "comedy-drama, dramedy"], "dis": [[14598, 4101], [3597, 29686], [4114, 4264]]}, {"prop": "genre", "subj": "Gorillaz", "obj": "reggae", "pop": 153105, "ids": [41622, 1487, 2215, 21933, 1385, 1358, 25979, 1379, 1936, 59338, 3770, 72], "a": 8, "n": 3, "alts": ["alternative rock", "alternative music", "alt-rock", "alternative", "alt rock", "pop/rock", "alternative rock", "reggae", "reggae music"], "dis": [[9406, 13186, 4264], [1392, 2049, 3008], [2998, 20239, 4101]]}, {"prop": "genre", "subj": "Avatar: The Last Airbender", "obj": "animated series", "pop": 152913, "ids": [9383, 12361, 59358, 1507, 12342, 6147, 59340, 3865, 21933, 1385, 1358, 25979, 1379, 26422, 3512, 72], "a": 13, "n": 2, "alts": ["animated series", "cartoon series", "animated tv series", "animated television series", "animated television show", "animated tv show", "animation series", "animation television series", "animation tv series", "animation show", "animation television show", "animation tv show"], "dis": [[4101, 7215], [1445, 8958], [11524, 50229]]}, {"prop": "genre", "subj": "Back to the Future", "obj": "science fiction film", "pop": 150710, "ids": [10203, 1385, 1358, 19186, 21933, 1385, 1358, 25979, 1379, 7517, 20307, 4101, 72], "a": 9, "n": 3, "alts": ["comedy film", "comedy movie", "science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[2998, 20239, 4101], [1427, 7022, 59325], [16685, 10524, 4101]]}, {"prop": "genre", "subj": "League of Legends", "obj": "multiplayer online battle arena", "pop": 131468, "ids": [5668, 1379, 8654, 2623, 21933, 1385, 1358, 25979, 1379, 5295, 10415, 3734, 13550, 33407, 72], "a": 9, "n": 5, "alts": ["multiplayer online battle arena", "moba", "arts", "action real-time strategy", "hero brawler", "team brawler", "aeon of strife"], "dis": [[45287, 63, 1644, 15227, 7653], [1490, 10287, 2197, 2727, 59329], [2503, 59326, 63, 52767, 9573]]}, {"prop": "genre", "subj": "Grease", "obj": "musical film", "pop": 127970, "ids": [26094, 1582, 21933, 1385, 1358, 25979, 1379, 13229, 4101, 72], "a": 7, "n": 2, "alts": ["musical film", "musical movie", "romantic comedy", "romedy", "romcom"], "dis": [[18167, 3299], [2503, 59326], [5118, 8252]]}, {"prop": "genre", "subj": "Johann Sebastian Bach", "obj": "Western classical music", "pop": 120538, "ids": [59147, 53213, 41832, 21933, 1385, 1358, 25979, 1379, 9406, 13186, 4264, 72], "a": 8, "n": 3, "alts": ["baroque music", "baroque", "baroque music", "baroque period", "music of the baroque", "baroque era", "western classical music", "western art music", "european classical music", "classical music"], "dis": [[12649, 2116, 8252], [1488, 1442, 6932], [16685, 10524, 4101]]}, {"prop": "genre", "subj": "Troy", "obj": "war film", "pop": 118889, "ids": [50634, 21933, 1385, 1358, 25979, 1379, 7092, 4101, 72], "a": 6, "n": 2, "alts": ["war film", "war movie"], "dis": [[2503, 59326], [24008, 8252], [13914, 2744]]}, {"prop": "genre", "subj": "Simon Le Bon", "obj": "pop music", "pop": 116553, "ids": [21861, 3013, 16007, 21933, 1385, 1358, 25979, 1379, 2744, 4264, 72], "a": 8, "n": 2, "alts": ["pop music", "pop", "pop"], "dis": [[8322, 7653], [10990, 4264], [45528, 7653]]}, {"prop": "genre", "subj": "Beck", "obj": "alternative rock", "pop": 116390, "ids": [32422, 21933, 1385, 1358, 25979, 1379, 8322, 7653, 72], "a": 6, "n": 2, "alts": ["alternative rock", "alternative music", "alt-rock", "alternative", "alt rock", "pop/rock", "alternative rock", "anti-folk", "antifolk", "unfolk", "anti-folk"], "dis": [[26422, 3512], [19065, 4114], [5678, 4114]]}, {"prop": "genre", "subj": "The Challenge", "obj": "reality television", "pop": 114391, "ids": [1507, 21498, 21933, 1385, 1358, 25979, 1379, 11519, 7463, 72], "a": 7, "n": 2, "alts": ["reality television", "reality tv", "reality television program", "reality tv program", "reality television show", "television reality program", "television reality show", "tv reality program", "tv reality show"], "dis": [[1577, 53900], [26422, 3512], [8322, 8252]]}, {"prop": "genre", "subj": "Rush", "obj": "progressive rock", "pop": 112510, "ids": [44024, 21933, 1385, 1358, 25979, 1379, 19065, 7653, 72], "a": 6, "n": 2, "alts": ["progressive rock", "prog-rock", "prog", "classical rock", "symphonic rock", "art rock", "progressive pop", "prog rock", "progressive rock", "hard rock"], "dis": [[6035, 13658], [16662, 8252], [2744, 4264]]}, {"prop": "genre", "subj": "The Police", "obj": "rock music", "pop": 112137, "ids": [1507, 13438, 21933, 1385, 1358, 25979, 1379, 7653, 4264, 72], "a": 7, "n": 2, "alts": ["rock music", "rock and roll", "rock", "rock"], "dis": [[13914, 4101], [3516, 3299], [3987, 8252]]}, {"prop": "genre", "subj": "Oasis", "obj": "Britpop", "pop": 110761, "ids": [1549, 8286, 21933, 1385, 1358, 25979, 1379, 5045, 8082, 72], "a": 7, "n": 2, "alts": ["britpop", "british pop", "brit-pop"], "dis": [[1560, 1659], [13431, 34085], [1469, 8383]]}, {"prop": "genre", "subj": "Tool", "obj": "progressive rock", "pop": 101162, "ids": [19714, 21933, 1385, 1358, 25979, 1379, 19065, 7653, 72], "a": 6, "n": 2, "alts": ["art rock", "art rock", "progressive rock", "prog-rock", "prog", "classical rock", "symphonic rock", "art rock", "progressive pop", "prog rock", "progressive rock"], "dis": [[5602, 4264], [18546, 27871], [1427, 10134]]}, {"prop": "genre", "subj": "They Live", "obj": "science fiction film", "pop": 90266, "ids": [3330, 13588, 21933, 1385, 1358, 25979, 1379, 7517, 20307, 4101, 72], "a": 7, "n": 3, "alts": ["horror film", "horror movie", "science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[10443, 4001, 3299], [21792, 2463, 5549], [56093, 1349, 4101]]}, {"prop": "genre", "subj": "The Band", "obj": "rock music", "pop": 85780, "ids": [1507, 16745, 21933, 1385, 1358, 25979, 1379, 7653, 4264, 72], "a": 7, "n": 2, "alts": ["rock music", "rock and roll", "rock", "rock", "folk rock", "folk rock", "roots rock", "roots rock", "blues rock", "blues rock", "alternative country", "alt-country", "insurgent country", "americana", "alt-country", "country rock", "country rock", "country music", "country and western", "country & western", "country", "nashville sound"], "dis": [[28089, 6671], [27271, 8077], [3072, 1547]]}, {"prop": "genre", "subj": "King Crimson", "obj": "progressive rock", "pop": 84330, "ids": [6132, 32440, 2162, 21933, 1385, 1358, 25979, 1379, 19065, 7653, 72], "a": 8, "n": 2, "alts": ["art rock", "art rock", "progressive rock", "prog-rock", "prog", "classical rock", "symphonic rock", "art rock", "progressive pop", "prog rock", "progressive rock"], "dis": [[2503, 59326], [38323, 4101], [18363, 3299]]}, {"prop": "genre", "subj": "Heart", "obj": "rock music", "pop": 83062, "ids": [17210, 21933, 1385, 1358, 25979, 1379, 7653, 4264, 72], "a": 6, "n": 2, "alts": ["rock music", "rock and roll", "rock", "rock"], "dis": [[8322, 13431], [1577, 53900], [33149, 4101]]}, {"prop": "genre", "subj": "The Fly", "obj": "horror film", "pop": 82626, "ids": [1507, 26728, 21933, 1385, 1358, 25979, 1379, 27871, 4101, 72], "a": 7, "n": 2, "alts": ["horror film", "horror movie", "science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[3516, 3299], [3072, 1547], [11519, 7463]]}, {"prop": "genre", "subj": "Battlefield", "obj": "first-person shooter", "pop": 80560, "ids": [19585, 3548, 21933, 1385, 1358, 25979, 1379, 1933, 63, 10061, 45726, 72], "a": 7, "n": 4, "alts": ["first-person shooter", "fps", "first person shooter video games", "first-person shooter video game", "fps", "fps game", "fps video game", "first-person shooter game"], "dis": [[4487, 63, 10061, 45726], [2385, 63, 1755, 29563], [8944, 1772, 1454, 1712]]}, {"prop": "genre", "subj": "Carrie", "obj": "film adaptation", "pop": 80170, "ids": [3684, 9477, 21933, 1385, 1358, 25979, 1379, 4101, 20443, 72], "a": 7, "n": 2, "alts": ["film adaptation", "based upon", "adapted for the screen", "cinematic adaptation", "horror film", "horror movie"], "dis": [[1715, 8546], [4537, 14780], [33149, 4101]]}, {"prop": "genre", "subj": "Nine Inch Nails", "obj": "industrial metal", "pop": 78652, "ids": [38752, 1646, 1439, 1515, 2951, 21933, 1385, 1358, 25979, 1379, 11312, 8252, 72], "a": 10, "n": 2, "alts": ["alternative rock", "alternative music", "alt-rock", "alternative", "alt rock", "pop/rock", "alternative rock", "ambient music", "ambient", "ambient", "industrial metal", "industrial metal", "industrial music", "industrial", "industrial rock"], "dis": [[9350, 8252], [2197, 29563], [1357, 51843]]}, {"prop": "genre", "subj": "The Karate Kid", "obj": "action film", "pop": 77948, "ids": [1507, 12065, 1446, 32535, 21933, 1385, 1358, 25979, 1379, 3516, 4101, 72], "a": 9, "n": 2, "alts": ["martial arts film", "action film", "action movie", "film action", "movie action", "action"], "dis": [[15652, 20307], [12029, 14001], [4575, 1604]]}, {"prop": "genre", "subj": "The Mist", "obj": "monster film", "pop": 76264, "ids": [1507, 49567, 21933, 1385, 1358, 25979, 1379, 38299, 4101, 72], "a": 7, "n": 2, "alts": ["monster film", "monster movie", "creature feature", "giant monster film", "monster horror films", "horror film", "horror movie"], "dis": [[45143, 7653], [4258, 1437], [4575, 1604]]}, {"prop": "genre", "subj": "Chris Jericho", "obj": "rock music", "pop": 75216, "ids": [13765, 9196, 1620, 59326, 21933, 1385, 1358, 25979, 1379, 7653, 4264, 72], "a": 9, "n": 2, "alts": ["rock music", "rock and roll", "rock", "rock"], "dis": [[5482, 3299], [4537, 14780], [7663, 7653]]}, {"prop": "genre", "subj": "Inside Out", "obj": "adventure film", "pop": 74976, "ids": [33044, 6249, 21933, 1385, 1358, 25979, 1379, 18167, 4101, 72], "a": 7, "n": 2, "alts": ["adventure film", "adventure movie", "adventure", "comedy-drama", "dramedy", "comedic drama", "tragicomedy", "seriocomedy", "comedy drama", "dramatic comedy", "comedy-drama, dramedy"], "dis": [[11312, 8252], [1560, 1659], [27862, 4101]]}, {"prop": "genre", "subj": "The Little Prince", "obj": "novella", "pop": 72579, "ids": [1507, 14404, 18967, 21933, 1385, 1358, 25979, 1379, 27271, 8077, 72], "a": 8, "n": 2, "alts": ["novella"], "dis": [[18167, 4101], [2446, 4101], [7463, 15899]]}, {"prop": "genre", "subj": "Steps", "obj": "pop music", "pop": 71394, "ids": [32255, 21933, 1385, 1358, 25979, 1379, 2744, 4264, 72], "a": 6, "n": 2, "alts": ["pop music", "pop", "pop"], "dis": [[7663, 4264], [9350, 8252], [11519, 7463]]}, {"prop": "genre", "subj": "RCA Records", "obj": "pop music", "pop": 65507, "ids": [1490, 6253, 11540, 21933, 1385, 1358, 25979, 1379, 2744, 4264, 72], "a": 8, "n": 2, "alts": ["rock music", "rock and roll", "rock", "rock", "pop music", "pop", "pop", "rhythm and blues", "r&b", "rnb", "rnb", "r and b", "r & b", "jazz", "jazz music", "jass", "jas", "jaz", "jazz"], "dis": [[5482, 3299], [7663, 4101], [13914, 2744]]}, {"prop": "genre", "subj": "The Wall", "obj": "progressive rock", "pop": 65420, "ids": [1507, 11796, 21933, 1385, 1358, 25979, 1379, 19065, 7653, 72], "a": 7, "n": 2, "alts": ["progressive rock", "prog-rock", "prog", "classical rock", "symphonic rock", "art rock", "progressive pop", "prog rock", "progressive rock"], "dis": [[7463, 15899], [4575, 1604], [19065, 8252]]}, {"prop": "genre", "subj": "GG Allin", "obj": "hardcore punk", "pop": 61786, "ids": [1527, 59394, 3030, 1347, 21933, 1385, 1358, 25979, 1379, 45680, 44726, 72], "a": 9, "n": 2, "alts": ["hardcore punk", "hxc", "punk/hc", "hardcore", "hardcore punk", "punk rock", "punk", "punk music"], "dis": [[26844, 7653], [14598, 4101], [45528, 2744]]}, {"prop": "genre", "subj": "The Internship", "obj": "comedy film", "pop": 61374, "ids": [1507, 4387, 6073, 21933, 1385, 1358, 25979, 1379, 15899, 4101, 72], "a": 8, "n": 2, "alts": ["comedy film", "comedy movie"], "dis": [[4136, 59346], [16662, 8252], [45528, 7653]]}, {"prop": "genre", "subj": "Godzilla", "obj": "B movie", "pop": 61269, "ids": [6274, 59398, 8034, 21933, 1385, 1358, 25979, 1379, 1469, 8383, 72], "a": 8, "n": 2, "alts": ["b movie", "b-movie", "b film", "b-film", "b picture", "b-picture"], "dis": [[1427, 10134], [5045, 8082], [54615, 32779]]}, {"prop": "genre", "subj": "Monk", "obj": "comedy-drama", "pop": 59984, "ids": [5246, 59346, 21933, 1385, 1358, 25979, 1379, 15899, 63, 9283, 4251, 72], "a": 7, "n": 4, "alts": ["police procedural", "comedy-drama", "dramedy", "comedic drama", "tragicomedy", "seriocomedy", "comedy drama", "dramatic comedy", "comedy-drama, dramedy"], "dis": [[1346, 15243, 52549, 59324], [30154, 5033, 48230, 7653], [4487, 63, 10061, 45726]]}, {"prop": "genre", "subj": "Tales", "obj": "role-playing video game", "pop": 59362, "ids": [1408, 3337, 21933, 1385, 1358, 25979, 1379, 3470, 63, 36185, 4001, 3299, 72], "a": 7, "n": 5, "alts": ["role-playing video game", "rpg", "rpvg", "role playing video game", "role-playing game", "role playing video games", "crpg", "computer rpg", "computer role-playing video game", "rp video game"], "dis": [[45287, 63, 17020, 5033, 39676], [1675, 59326, 63, 59341, 1464], [45287, 63, 1644, 15227, 7653]]}, {"prop": "genre", "subj": "REO Speedwagon", "obj": "rock music", "pop": 57846, "ids": [6097, 59383, 21778, 59341, 7443, 21933, 1385, 1358, 25979, 1379, 7653, 4264, 72], "a": 10, "n": 2, "alts": ["rock music", "rock and roll", "rock", "rock"], "dis": [[1445, 8958], [1560, 1659], [11312, 7653]]}, {"prop": "genre", "subj": "Red Velvet", "obj": "K-pop", "pop": 56793, "ids": [5430, 25126, 20315, 21933, 1385, 1358, 25979, 1379, 1681, 63, 8082, 72], "a": 8, "n": 3, "alts": ["k-pop", "k-pop", "k pop", "kpop", "korean pop", "kpop", "k-pop"], "dis": [[44388, 1454, 10222], [1366, 3600, 1627], [1414, 39755, 8252]]}, {"prop": "genre", "subj": "Sonic Youth", "obj": "noise rock", "pop": 56364, "ids": [1413, 6956, 20647, 21933, 1385, 1358, 25979, 1379, 10733, 7653, 72], "a": 8, "n": 2, "alts": ["alternative rock", "alternative music", "alt-rock", "alternative", "alt rock", "pop/rock", "alternative rock", "experimental rock", "avant-garde rock", "avant rock", "noise rock", "noise punk", "indie rock", "independent rock", "indie rock", "no wave"], "dis": [[2446, 7653], [3554, 5024], [45726, 3299]]}, {"prop": "genre", "subj": "The Host", "obj": "science fiction film", "pop": 55711, "ids": [1507, 21853, 21933, 1385, 1358, 25979, 1379, 7517, 20307, 4101, 72], "a": 7, "n": 3, "alts": ["monster film", "monster movie", "creature feature", "giant monster film", "monster horror films", "science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[2385, 63, 17504], [2385, 63, 39250], [13517, 1464, 1464]]}, {"prop": "genre", "subj": "Great Expectations", "obj": "fiction", "pop": 54249, "ids": [7566, 46486, 1641, 21933, 1385, 1358, 25979, 1379, 20307, 72], "a": 8, "n": 1, "alts": ["fiction", "fictional"], "dis": [[43178], [4537], [29686]]}, {"prop": "genre", "subj": "No Doubt", "obj": "ska", "pop": 54047, "ids": [3721, 15088, 4943, 21933, 1385, 1358, 25979, 1379, 2709, 59324, 72], "a": 8, "n": 2, "alts": ["alternative rock", "alternative music", "alt-rock", "alternative", "alt rock", "pop/rock", "alternative rock", "rock music", "rock and roll", "rock", "rock", "punk rock", "punk", "punk music", "pop music", "pop", "pop", "ska punk", "ska-core", "punk ska", "pop rock", "pop/rock", "pop-rock", "ska", "ska music"], "dis": [[24008, 8252], [51412, 4101], [12029, 14001]]}, {"prop": "genre", "subj": "Garbage", "obj": "alternative rock", "pop": 53201, "ids": [13632, 29944, 21933, 1385, 1358, 25979, 1379, 8322, 7653, 72], "a": 7, "n": 2, "alts": ["alternative rock", "alternative music", "alt-rock", "alternative", "alt rock", "pop/rock", "alternative rock"], "dis": [[7663, 7653], [4136, 59346], [14598, 4841]]}, {"prop": "genre", "subj": "General Hospital", "obj": "soap opera", "pop": 53023, "ids": [5297, 11351, 21933, 1385, 1358, 25979, 1379, 29766, 29686, 72], "a": 7, "n": 2, "alts": ["soap opera", "soap", "soapie", "soapopera"], "dis": [[29958, 7653], [7440, 14001], [38299, 4101]]}, {"prop": "genre", "subj": "All Good Things", "obj": "romance film", "pop": 52993, "ids": [3030, 9158, 24646, 21933, 1385, 1358, 25979, 1379, 33149, 4101, 72], "a": 8, "n": 2, "alts": ["romance film", "romance genre", "romantic film"], "dis": [[18363, 3299], [2744, 44726], [4101, 20443]]}, {"prop": "genre", "subj": "The Rite", "obj": "thriller", "pop": 52880, "ids": [1507, 1490, 1671, 21933, 1385, 1358, 25979, 1379, 43178, 72], "a": 8, "n": 1, "alts": ["thriller", "suspense", "thriller television program", "thriller tv program", "thriller television show", "thriller tv show", "suspense television program", "suspense tv program", "suspense television show", "suspense tv show", "horror film", "horror movie"], "dis": [[26874], [34824], [36756]]}, {"prop": "genre", "subj": "Pok\u00e9mon Red and Blue", "obj": "role-playing video game", "pop": 52157, "ids": [31277, 57867, 5430, 1384, 11003, 21933, 1385, 1358, 25979, 1379, 3470, 63, 36185, 4001, 3299, 72], "a": 10, "n": 5, "alts": ["role-playing video game", "rpg", "rpvg", "role playing video game", "role-playing game", "role playing video games", "crpg", "computer rpg", "computer role-playing video game", "rp video game"], "dis": [[3516, 63, 1422, 57131, 3299], [1618, 1442, 1351, 45736, 15383], [3516, 3470, 63, 36185, 3299]]}, {"prop": "genre", "subj": "The Invasion", "obj": "science fiction film", "pop": 51885, "ids": [1507, 1646, 8493, 1373, 21933, 1385, 1358, 25979, 1379, 7517, 20307, 4101, 72], "a": 9, "n": 3, "alts": ["film adaptation", "based upon", "adapted for the screen", "cinematic adaptation", "thriller", "suspense", "thriller television program", "thriller tv program", "thriller television show", "thriller tv show", "suspense television program", "suspense tv program", "suspense television show", "suspense tv show", "horror film", "horror movie", "science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[18006, 40469, 6522], [4294, 5418, 13658], [26730, 1464, 1464]]}, {"prop": "genre", "subj": "The Others", "obj": "psychological horror", "pop": 51088, "ids": [1507, 36217, 21933, 1385, 1358, 25979, 1379, 18546, 27871, 72], "a": 7, "n": 2, "alts": ["horror film", "horror movie", "psychological horror"], "dis": [[15652, 20307], [7463, 15899], [18167, 4101]]}, {"prop": "genre", "subj": "Al Green", "obj": "soul music", "pop": 50264, "ids": [2270, 7868, 21933, 1385, 1358, 25979, 1379, 19538, 4264, 72], "a": 7, "n": 2, "alts": ["soul music", "soul", "soul"], "dis": [[7663, 4101], [27862, 4101], [18167, 4101]]}, {"prop": "genre", "subj": "The Intern", "obj": "comedy film", "pop": 48727, "ids": [1507, 4387, 21933, 1385, 1358, 25979, 1379, 15899, 4101, 72], "a": 7, "n": 2, "alts": ["comedy film", "comedy movie"], "dis": [[53821, 43178], [2744, 7653], [8295, 4378]]}, {"prop": "genre", "subj": "Species", "obj": "science fiction film", "pop": 48238, "ids": [46833, 21933, 1385, 1358, 25979, 1379, 7517, 20307, 4101, 72], "a": 6, "n": 3, "alts": ["science fiction film", "sci-fi film", "science fiction movie", "sci-fi movie", "scifi film", "scifi movie", "sci fi film", "sci fi movie", "scifi-film", "scifi"], "dis": [[7409, 1411, 3945], [45680, 17506, 7083], [13431, 63, 8082]]}, {"prop": "genre", "subj": "Dallas", "obj": "soap opera", "pop": 48104, "ids": [22941, 21933, 1385, 1358, 25979, 1379, 29766, 29686, 72], "a": 6, "n": 2, "alts": ["soap opera", "soap", "soapie", "soapopera"], "dis": [[40238, 13658], [1469, 8383], [26874, 4101]]}, {"prop": "genre", "subj": "Ministry", "obj": "industrial metal", "pop": 47939, "ids": [16714, 21933, 1385, 1358, 25979, 1379, 11312, 8252, 72], "a": 6, "n": 2, "alts": ["industrial metal", "industrial metal"], "dis": [[7653, 4264], [8445, 7653], [2415, 59326]]}, {"prop": "genre", "subj": "Wish You Were Here", "obj": "progressive rock", "pop": 47816, "ids": [52350, 2292, 53940, 5792, 21933, 1385, 1358, 25979, 1379, 19065, 7653, 72], "a": 9, "n": 2, "alts": ["progressive rock", "prog-rock", "prog", "classical rock", "symphonic rock", "art rock", "progressive pop", "prog rock", "progressive rock"], "dis": [[11312, 4264], [18167, 3299], [3538, 7653]]}, {"prop": "genre", "subj": "Lady A", "obj": "country music", "pop": 47339, "ids": [20231, 1420, 21933, 1385, 1358, 25979, 1379, 4451, 4264, 72], "a": 7, "n": 2, "alts": ["country music", "country and western", "country & western", "country", "nashville sound"], "dis": [[38299, 4101], [2446, 7653], [4244, 8082]]}, {"prop": "mother", "subj": "Elizabeth II", "obj": "Queen Elizabeth The Queen Mother", "pop": 1291432, "ids": [1507, 8184, 1379, 19594, 4401, 1523, 16039, 19594, 1507, 16039, 24497, 72], "a": 6, "n": 5, "alts": ["queen elizabeth the queen mother", "elizabeth bowes-lyon", "the queen mother", "elizabeth angela marguerite bowes-lyon", "queen consort elizabeth", "lady elizabeth bowes-lyon", "the queen mum"], "dis": [[42876, 1351, 7213, 1360, 1411], [5279, 9321, 8176, 54762, 12657], [1419, 2447, 1771, 1687, 1440]]}, {"prop": "mother", "subj": "Michael Jackson", "obj": "Katherine Jackson", "pop": 753723, "ids": [1507, 8184, 1379, 9114, 15670, 1523, 1681, 26225, 15670, 72], "a": 6, "n": 3, "alts": ["katherine jackson", "kattie b. screws", "katherine scruse", "katherine esther scruse", "katherine esther jackson"], "dis": [[44420, 59331, 2420], [13344, 4820, 1382], [34187, 16506, 44244]]}, {"prop": "mother", "subj": "Barack Obama", "obj": "Ann Dunham", "pop": 592684, "ids": [1507, 8184, 1379, 45072, 15469, 1523, 9282, 20460, 5399, 72], "a": 6, "n": 3, "alts": ["ann dunham", "stanley ann dunham", "ann obama", "ann soetoro", "ann sutoro", "s. ann dunham", "dr. stanley ann obama"], "dis": [[54279, 23405, 4140], [8959, 15818, 8225], [13344, 4820, 1382]]}, {"prop": "mother", "subj": "George VI", "obj": "Mary of Teck", "pop": 420204, "ids": [1507, 8184, 1379, 9595, 31650, 1523, 8959, 1379, 1408, 9725, 72], "a": 6, "n": 4, "alts": ["mary of teck", "victoria mary augusta louise olga pauline claudine agnes", "queen mary of the united kingdom", "mary, queen of the united kingdom", "mary, queen consort of the united kingdom", "princess victoria mary of teck", "princess victoria mary augusta louise olga pauline claudine agnes of teck", "victoria mary of teck", "victoria mary augusta louise olga pauline claudine agnes of teck", "princess may of teck", "may of teck", "mary, queen of the united kingdom, empress of india"], "dis": [[2572, 1394, 59336, 2420], [37160, 1491, 3411, 23423], [5279, 26756, 1452, 5323]]}, {"prop": "mother", "subj": "Alexander the Great", "obj": "Olympias", "pop": 394950, "ids": [1507, 8184, 1379, 19488, 1358, 7566, 1523, 6908, 6158, 72], "a": 7, "n": 2, "alts": ["olympias", "olympiada anayka"], "dis": [[2899, 1360], [10022, 8959], [21872, 29865]]}, {"prop": "mother", "subj": "Abraham Lincoln", "obj": "Nancy Hanks Lincoln", "pop": 394926, "ids": [1507, 8184, 1379, 46753, 23423, 1523, 37160, 1491, 3411, 23423, 72], "a": 6, "n": 4, "alts": ["nancy hanks lincoln", "nancy lincoln", "nancy hanks"], "dis": [[30540, 1681, 3243, 49727], [5279, 26756, 1452, 5323], [18257, 13109, 1445, 46776]]}, {"prop": "mother", "subj": "Napoleon", "obj": "Letizia Bonaparte", "pop": 384540, "ids": [1507, 8184, 1379, 24223, 2325, 1350, 1523, 2965, 1557, 1627, 16007, 1461, 48343, 72], "a": 7, "n": 6, "alts": ["letizia bonaparte", "maria letizia bonaparte", "maria letizia buonaparte", "maria letizia ramolino", "madame m\u00e8re", "laetitia bonaparte", "letizia ramallo", "letizia ramolino", "maria-letizia buonaparte"], "dis": [[10576, 59339, 5419, 1690, 43058, 4270], [1490, 2874, 6741, 2899, 13953, 12375], [4589, 1416, 1630, 58382, 29617, 10170]]}, {"prop": "mother", "subj": "Jesus", "obj": "Virgin Mary", "pop": 353596, "ids": [1507, 8184, 1379, 13855, 1523, 10022, 8959, 72], "a": 5, "n": 2, "alts": ["virgin mary", "maryam", "blessed virgin mary", "our lady", "blessed virgin", "madonna", "blessed mother", "theotokos", "the blessed mother", "the blessed virgin", "the blessed virgin mary", "mother of god", "immaculate mary", "saint mary", "holy mary", "holy virgin", "notre dame", "most blessed", "god-bearer", "virgin god-bearer", "mary, mother of jesus", "mother of jesus", "st mary the virgin", "queen of the universe", "mary", "maria", "the virgin mary", "mary, mother of our lord"], "dis": [[14227, 24423], [1716, 34498], [41220, 1660]]}, {"prop": "mother", "subj": "George Washington", "obj": "Mary Ball Washington", "pop": 350454, "ids": [1507, 8184, 1379, 9595, 8225, 1523, 8959, 15818, 8225, 72], "a": 6, "n": 3, "alts": ["mary ball washington", "mary ball", "mary washington"], "dis": [[8959, 1377, 15213], [44420, 59331, 2420], [17821, 1738, 53340]]}, {"prop": "mother", "subj": "George W. Bush", "obj": "Barbara Bush", "pop": 311254, "ids": [1507, 8184, 1379, 9595, 1495, 72, 24666, 1523, 30791, 24666, 72], "a": 8, "n": 2, "alts": ["barbara bush", "barbara pierce bush", "barbara pierce"], "dis": [[41220, 1660], [21872, 29865], [1507, 1627]]}, {"prop": "mother", "subj": "Amitabh Bachchan", "obj": "Teji Bachchan", "pop": 306074, "ids": [1507, 8184, 1379, 1420, 3368, 52682, 41832, 3165, 1523, 3288, 11140, 41832, 3165, 72], "a": 9, "n": 4, "alts": ["teji bachchan", "teji harivansh rai srivastava bachchan"], "dis": [[8959, 1379, 1408, 9725], [14227, 32740, 1379, 15280], [6537, 1459, 12911, 1657]]}, {"prop": "mother", "subj": "Edgar Allan Poe", "obj": "Eliza Poe", "pop": 287843, "ids": [1507, 8184, 1379, 5929, 14525, 52471, 1452, 5323, 1523, 5279, 26756, 1452, 5323, 72], "a": 9, "n": 4, "alts": ["eliza poe"], "dis": [[21872, 30385, 23902, 38061], [20852, 11383, 1379, 25650], [28875, 1494, 20242, 59325]]}, {"prop": "mother", "subj": "Cher", "obj": "Georgia Holt", "pop": 281641, "ids": [1507, 8184, 1379, 57165, 1523, 14185, 6875, 59323, 72], "a": 5, "n": 3, "alts": ["georgia holt", "jackie jean crouch", "georgia pelham", "georgia lapiere"], "dis": [[20609, 59327, 4126], [1491, 1501, 28161], [2278, 1783, 6511]]}, {"prop": "mother", "subj": "Sean Penn", "obj": "Eileen Ryan", "pop": 271420, "ids": [1507, 8184, 1379, 32392, 34970, 1523, 1529, 1603, 1356, 19685, 72], "a": 6, "n": 4, "alts": ["eileen ryan"], "dis": [[36027, 2572, 1388, 1770], [1445, 7839, 22169, 2768], [32740, 1408, 1528, 1355]]}, {"prop": "mother", "subj": "Gautama Buddha", "obj": "Maya", "pop": 270891, "ids": [1507, 8184, 1379, 1527, 2871, 4251, 53188, 5031, 1523, 3491, 59324, 72], "a": 9, "n": 2, "alts": ["maya"], "dis": [[6908, 6158], [15437, 28608], [58798, 16032]]}, {"prop": "mother", "subj": "Thomas Jefferson", "obj": "Jane Randolph Jefferson", "pop": 254256, "ids": [1507, 8184, 1379, 10804, 38061, 1523, 21872, 30385, 23902, 38061, 72], "a": 6, "n": 4, "alts": ["jane randolph jefferson"], "dis": [[8959, 1379, 1408, 9725], [24423, 17537, 3152, 7884], [3288, 11140, 41832, 3165]]}, {"prop": "mother", "subj": "Rekha", "obj": "Pushpavalli", "pop": 244093, "ids": [1507, 8184, 1379, 1490, 2469, 5031, 1523, 37069, 59336, 1591, 48349, 72], "a": 7, "n": 4, "alts": ["pushpavalli", "kandala venkata pushpavalli tayaramma"], "dis": [[3288, 11140, 41832, 3165], [2899, 1422, 1512, 44024], [22035, 1861, 21193, 2825]]}, {"prop": "mother", "subj": "King Arthur", "obj": "Igraine", "pop": 176470, "ids": [1507, 8184, 1379, 6132, 25340, 1523, 1417, 1755, 7290, 72], "a": 6, "n": 3, "alts": ["igraine", "eigyr"], "dis": [[1646, 24208, 29427], [1491, 1501, 28161], [13344, 4820, 1382]]}, {"prop": "mother", "subj": "Krishna", "obj": "Devaki", "pop": 164554, "ids": [1507, 8184, 1379, 46003, 3003, 1523, 9553, 14770, 72], "a": 6, "n": 2, "alts": ["devaki"], "dis": [[33686, 34333], [15437, 28608], [17657, 30155]]}, {"prop": "mother", "subj": "Apollo", "obj": "Leto", "pop": 130900, "ids": [1507, 8184, 1379, 47588, 1523, 2965, 59326, 72], "a": 5, "n": 2, "alts": ["leto"], "dis": [[1507, 44338], [41220, 1660], [53193, 2921]]}, {"prop": "mother", "subj": "Aung San Suu Kyi", "obj": "Khin Kyi", "pop": 127882, "ids": [1507, 8184, 1379, 1420, 3300, 6312, 24051, 59334, 23069, 59325, 1523, 14196, 1347, 23069, 59325, 72], "a": 11, "n": 4, "alts": ["khin kyi"], "dis": [[1494, 13660, 3509, 2338], [36027, 2572, 1388, 1770], [3288, 11140, 41832, 3165]]}, {"prop": "mother", "subj": "Ivan IV Vasilyevich", "obj": "Elena Glinskaya", "pop": 98033, "ids": [1507, 8184, 1379, 36545, 13461, 35296, 2168, 2036, 1620, 1523, 5279, 9321, 8176, 54762, 12657, 72], "a": 10, "n": 5, "alts": ["elena glinskaya", "yelena vasilyevna glinskaya"], "dis": [[2278, 1528, 1778, 1716, 29326], [1681, 1815, 1356, 1445, 5303], [2584, 18523, 3846, 39742, 1522]]}, {"prop": "mother", "subj": "Charles V, Holy Roman Emperor", "obj": "Joanna of Castile", "pop": 96101, "ids": [1507, 8184, 1379, 12583, 1690, 59342, 22240, 9486, 40960, 1523, 14195, 13808, 1379, 12150, 1603, 72], "a": 10, "n": 5, "alts": ["joanna of castile", "joanna", "joanna the mad", "juana of castile", "juana la loca", "juana i of castile and aragon", "juana i of aragon", "joanna i of castile and aragon", "joanna i of aragon"], "dis": [[2278, 1528, 1778, 1716, 29326], [2899, 1456, 8077, 1379, 7610], [1419, 2447, 1771, 1687, 1440]]}, {"prop": "mother", "subj": "Rama", "obj": "Kausalya", "pop": 93777, "ids": [1507, 8184, 1379, 1490, 4251, 1523, 1681, 37289, 6745, 72], "a": 6, "n": 3, "alts": ["kausalya"], "dis": [[19031, 8124, 6492], [1609, 21171, 63061], [54279, 1874, 28885]]}, {"prop": "mother", "subj": "Edward VI of England", "obj": "Jane Seymour", "pop": 90629, "ids": [1507, 8184, 1379, 18223, 31650, 1379, 7610, 1523, 21872, 2989, 2049, 1563, 72], "a": 8, "n": 4, "alts": ["jane seymour"], "dis": [[39933, 4401, 1379, 11114], [24423, 17537, 3152, 7884], [18257, 13109, 1445, 46776]]}, {"prop": "mother", "subj": "Thor", "obj": "J\u00f6r\u00f0", "pop": 87051, "ids": [1507, 8184, 1379, 35689, 1523, 1609, 21171, 63061, 72], "a": 5, "n": 3, "alts": ["j\u00f6r\u00f0", "erda", "jord"], "dis": [[25144, 7856, 59324], [52430, 1469, 4376], [8959, 1377, 15213]]}, {"prop": "mother", "subj": "Helen of Troy", "obj": "Leda", "pop": 86909, "ids": [1507, 8184, 1379, 40297, 1379, 50634, 1523, 1494, 26546, 72], "a": 7, "n": 2, "alts": ["leda"], "dis": [[45342, 61373], [1716, 34498], [15511, 56253]]}, {"prop": "mother", "subj": "Ganesha", "obj": "Parvati", "pop": 85302, "ids": [1507, 8184, 1379, 1527, 11431, 5031, 1523, 4002, 59343, 11561, 72], "a": 7, "n": 3, "alts": ["parvati", "uma", "parbati", "gauri"], "dis": [[44420, 59331, 2420], [54279, 23405, 4140], [1681, 37289, 6745]]}, {"prop": "mother", "subj": "Jesus in Islam", "obj": "Mary in Islam", "pop": 78584, "ids": [1507, 8184, 1379, 13855, 1377, 15213, 1523, 8959, 1377, 15213, 72], "a": 7, "n": 3, "alts": ["mary in islam", "christian"], "dis": [[56182, 1788, 8680], [1609, 41157, 5564], [4002, 59343, 11561]]}, {"prop": "mother", "subj": "Ronan Farrow", "obj": "Mia Farrow", "pop": 76944, "ids": [1507, 8184, 1379, 23604, 1363, 1493, 4033, 1523, 1445, 1627, 1493, 4033, 72], "a": 8, "n": 4, "alts": ["mia farrow", "maria de lourdes villiers-farrow"], "dis": [[22035, 1861, 21193, 2825], [5279, 26756, 1452, 5323], [28875, 1494, 20242, 59325]]}, {"prop": "mother", "subj": "Mary, Princess Royal and Countess of Harewood", "obj": "Mary of Teck", "pop": 76000, "ids": [1507, 8184, 1379, 8959, 59342, 31849, 10297, 1384, 3449, 1481, 1379, 1491, 1579, 6593, 1523, 8959, 1379, 1408, 9725, 72], "a": 15, "n": 4, "alts": ["mary of teck", "victoria mary augusta louise olga pauline claudine agnes", "queen mary of the united kingdom", "mary, queen of the united kingdom", "mary, queen consort of the united kingdom", "princess victoria mary of teck", "princess victoria mary augusta louise olga pauline claudine agnes of teck", "victoria mary of teck", "victoria mary augusta louise olga pauline claudine agnes of teck", "princess may of teck", "may of teck", "mary, queen of the united kingdom, empress of india"], "dis": [[21872, 2989, 2049, 1563], [14196, 1460, 19283, 39001], [28875, 1494, 20242, 59325]]}, {"prop": "mother", "subj": "Vajiralongkorn", "obj": "Sirikit", "pop": 74973, "ids": [1507, 8184, 1379, 1690, 2549, 18912, 1635, 59346, 2355, 1523, 15511, 56253, 72], "a": 10, "n": 2, "alts": ["sirikit", "sirikit kitiyakara"], "dis": [[10022, 8959], [14227, 24423], [21872, 29865]]}, {"prop": "mother", "subj": "Horus", "obj": "Isis", "pop": 71001, "ids": [1507, 8184, 1379, 13993, 1406, 1523, 2899, 1360, 72], "a": 6, "n": 2, "alts": ["isis"], "dis": [[23245, 1359], [9553, 14770], [15511, 56253]]}, {"prop": "mother", "subj": "Paul I of Russia", "obj": "Catherine II of Russia", "pop": 57543, "ids": [1507, 8184, 1379, 7821, 1417, 1379, 11114, 1523, 39933, 4401, 1379, 11114, 72], "a": 8, "n": 4, "alts": ["catherine ii of russia", "ekaterina alexeyevna", "sophie friederike auguste von anhalt-zerbst", "empress catherine ii", "\u0435\u043a\u0430\u0442\u0435\u0440\u0438\u043d\u0430 ii \u0432\u0435\u043b\u0438\u043a\u0430\u044f", "czarina of russia catherine ii", "empress of russia catherine ii", "catherine ii", "ekaterina ii of russia", "catherine the great"], "dis": [[30540, 1681, 3243, 49727], [2572, 1394, 59336, 2420], [3308, 26685, 16557, 2888]]}, {"prop": "mother", "subj": "Samson", "obj": "Wife of Manoah", "pop": 56598, "ids": [1507, 8184, 1379, 9137, 2162, 1523, 1495, 2304, 1379, 5174, 59326, 2420, 72], "a": 6, "n": 6, "alts": ["wife of manoah"], "dis": [[1527, 7839, 1454, 32717, 1379, 54634], [1419, 60691, 59333, 32224, 1874, 1640], [16839, 59324, 1379, 1491, 1485, 2313]]}, {"prop": "mother", "subj": "Stephen of England", "obj": "Adela of Normandy", "pop": 52061, "ids": [1507, 8184, 1379, 19378, 1379, 7610, 1523, 3227, 12941, 1379, 20201, 14097, 72], "a": 7, "n": 5, "alts": ["adela of normandy", "adela of blois", "adela of england"], "dis": [[20852, 10564, 32802, 4499, 12657], [20852, 17141, 23015, 1379, 25650], [42876, 1351, 7213, 1360, 1411]]}, {"prop": "mother", "subj": "Mahavira", "obj": "Trishala", "pop": 50616, "ids": [1507, 8184, 1379, 14257, 1591, 11010, 1523, 2278, 1783, 6511, 72], "a": 7, "n": 3, "alts": ["trishala"], "dis": [[58653, 59329, 1394], [56182, 1788, 8680], [19031, 8124, 6492]]}, {"prop": "mother", "subj": "Mikheil Saakashvili", "obj": "Giuli Alasania", "pop": 43350, "ids": [1507, 8184, 1379, 38509, 1351, 1394, 16250, 1622, 2116, 59343, 8143, 1523, 16260, 24492, 2270, 1382, 19370, 72], "a": 12, "n": 5, "alts": ["giuli alasania", "giuli givievna alasania"], "dis": [[1529, 1692, 2447, 1445, 1582], [42876, 1351, 7213, 1360, 1411], [46390, 1627, 1527, 56603, 9629]]}, {"prop": "mother", "subj": "Terry Fox", "obj": "Betty Fox", "pop": 42263, "ids": [1507, 8184, 1379, 35219, 16032, 1523, 58798, 16032, 72], "a": 6, "n": 2, "alts": ["betty fox", "betty lou fox", "betty lou wark", "betty wark"], "dis": [[36391, 53340], [45342, 61373], [2965, 59326]]}, {"prop": "mother", "subj": "David Rockefeller", "obj": "Abigail Greene Aldrich", "pop": 42006, "ids": [1507, 8184, 1379, 7524, 25024, 1509, 59337, 12074, 1523, 4589, 1416, 1630, 58382, 29617, 10170, 72], "a": 9, "n": 6, "alts": ["abigail greene aldrich", "abby aldrich", "abigail aldrich", "abby greene aldrich rockefeller", "abigail greene aldrich", "abby aldrich rockefeller", "mrs. j.d. rockefeller jr.", "abigail greene aldrich rockefeller"], "dis": [[20852, 1452, 1627, 1379, 18869, 2004], [30791, 1529, 59331, 2261, 1352, 1620], [10576, 59339, 5419, 1690, 43058, 4270]]}, {"prop": "mother", "subj": "Nike", "obj": "Styx", "pop": 41563, "ids": [1507, 8184, 1379, 45625, 1523, 1716, 34498, 72], "a": 5, "n": 2, "alts": ["styx"], "dis": [[1515, 34498], [6908, 6158], [53193, 2921]]}, {"prop": "mother", "subj": "Lucy DeVito", "obj": "Rhea Perlman", "pop": 38381, "ids": [1507, 8184, 1379, 51354, 2748, 59408, 11655, 1523, 1490, 28170, 56112, 2450, 72], "a": 8, "n": 4, "alts": ["rhea perlman", "rhea jo perlman"], "dis": [[6485, 59324, 22658, 10955], [28875, 1494, 20242, 59325], [2572, 1394, 59336, 2420]]}, {"prop": "mother", "subj": "Paris", "obj": "Hecuba", "pop": 37426, "ids": [1507, 8184, 1379, 11225, 1523, 1491, 1501, 28161, 72], "a": 5, "n": 3, "alts": ["hecuba"], "dis": [[52430, 1469, 4376], [1515, 1371, 1364], [1417, 1755, 7290]]}, {"prop": "mother", "subj": "Eliot Sumner", "obj": "Trudie Styler", "pop": 35120, "ids": [1507, 8184, 1379, 34974, 1411, 27646, 2120, 1523, 2278, 1528, 1778, 1716, 29326, 72], "a": 8, "n": 5, "alts": ["trudie styler"], "dis": [[1529, 1692, 2447, 1445, 1582], [5279, 9321, 8176, 54762, 12657], [44218, 12941, 2063, 59340, 2642]]}, {"prop": "mother", "subj": "Mary Cheney", "obj": "Lynne Cheney", "pop": 27500, "ids": [1507, 8184, 1379, 8959, 21193, 2825, 1523, 22035, 1861, 21193, 2825, 72], "a": 7, "n": 4, "alts": ["lynne cheney", "lynne ann cheney", "lynne ann vincent cheney"], "dis": [[38911, 1967, 1495, 7212], [21872, 2989, 2049, 1563], [2572, 1394, 59336, 2420]]}, {"prop": "mother", "subj": "Aaron", "obj": "Jochebed", "pop": 26977, "ids": [1507, 8184, 1379, 35037, 1523, 1609, 41157, 5564, 72], "a": 5, "n": 3, "alts": ["jochebed", "yocheved"], "dis": [[24423, 11318, 2490], [14185, 6875, 59323], [8959, 15818, 8225]]}, {"prop": "mother", "subj": "Menelaus", "obj": "Aerope", "pop": 26335, "ids": [1507, 8184, 1379, 12180, 12941, 1406, 1523, 53193, 2921, 72], "a": 7, "n": 2, "alts": ["aerope"], "dis": [[30791, 24666], [45342, 61373], [5433, 15465]]}, {"prop": "mother", "subj": "Cassandra Cain", "obj": "Lady Shiva", "pop": 24955, "ids": [1507, 8184, 1379, 27475, 21400, 1419, 1485, 1523, 20231, 2419, 15653, 72], "a": 8, "n": 3, "alts": ["lady shiva", "sandra woosan", "sandra wu-san"], "dis": [[24423, 11318, 2490], [14185, 6875, 59323], [52430, 1469, 4376]]}, {"prop": "mother", "subj": "Carl Grimes", "obj": "Lori Grimes", "pop": 23926, "ids": [1507, 8184, 1379, 14881, 3509, 2338, 1523, 1494, 13660, 3509, 2338, 72], "a": 7, "n": 4, "alts": ["lori grimes"], "dis": [[14196, 1347, 23069, 59325], [6485, 59324, 22658, 10955], [1529, 1603, 1356, 19685]]}, {"prop": "mother", "subj": "Eos", "obj": "Theia", "pop": 20437, "ids": [1507, 8184, 1379, 1529, 1472, 1523, 1507, 1627, 72], "a": 6, "n": 2, "alts": ["theia", "basileia"], "dis": [[23245, 1359], [2965, 59326], [1494, 26546]]}, {"prop": "mother", "subj": "Crispian Mills", "obj": "Hayley Mills", "pop": 19871, "ids": [1507, 8184, 1379, 5758, 22311, 1770, 37194, 1523, 19093, 3389, 37194, 72], "a": 8, "n": 3, "alts": ["hayley mills", "hayley catherine rose vivien mills"], "dis": [[54279, 1874, 28885], [20609, 59327, 4126], [54279, 23405, 4140]]}, {"prop": "mother", "subj": "Parikshit", "obj": "Uttar\u0101", "pop": 18554, "ids": [1507, 8184, 1379, 1452, 1920, 1742, 31097, 1523, 45342, 61373, 72], "a": 8, "n": 2, "alts": ["uttar\u0101"], "dis": [[2899, 1360], [21872, 29865], [15437, 28608]]}, {"prop": "mother", "subj": "Cable", "obj": "Madelyne Pryor", "pop": 18178, "ids": [1507, 8184, 1379, 45895, 1523, 9405, 1769, 1861, 1452, 1687, 1355, 72], "a": 5, "n": 6, "alts": ["madelyne pryor", "madelyne jennifer pryor-summers", "maddie pryor", "goblyn queen", "red queen"], "dis": [[20852, 1452, 1627, 1379, 18869, 2004], [31174, 1373, 1793, 3776, 59341, 9124], [1529, 1497, 2112, 1627, 1564, 20119]]}, {"prop": "mother", "subj": "Mary in Islam", "obj": "Saint Anne", "pop": 17357, "ids": [1507, 8184, 1379, 8959, 1377, 15213, 1523, 14227, 24423, 72], "a": 7, "n": 2, "alts": ["saint anne", "saint anne", "st. anne", "st. anna", "st anne", "st anna", "anne", "anna", "saint anna"], "dis": [[17657, 30155], [53193, 2921], [45342, 61373]]}, {"prop": "mother", "subj": "August Coppola", "obj": "Italia Coppola", "pop": 16762, "ids": [1507, 8184, 1379, 5445, 18030, 8021, 59324, 1523, 6504, 1627, 18030, 8021, 59324, 72], "a": 8, "n": 5, "alts": ["italia coppola", "italia pennino", "italia pennino coppola"], "dis": [[5279, 9321, 8176, 54762, 12657], [2899, 2625, 1378, 9824, 1563], [1419, 2447, 1771, 1687, 1440]]}, {"prop": "mother", "subj": "Proserpina", "obj": "Ceres", "pop": 15124, "ids": [1507, 8184, 1379, 2078, 2066, 59336, 2888, 1523, 23245, 1359, 72], "a": 8, "n": 2, "alts": ["ceres"], "dis": [[33686, 34333], [1716, 34498], [14227, 24423]]}, {"prop": "mother", "subj": "Susanne Klatten", "obj": "Johanna Quandt", "pop": 14278, "ids": [1507, 8184, 1379, 15526, 24860, 1681, 12279, 2405, 1523, 3969, 13808, 4286, 1483, 59323, 72], "a": 9, "n": 5, "alts": ["johanna quandt", "johanna maria quandt"], "dis": [[2278, 1528, 1778, 1716, 29326], [8486, 1653, 59324, 1379, 26955], [16260, 24492, 2270, 1382, 19370]]}, {"prop": "mother", "subj": "Cadmus", "obj": "Telephassa", "pop": 14052, "ids": [1507, 8184, 1379, 33511, 22450, 1523, 17821, 1738, 53340, 72], "a": 6, "n": 3, "alts": ["telephassa"], "dis": [[1491, 1501, 28161], [58653, 59329, 1394], [8959, 15818, 8225]]}, {"prop": "mother", "subj": "James I of Scotland", "obj": "Anabella Drummond", "pop": 13846, "ids": [1507, 8184, 1379, 8266, 1417, 1379, 15280, 1523, 2426, 1456, 8077, 4450, 3092, 1810, 72], "a": 8, "n": 6, "alts": ["anabella drummond"], "dis": [[1479, 1423, 1437, 1379, 2548, 7443], [30791, 1529, 59331, 2261, 1352, 1620], [16839, 59324, 1379, 1491, 1485, 2313]]}, {"prop": "mother", "subj": "Emma of Normandy", "obj": "Gunnora", "pop": 12848, "ids": [1507, 8184, 1379, 40813, 1379, 20201, 14097, 1523, 25144, 7856, 59324, 72], "a": 8, "n": 3, "alts": ["gunnora", "gunnor"], "dis": [[8959, 15818, 8225], [4002, 59343, 11561], [1681, 26225, 15670]]}, {"prop": "mother", "subj": "Franklin Richards", "obj": "Invisible Woman", "pop": 11803, "ids": [1507, 8184, 1379, 25459, 58462, 1523, 1646, 24208, 29427, 72], "a": 6, "n": 3, "alts": ["invisible woman", "susan storm", "susan storm-richards", "sue stor"], "dis": [[19031, 8124, 6492], [54279, 23405, 4140], [14185, 6875, 59323]]}, {"prop": "mother", "subj": "Utu", "obj": "Ningal", "pop": 11514, "ids": [1507, 8184, 1379, 16081, 59334, 1523, 1515, 1371, 1364, 72], "a": 6, "n": 3, "alts": ["ningal"], "dis": [[1491, 1501, 28161], [17821, 1738, 53340], [20231, 2419, 15653]]}, {"prop": "mother", "subj": "Astraea", "obj": "Themis", "pop": 11237, "ids": [1507, 8184, 1379, 33148, 45957, 1523, 1507, 44338, 72], "a": 6, "n": 2, "alts": ["themis"], "dis": [[1716, 34498], [6908, 6158], [23245, 1359]]}, {"prop": "mother", "subj": "Margaret Douglas", "obj": "Margaret Tudor", "pop": 11095, "ids": [1507, 8184, 1379, 32740, 27816, 1523, 32740, 1408, 1528, 1355, 72], "a": 6, "n": 4, "alts": ["margaret tudor"], "dis": [[39933, 4401, 1379, 11114], [28875, 1494, 20242, 59325], [38911, 1967, 1495, 7212]]}, {"prop": "mother", "subj": "Sin", "obj": "Ninlil", "pop": 10888, "ids": [1507, 8184, 1379, 20362, 1523, 58653, 59329, 1394, 72], "a": 5, "n": 3, "alts": ["ninlil"], "dis": [[1681, 37289, 6745], [1609, 41157, 5564], [1681, 26225, 15670]]}, {"prop": "mother", "subj": "Horae", "obj": "Themis", "pop": 10127, "ids": [1507, 8184, 1379, 13993, 3770, 1523, 1507, 44338, 72], "a": 6, "n": 2, "alts": ["themis"], "dis": [[30791, 24666], [14227, 24423], [2965, 59326]]}, {"prop": "mother", "subj": "Koxinga", "obj": "Tagawa Matsu", "pop": 9747, "ids": [1507, 8184, 1379, 1681, 2092, 1371, 59324, 1523, 18257, 13109, 1445, 46776, 72], "a": 8, "n": 4, "alts": ["tagawa matsu"], "dis": [[21872, 2989, 2049, 1563], [3308, 26685, 16557, 2888], [37160, 1491, 3411, 23423]]}, {"prop": "mother", "subj": "Mao Anqing", "obj": "Yang Kaihui", "pop": 9622, "ids": [1507, 8184, 1379, 1445, 7022, 2426, 52321, 1523, 30540, 1681, 3243, 49727, 72], "a": 8, "n": 4, "alts": ["yang kaihui"], "dis": [[1494, 13660, 3509, 2338], [38911, 1967, 1495, 7212], [28875, 1494, 20242, 59325]]}, {"prop": "mother", "subj": "Rachel Summers", "obj": "Jean Grey", "pop": 9556, "ids": [1507, 8184, 1379, 39327, 24657, 1465, 1523, 17657, 30155, 72], "a": 7, "n": 2, "alts": ["jean grey", "marvel girl", "jean elaine grises summers", "jean elaine gray", "fenix", "jean grey-summers", "dark fenix"], "dis": [[33686, 34333], [15437, 28608], [30791, 24666]]}, {"prop": "mother", "subj": "Sajeeb Wazed", "obj": "Sheikh Hasina", "pop": 9544, "ids": [1507, 8184, 1379, 1413, 2549, 2199, 59340, 1495, 29077, 1523, 3308, 26685, 16557, 2888, 72], "a": 10, "n": 4, "alts": ["sheikh hasina", "sheikh hasina wazed", "hasina wazed"], "dis": [[1529, 1603, 1356, 19685], [14227, 32740, 1379, 15280], [38911, 1967, 1495, 7212]]}, {"prop": "occupation", "subj": "Donald Trump", "obj": "real estate development", "pop": 15101521, "ids": [16234, 7562, 5901, 1467, 1348, 2902, 10604, 2956, 72], "a": 5, "n": 3, "alts": ["actor", "actress", "actors", "actresses", "socialite", "prominent person", "real estate development", "property development", "real estate development", "politician", "political leader", "political figure", "polit.", "pol", "business magnate", "businessperson", "tycoon", "magnate"], "dis": [[2998, 45053, 1466], [1577, 11976, 62258], [3035, 1470, 6716]]}, {"prop": "occupation", "subj": "Kanye West", "obj": "record producer", "pop": 940772, "ids": [1681, 2186, 59321, 5308, 5901, 1467, 1348, 3175, 14109, 72], "a": 7, "n": 2, "alts": ["record producer", "music producer", "songwriter", "song writer"], "dis": [[4101, 14109], [10462, 13448], [4034, 1671]]}, {"prop": "occupation", "subj": "Michael Jackson", "obj": "actor", "pop": 753723, "ids": [9114, 15670, 5901, 1467, 1348, 13448, 72], "a": 5, "n": 1, "alts": ["record producer", "music producer", "beatboxing", "human beat box", "film director", "movie director", "director", "motion picture director", "screenwriter", "scenarist", "writer", "screen writer", "script writer", "scriptwriter", "actor", "actress", "actors", "actresses", "composer", "businessperson", "business person", "dealer", "businesswoman", "business man", "business woman", "businessman", "singer-songwriter", "singer songwriter", "singer/songwriter", "singersongwriter", "singer-songwriter", "poet", "poetess", "bard", "musician", "musical artist", "songwriter", "song writer"], "dis": [[18856], [52310], [7415]]}, {"prop": "occupation", "subj": "Mike Flanagan", "obj": "screenwriter", "pop": 449587, "ids": [13738, 6363, 3624, 1363, 5901, 1467, 1348, 4925, 12652, 72], "a": 7, "n": 2, "alts": ["screenwriter", "scenarist", "writer", "screen writer", "script writer", "scriptwriter"], "dis": [[55777, 8084], [4048, 2073], [35859, 2230]]}, {"prop": "occupation", "subj": "Will Smith", "obj": "film producer", "pop": 345519, "ids": [9878, 10459, 5901, 1467, 1348, 4101, 14109, 72], "a": 5, "n": 2, "alts": ["record producer", "music producer", "film director", "movie director", "director", "motion picture director", "film producer", "movie producer", "producer", "actor", "actress", "actors", "actresses"], "dis": [[33594, 1466], [4958, 1466], [1616, 10833]]}, {"prop": "occupation", "subj": "Amitabh Bachchan", "obj": "film producer", "pop": 306074, "ids": [1420, 3368, 52682, 41832, 3165, 5901, 1467, 1348, 4101, 14109, 72], "a": 8, "n": 2, "alts": ["film producer", "movie producer", "producer", "actor", "actress", "actors", "actresses"], "dis": [[2262, 59346], [11990, 54217], [29872, 13924]]}, {"prop": "occupation", "subj": "Edgar Allan Poe", "obj": "playwright", "pop": 287843, "ids": [5929, 14525, 52471, 1452, 5323, 5901, 1467, 1348, 50014, 72], "a": 8, "n": 1, "alts": ["journalist", "journo", "journalists", "playwright", "dramatist", "playwrite", "scriptwriter", "poet", "poetess", "bard"], "dis": [[48829], [36281], [13448]]}, {"prop": "occupation", "subj": "Ruby Rose", "obj": "disc jockey", "pop": 245402, "ids": [28719, 17441, 5901, 1467, 1348, 2415, 1616, 10833, 72], "a": 5, "n": 3, "alts": ["disc jockey", "dj", "dee jay", "dj", "d.j.", "disk jockey", "dee-jay", "deejay", "disquaire", "discaire", "actor", "actress", "actors", "actresses", "television presenter", "host", "hostess", "tv host", "television host", "tv presenter", "tv anchor", "television anchor"], "dis": [[12767, 19173, 30292], [13577, 63, 50049], [4633, 6545, 1446]]}, {"prop": "occupation", "subj": "Tyler, the Creator", "obj": "record producer", "pop": 194634, "ids": [40795, 59342, 1358, 13712, 1355, 5901, 1467, 1348, 3175, 14109, 72], "a": 8, "n": 2, "alts": ["record producer", "music producer", "actor", "actress", "actors", "actresses", "graphic designer"], "dis": [[8447, 17436], [4034, 1671], [1616, 10833]]}, {"prop": "occupation", "subj": "Billy Joel", "obj": "singer-songwriter", "pop": 190041, "ids": [33596, 46319, 5901, 1467, 1348, 13577, 63, 50049, 72], "a": 5, "n": 3, "alts": ["composer", "pianist", "singer-songwriter", "singer songwriter", "singer/songwriter", "singersongwriter", "singer-songwriter"], "dis": [[2902, 10604, 2956], [29013, 2296, 1466], [12137, 12472, 12772]]}, {"prop": "occupation", "subj": "Dennis Rodman", "obj": "actor", "pop": 157667, "ids": [37826, 16162, 2450, 5901, 1467, 1348, 13448, 72], "a": 6, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[25473], [25319], [25299]]}, {"prop": "occupation", "subj": "Paul", "obj": "missionary", "pop": 135900, "ids": [7821, 5901, 1467, 1348, 9935, 1629, 72], "a": 4, "n": 2, "alts": ["missionary", "christian missionary"], "dis": [[1616, 10833], [18928, 18661], [8447, 17436]]}, {"prop": "occupation", "subj": "Famke Janssen", "obj": "actor", "pop": 126052, "ids": [1493, 1398, 1509, 1609, 1611, 15114, 5901, 1467, 1348, 13448, 72], "a": 9, "n": 1, "alts": ["screenwriter", "scenarist", "writer", "screen writer", "script writer", "scriptwriter", "actor", "actress", "actors", "actresses", "model", "fashion model", "sitter", "poser"], "dis": [[38778], [31525], [13129]]}, {"prop": "occupation", "subj": "Rumi", "obj": "poet", "pop": 92800, "ids": [1490, 29976, 5901, 1467, 1348, 25473, 72], "a": 5, "n": 1, "alts": ["philosopher", "poet", "poetess", "bard"], "dis": [[13129], [25319], [48829]]}, {"prop": "occupation", "subj": "Carl Linnaeus", "obj": "physician", "pop": 87289, "ids": [14881, 8351, 3003, 59321, 1406, 5901, 1467, 1348, 18856, 72], "a": 8, "n": 1, "alts": ["physician", "physicians", "medical doctor", "medical practitioner", "doctor", "medical doctors"], "dis": [[35684], [25299], [32373]]}, {"prop": "occupation", "subj": "Brigitte Bardot", "obj": "actor", "pop": 78912, "ids": [30505, 1362, 1739, 1469, 1578, 1411, 5901, 1467, 1348, 13448, 72], "a": 9, "n": 1, "alts": ["actor", "actress", "actors", "actresses", "model", "fashion model", "sitter", "poser"], "dis": [[33482], [18856], [25299]]}, {"prop": "occupation", "subj": "James Cook", "obj": "sailor", "pop": 77049, "ids": [8266, 19201, 5901, 1467, 1348, 26191, 1355, 72], "a": 5, "n": 2, "alts": ["sailor", "seaman", "mariner", "seafarer", "merchant sailor", "merchant seaman", "merchant mariner"], "dis": [[6035, 29355], [4841, 1466], [16469, 33936]]}, {"prop": "occupation", "subj": "Chris Jericho", "obj": "actor", "pop": 75216, "ids": [13765, 9196, 1620, 59326, 5901, 1467, 1348, 13448, 72], "a": 7, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[52310], [32373], [25473]]}, {"prop": "occupation", "subj": "Brian Eno", "obj": "composer", "pop": 74683, "ids": [19562, 1529, 3768, 5901, 1467, 1348, 21840, 72], "a": 6, "n": 1, "alts": ["record producer", "music producer", "composer"], "dis": [[33482], [50014], [16855]]}, {"prop": "occupation", "subj": "Benazir Bhutto", "obj": "politician", "pop": 73051, "ids": [7370, 2215, 1448, 24161, 1403, 2016, 5901, 1467, 1348, 13129, 72], "a": 9, "n": 1, "alts": ["politician", "political leader", "political figure", "polit.", "pol"], "dis": [[16855], [33482], [32373]]}, {"prop": "occupation", "subj": "Scooter Braun", "obj": "talent manager", "pop": 66280, "ids": [2732, 20425, 23986, 1444, 5901, 1467, 1348, 17400, 8869, 72], "a": 7, "n": 2, "alts": ["talent manager", "artist manager", "band manager", "music manager", "musician manager", "manager", "pop entrepreneur"], "dis": [[20972, 15951], [16469, 33936], [4958, 1466]]}, {"prop": "occupation", "subj": "Burna Boy", "obj": "songwriter", "pop": 62589, "ids": [20696, 59324, 16433, 5901, 1467, 1348, 48829, 72], "a": 6, "n": 1, "alts": ["songwriter", "song writer"], "dis": [[51484], [31525], [33482]]}, {"prop": "occupation", "subj": "Richard Rishi", "obj": "actor", "pop": 53974, "ids": [11646, 1490, 37207, 5901, 1467, 1348, 13448, 72], "a": 6, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[25319], [32373], [33482]]}, {"prop": "occupation", "subj": "Bruce Cockburn", "obj": "composer", "pop": 52017, "ids": [27229, 46252, 19079, 5901, 1467, 1348, 21840, 72], "a": 6, "n": 1, "alts": ["composer", "singer-songwriter", "singer songwriter", "singer/songwriter", "singersongwriter", "singer-songwriter", "songwriter", "song writer"], "dis": [[25473], [13448], [37764]]}, {"prop": "occupation", "subj": "Jason Reitman", "obj": "screenwriter", "pop": 50572, "ids": [24052, 2518, 1362, 2450, 5901, 1467, 1348, 4925, 12652, 72], "a": 7, "n": 2, "alts": ["film director", "movie director", "director", "motion picture director", "screenwriter", "scenarist", "writer", "screen writer", "script writer", "scriptwriter"], "dis": [[6862, 13924], [4034, 1671], [16410, 1466]]}, {"prop": "occupation", "subj": "Leigh Whannell", "obj": "screenwriter", "pop": 44715, "ids": [3013, 1794, 2814, 2234, 1568, 5901, 1467, 1348, 4925, 12652, 72], "a": 8, "n": 2, "alts": ["screenwriter", "scenarist", "writer", "screen writer", "script writer", "scriptwriter", "actor", "actress", "actors", "actresses"], "dis": [[4101, 8084], [9935, 1629], [5511, 46629]]}, {"prop": "occupation", "subj": "Salvador Allende", "obj": "politician", "pop": 43544, "ids": [46971, 3030, 34569, 5901, 1467, 1348, 13129, 72], "a": 6, "n": 1, "alts": ["physician", "physicians", "medical doctor", "medical practitioner", "doctor", "medical doctors", "politician", "political leader", "political figure", "polit.", "pol"], "dis": [[21840], [31525], [37764]]}, {"prop": "occupation", "subj": "Darrell Hammond", "obj": "actor", "pop": 34399, "ids": [15319, 24081, 36578, 1810, 5901, 1467, 1348, 13448, 72], "a": 7, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[25299], [16855], [52310]]}, {"prop": "occupation", "subj": "Robby Krieger", "obj": "guitarist", "pop": 31792, "ids": [9027, 2645, 1681, 1460, 4341, 5901, 1467, 1348, 38778, 72], "a": 8, "n": 1, "alts": ["guitarist", "guitar player", "guitarists"], "dis": [[32373], [35684], [37764]]}, {"prop": "occupation", "subj": "Seungri", "obj": "actor", "pop": 30935, "ids": [2989, 3300, 1460, 5901, 1467, 1348, 13448, 72], "a": 6, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[36281], [48829], [50014]]}, {"prop": "occupation", "subj": "Natasha Leggero", "obj": "actor", "pop": 25799, "ids": [30407, 33057, 8654, 2683, 59326, 5901, 1467, 1348, 13448, 72], "a": 8, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[25299], [52310], [33482]]}, {"prop": "occupation", "subj": "Beth Hart", "obj": "singer-songwriter", "pop": 25256, "ids": [34362, 20989, 5901, 1467, 1348, 13577, 63, 50049, 72], "a": 5, "n": 3, "alts": ["singer-songwriter", "singer songwriter", "singer/songwriter", "singersongwriter", "singer-songwriter"], "dis": [[12767, 19173, 30292], [2998, 45053, 1466], [12137, 12472, 12772]]}, {"prop": "occupation", "subj": "Jos\u00e9 Feliciano", "obj": "composer", "pop": 23417, "ids": [35478, 29481, 7183, 59326, 5901, 1467, 1348, 21840, 72], "a": 7, "n": 1, "alts": ["composer"], "dis": [[25299], [31525], [38778]]}, {"prop": "occupation", "subj": "Bala", "obj": "film director", "pop": 21931, "ids": [1469, 6511, 5901, 1467, 1348, 4101, 8084, 72], "a": 5, "n": 2, "alts": ["film director", "movie director", "director", "motion picture director", "screenwriter", "scenarist", "writer", "screen writer", "script writer", "scriptwriter"], "dis": [[1616, 10833], [3175, 14109], [17400, 8869]]}, {"prop": "occupation", "subj": "Glenn Gould", "obj": "composer", "pop": 21802, "ids": [49344, 1527, 1619, 5901, 1467, 1348, 21840, 72], "a": 6, "n": 1, "alts": ["composer", "pianist"], "dis": [[31525], [33482], [25299]]}, {"prop": "occupation", "subj": "Ayumi Hamasaki", "obj": "lyricist", "pop": 19177, "ids": [27700, 29976, 14796, 54395, 5901, 1467, 1348, 29013, 2296, 1466, 72], "a": 7, "n": 3, "alts": ["actor", "actress", "actors", "actresses", "singer-songwriter", "singer songwriter", "singer/songwriter", "singersongwriter", "singer-songwriter", "songwriter", "song writer", "lyricist", "singer-lyricist"], "dis": [[4633, 6545, 1446], [1380, 1657, 11875], [13577, 63, 50049]]}, {"prop": "occupation", "subj": "Tiffany Pollard", "obj": "actor", "pop": 18722, "ids": [1408, 4696, 2186, 32058, 1578, 5901, 1467, 1348, 13448, 72], "a": 8, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[7415], [35684], [13129]]}, {"prop": "occupation", "subj": "Kamya Panjabi", "obj": "actor", "pop": 17881, "ids": [23302, 6745, 10998, 59392, 22094, 5901, 1467, 1348, 13448, 72], "a": 8, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[25319], [23202], [31525]]}, {"prop": "occupation", "subj": "Charles Martin Smith", "obj": "actor", "pop": 16435, "ids": [12583, 12489, 10459, 5901, 1467, 1348, 13448, 72], "a": 6, "n": 1, "alts": ["film director", "movie director", "director", "motion picture director", "actor", "actress", "actors", "actresses"], "dis": [[21840], [16855], [32373]]}, {"prop": "occupation", "subj": "Teddy Riley", "obj": "record producer", "pop": 16203, "ids": [30433, 8235, 58658, 5901, 1467, 1348, 3175, 14109, 72], "a": 6, "n": 2, "alts": ["record producer", "music producer", "singer-songwriter", "singer songwriter", "singer/songwriter", "singersongwriter", "singer-songwriter"], "dis": [[2076, 14743], [5202, 13924], [2262, 59346]]}, {"prop": "occupation", "subj": "Vanessa Angel", "obj": "actor", "pop": 16012, "ids": [13389, 33867, 9224, 5901, 1467, 1348, 13448, 72], "a": 6, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[2158], [33482], [41865]]}, {"prop": "occupation", "subj": "Malcolm David Kelley", "obj": "actor", "pop": 15171, "ids": [7341, 52465, 7524, 32637, 3389, 5901, 1467, 1348, 13448, 72], "a": 8, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[2158], [18856], [16855]]}, {"prop": "occupation", "subj": "Jean-Baptiste de Lamarck", "obj": "biologist", "pop": 14972, "ids": [17657, 63, 59378, 29501, 59321, 1564, 1494, 20170, 2108, 5901, 1467, 1348, 2998, 13924, 72], "a": 12, "n": 2, "alts": ["biologist", "biologists"], "dis": [[37143, 1466], [4925, 12652], [3568, 3739]]}, {"prop": "occupation", "subj": "Mark T. Vande Hei", "obj": "astronaut", "pop": 14410, "ids": [5378, 1408, 72, 1690, 51437, 2063, 59325, 5901, 1467, 1348, 23290, 37060, 72], "a": 10, "n": 2, "alts": ["astronaut", "spacionaut", "astronauts", "space traveller"], "dis": [[16410, 1466], [4841, 1466], [1616, 10833]]}, {"prop": "occupation", "subj": "Linden Ashby", "obj": "actor", "pop": 13702, "ids": [18857, 1356, 16013, 2645, 5901, 1467, 1348, 13448, 72], "a": 7, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[38778], [50014], [35684]]}, {"prop": "occupation", "subj": "Kiko Mizuhara", "obj": "actor", "pop": 11990, "ids": [1681, 26487, 1445, 1557, 12531, 5818, 5901, 1467, 1348, 13448, 72], "a": 9, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[13129], [41865], [21840]]}, {"prop": "occupation", "subj": "G.E.M.", "obj": "singer-songwriter", "pop": 11127, "ids": [1527, 72, 59364, 72, 59365, 72, 5901, 1467, 1348, 13577, 63, 50049, 72], "a": 9, "n": 3, "alts": ["singer-songwriter", "singer songwriter", "singer/songwriter", "singersongwriter", "singer-songwriter"], "dis": [[2902, 10604, 2956], [29013, 2296, 1466], [4633, 6545, 1446]]}, {"prop": "occupation", "subj": "Bed\u0159ich Smetana", "obj": "pianist", "pop": 10742, "ids": [16023, 62553, 1620, 1413, 3119, 4270, 5901, 1467, 1348, 52310, 72], "a": 9, "n": 1, "alts": ["composer", "pianist"], "dis": [[32373], [36281], [2158]]}, {"prop": "occupation", "subj": "Dash", "obj": "record producer", "pop": 10374, "ids": [1479, 2116, 5901, 1467, 1348, 3175, 14109, 72], "a": 5, "n": 2, "alts": ["record producer", "music producer"], "dis": [[15859, 7481], [4958, 1466], [10790, 18423]]}, {"prop": "occupation", "subj": "Mariana Vicente", "obj": "actor", "pop": 10122, "ids": [3318, 11299, 37059, 23360, 5901, 1467, 1348, 13448, 72], "a": 7, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[7415], [50014], [25319]]}, {"prop": "occupation", "subj": "Ameer Sultan", "obj": "film director", "pop": 9708, "ids": [1420, 3469, 1349, 54454, 5901, 1467, 1348, 4101, 8084, 72], "a": 7, "n": 2, "alts": ["film director", "movie director", "director", "motion picture director", "actor", "actress", "actors", "actresses"], "dis": [[2076, 14743], [35859, 2230], [9935, 1629]]}, {"prop": "occupation", "subj": "Alex Reid", "obj": "actor", "pop": 8236, "ids": [9855, 44332, 5901, 1467, 1348, 13448, 72], "a": 5, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[32373], [31525], [52310]]}, {"prop": "occupation", "subj": "Josh Mankiewicz", "obj": "journalist", "pop": 8042, "ids": [24846, 1445, 2231, 1800, 39024, 5901, 1467, 1348, 23202, 72], "a": 8, "n": 1, "alts": ["journalist", "journo", "journalists"], "dis": [[36281], [51484], [16855]]}, {"prop": "occupation", "subj": "Henry Campbell-Bannerman", "obj": "diplomat", "pop": 8010, "ids": [14943, 29253, 63, 59378, 2234, 9161, 5901, 1467, 1348, 18928, 18661, 72], "a": 9, "n": 2, "alts": ["diplomat", "politician", "political leader", "political figure", "polit.", "pol"], "dis": [[37143, 1466], [23290, 37060], [3175, 14109]]}, {"prop": "occupation", "subj": "Jeet Ganguly", "obj": "composer", "pop": 7878, "ids": [29627, 1388, 39141, 8790, 5901, 1467, 1348, 21840, 72], "a": 7, "n": 1, "alts": ["composer"], "dis": [[25319], [13448], [2158]]}, {"prop": "occupation", "subj": "Tharman Shanmugaratnam", "obj": "politician", "pop": 7412, "ids": [1632, 1370, 2450, 59109, 59335, 10748, 1353, 6849, 5901, 1467, 1348, 13129, 72], "a": 11, "n": 1, "alts": ["politician", "political leader", "political figure", "polit.", "pol"], "dis": [[18856], [50014], [52310]]}, {"prop": "occupation", "subj": "Paul Hindemith", "obj": "composer", "pop": 6705, "ids": [7821, 20594, 1454, 1429, 5901, 1467, 1348, 21840, 72], "a": 7, "n": 1, "alts": ["composer", "concertmaster", "principal violin", "principal violinist", "first chair", "leader"], "dis": [[16855], [33482], [23202]]}, {"prop": "occupation", "subj": "Rhea Silvia", "obj": "Vestal Virgin", "pop": 6512, "ids": [1490, 28170, 9555, 20182, 5901, 1467, 1348, 1690, 1447, 1364, 10022, 72], "a": 7, "n": 4, "alts": ["vestal virgin"], "dis": [[15715, 59320, 3293, 1604], [10434, 20994, 10434, 21595], [7528, 59332, 1674, 1349]]}, {"prop": "occupation", "subj": "Alexander Bogdanov", "obj": "economist", "pop": 6421, "ids": [19488, 39637, 33308, 1804, 5901, 1467, 1348, 16410, 1466, 72], "a": 7, "n": 2, "alts": ["economist", "economists", "journalist", "journo", "journalists", "physician", "physicians", "medical doctor", "medical practitioner", "doctor", "medical doctors", "philosopher"], "dis": [[11990, 54217], [20972, 15951], [4034, 1671]]}, {"prop": "occupation", "subj": "Ali Hillis", "obj": "actor", "pop": 5804, "ids": [22457, 10082, 1360, 5901, 1467, 1348, 13448, 72], "a": 6, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[41865], [16855], [37764]]}, {"prop": "occupation", "subj": "William Murray, 1st Earl of Mansfield", "obj": "politician", "pop": 5766, "ids": [9753, 30226, 59342, 59320, 59345, 1389, 31677, 1379, 35330, 3548, 5901, 1467, 1348, 13129, 72], "a": 13, "n": 1, "alts": ["politician", "political leader", "political figure", "polit.", "pol"], "dis": [[16855], [38778], [36281]]}, {"prop": "occupation", "subj": "Hawthorne James", "obj": "actor", "pop": 5497, "ids": [12836, 1440, 17290, 8266, 5901, 1467, 1348, 13448, 72], "a": 7, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[37764], [21840], [36281]]}, {"prop": "occupation", "subj": "Lothrop Stoddard", "obj": "journalist", "pop": 5396, "ids": [1494, 1760, 3190, 1716, 17046, 1578, 5901, 1467, 1348, 23202, 72], "a": 9, "n": 1, "alts": ["journalist", "journo", "journalists"], "dis": [[41865], [31525], [16855]]}, {"prop": "occupation", "subj": "Marjorie Margolies-Mezvinsky", "obj": "politician", "pop": 5299, "ids": [3318, 59392, 36531, 20189, 1404, 1532, 63, 10593, 59398, 59343, 40911, 5901, 1467, 1348, 13129, 72], "a": 14, "n": 1, "alts": ["journalist", "journo", "journalists", "politician", "political leader", "political figure", "polit.", "pol"], "dis": [[35684], [25319], [33482]]}, {"prop": "occupation", "subj": "Johnson", "obj": "composer", "pop": 5158, "ids": [14318, 5901, 1467, 1348, 21840, 72], "a": 4, "n": 1, "alts": ["composer"], "dis": [[13129], [48829], [38778]]}, {"prop": "occupation", "subj": "Veronica Franco", "obj": "poet", "pop": 5119, "ids": [9519, 1350, 3945, 9204, 59326, 5901, 1467, 1348, 25473, 72], "a": 8, "n": 1, "alts": ["poet", "poetess", "bard"], "dis": [[31525], [23202], [36281]]}, {"prop": "occupation", "subj": "Mohamed Bazoum", "obj": "politician", "pop": 5087, "ids": [18968, 4094, 1469, 2215, 1375, 59335, 5901, 1467, 1348, 13129, 72], "a": 9, "n": 1, "alts": ["politician", "political leader", "political figure", "polit.", "pol"], "dis": [[25299], [32373], [37764]]}, {"prop": "occupation", "subj": "Edward F. Cox", "obj": "politician", "pop": 5034, "ids": [18223, 1493, 72, 29580, 5901, 1467, 1348, 13129, 72], "a": 7, "n": 1, "alts": ["politician", "political leader", "political figure", "polit.", "pol"], "dis": [[35684], [2158], [36281]]}, {"prop": "occupation", "subj": "Richard Sandrak", "obj": "actor", "pop": 4759, "ids": [11646, 11388, 59327, 1622, 5901, 1467, 1348, 13448, 72], "a": 7, "n": 1, "alts": ["actor", "actress", "actors", "actresses"], "dis": [[2158], [41865], [33482]]}, {"prop": "occupation", "subj": "Edward Drinker Cope", "obj": "geologist", "pop": 4752, "ids": [18223, 4450, 51738, 1419, 2921, 5901, 1467, 1348, 5202, 13924, 72], "a": 8, "n": 2, "alts": ["geologist"], "dis": [[16410, 1466], [4958, 1466], [15859, 7481]]}, {"prop": "place of birth", "subj": "Batman", "obj": "Gotham City", "pop": 193508, "ids": [48457, 1523, 6052, 1377, 1527, 1760, 1398, 4599, 72], "a": 4, "n": 4, "alts": ["gotham city", "gotham"], "dis": [[1681, 9715, 1362, 1961], [1793, 21684, 8913, 34578], [18779, 59346, 1447, 1350]]}, {"prop": "place of birth", "subj": "Chris Brown", "obj": "Tappahannock", "pop": 151737, "ids": [13765, 11676, 1523, 6052, 1377, 1408, 1898, 2420, 2234, 1827, 72], "a": 5, "n": 5, "alts": ["tappahannock", "tappahannock, virginia"], "dis": [[6551, 59326, 1420, 3646, 1352], [1494, 2310, 59346, 1406, 1356], [1681, 3061, 1742, 20494, 59336]]}, {"prop": "place of birth", "subj": "Robert Kraft", "obj": "Brookline", "pop": 85312, "ids": [9812, 1681, 3572, 1523, 6052, 1377, 15681, 1996, 72], "a": 6, "n": 2, "alts": ["brookline", "brookline, massachusetts"], "dis": [[12607, 45724], [6485, 18313], [9840, 13149]]}, {"prop": "place of birth", "subj": "C. N. Annadurai", "obj": "Kanchipuram", "pop": 56779, "ids": [1419, 72, 1515, 72, 9282, 1422, 53250, 1523, 6052, 1377, 1681, 4687, 39874, 1398, 72], "a": 10, "n": 4, "alts": ["kanchipuram", "kancheepuram"], "dis": [[23115, 59334, 18937, 4481], [1493, 48328, 26599, 33167], [57669, 2801, 4423, 59327]]}, {"prop": "place of birth", "subj": "Ted Williams", "obj": "San Diego", "pop": 55160, "ids": [30433, 14603, 1523, 6052, 1377, 6312, 19564, 72], "a": 5, "n": 2, "alts": ["san diego", "san diego, california", "sd", "america's finest city", "sandi", "the birthplace of california"], "dis": [[28230, 59398], [1681, 20090], [4396, 35461]]}, {"prop": "place of birth", "subj": "Steve Kerr", "obj": "Beirut", "pop": 51795, "ids": [15816, 1681, 1969, 1523, 6052, 1377, 3594, 1448, 1403, 72], "a": 6, "n": 3, "alts": ["beirut"], "dis": [[6512, 2598, 59336], [1507, 1491, 3966], [1445, 39884, 1627]]}, {"prop": "place of birth", "subj": "Chance the Rapper", "obj": "Chicago", "pop": 46633, "ids": [1793, 1659, 1358, 20746, 1576, 1523, 6052, 1377, 11241, 72], "a": 8, "n": 1, "alts": ["chicago", "chicago, illinois", "the windy city", "city by the lake", "the queen of the west", "chi-town", "chitown", "the second city", "city of chicago", "city of broad shoulders", "chi-raq", "chicago, il"], "dis": [[17422], [27939], [7610]]}, {"prop": "place of birth", "subj": "Annie Oakley", "obj": "Darke County", "pop": 45680, "ids": [55617, 19932, 3389, 1523, 6052, 1377, 15319, 1509, 3889, 72], "a": 6, "n": 3, "alts": ["darke county", "darke county, ohio"], "dis": [[9525, 59341, 35513], [1690, 2230, 54923], [34835, 5546, 3106]]}, {"prop": "place of birth", "subj": "Julius Erving", "obj": "New York City", "pop": 44653, "ids": [4156, 4830, 13344, 4172, 1523, 6052, 1377, 2404, 4436, 4599, 72], "a": 7, "n": 3, "alts": ["new york city", "nyc", "new york", "the five boroughs", "big apple", "city of new york", "ny city", "new york, new york", "new york city, new york", "new york, ny", "new york city (nyc)"], "dis": [[16081, 1352, 26944], [2270, 3834, 59339], [20460, 1367, 1347]]}, {"prop": "place of birth", "subj": "Isaac", "obj": "Canaan", "pop": 43190, "ids": [48382, 1523, 6052, 1377, 3426, 59324, 1363, 72], "a": 4, "n": 3, "alts": ["canaan"], "dis": [[49083, 59322, 4830], [2404, 4436, 4599], [1716, 1350, 5939]]}, {"prop": "place of birth", "subj": "Ed Sullivan", "obj": "New York City", "pop": 42978, "ids": [5929, 46330, 1523, 6052, 1377, 2404, 4436, 4599, 72], "a": 5, "n": 3, "alts": ["new york city", "nyc", "new york", "the five boroughs", "big apple", "city of new york", "ny city", "new york, new york", "new york city, new york", "new york, ny", "new york city (nyc)"], "dis": [[17101, 6812, 1406], [1420, 1364, 40301], [10665, 4764, 2232]]}, {"prop": "place of birth", "subj": "Dr. John", "obj": "New Orleans", "pop": 33147, "ids": [4450, 72, 4801, 1523, 6052, 1377, 2404, 28470, 72], "a": 6, "n": 2, "alts": ["new orleans", "nola", "new orleans, louisiana", "new orleans, la", "the big easy", "crescent city"], "dis": [[59025, 19321], [38688, 8661], [28230, 59398]]}, {"prop": "place of birth", "subj": "Babyface", "obj": "Indianapolis", "pop": 31327, "ids": [24978, 2648, 1523, 6052, 1377, 46197, 72], "a": 5, "n": 1, "alts": ["indianapolis", "circle city", "indy", "naptown", "crossroads of america", "racing capital of the world", "amateur sports capital of the world", "indpls", "indianapolis, indiana", "india-no-place", "railroad city"], "dis": [[40718], [19524], [59191]]}, {"prop": "place of birth", "subj": "Wang Yaping", "obj": "Yantai", "pop": 18810, "ids": [28082, 1788, 18736, 1523, 6052, 1377, 1788, 1516, 3243, 72], "a": 6, "n": 3, "alts": ["yantai", "chefoo", "zhifu", "chi-fu"], "dis": [[8486, 1398, 42135], [1793, 38476, 33167], [25416, 1564, 56030]]}, {"prop": "place of birth", "subj": "Oi Va Voi", "obj": "London", "pop": 15216, "ids": [1549, 59325, 37399, 1690, 9917, 1523, 6052, 1377, 6098, 72], "a": 8, "n": 1, "alts": ["london", "london, uk", "london, united kingdom", "london, england", "modern babylon"], "dis": [[17422], [52063], [55457]]}, {"prop": "place of birth", "subj": "Kensho Ono", "obj": "Fukuoka Prefecture", "pop": 14672, "ids": [1681, 1643, 7214, 3154, 59326, 1523, 6052, 1377, 1493, 48328, 26599, 33167, 72], "a": 8, "n": 4, "alts": ["fukuoka prefecture"], "dis": [[26068, 7290, 1844, 2874], [1609, 1438, 59331, 15189], [14726, 1805, 15392, 1382]]}, {"prop": "place of birth", "subj": "Belly", "obj": "Jenin", "pop": 14138, "ids": [7276, 1434, 1523, 6052, 1377, 20493, 1347, 72], "a": 5, "n": 2, "alts": ["jenin"], "dis": [[1515, 41604], [5279, 5938], [15329, 30791]]}, {"prop": "place of birth", "subj": "Juan Luis Guerra", "obj": "Santo Domingo", "pop": 13336, "ids": [27604, 33491, 4255, 1349, 1765, 1523, 6052, 1377, 21109, 59326, 1479, 19435, 59326, 72], "a": 8, "n": 5, "alts": ["santo domingo", "santo domingo, dominican republic", "santo domingo de guzman", "santo domingo de guzm\u00e1n"], "dis": [[6551, 59326, 4407, 1363, 11561], [1793, 2349, 2647, 1370, 59331], [6551, 59326, 1420, 3646, 1352]]}, {"prop": "place of birth", "subj": "John Spencer-Churchill, 11th Duke of Marlborough", "obj": "Blenheim Palace", "pop": 13325, "ids": [4801, 43738, 63, 2778, 4613, 1487, 59342, 59320, 59345, 59345, 1440, 22045, 1379, 58573, 26155, 1523, 6052, 1377, 3474, 1356, 22901, 29225, 72], "a": 18, "n": 4, "alts": ["blenheim palace"], "dis": [[1529, 1585, 59331, 23308], [1408, 3418, 7443, 59324], [1445, 1501, 3761, 1356]]}, {"prop": "place of birth", "subj": "Camillo Benso di Cavour", "obj": "Turin", "pop": 10620, "ids": [7751, 25324, 1469, 1643, 59326, 2256, 1419, 9783, 1523, 6052, 1377, 9332, 1347, 72], "a": 11, "n": 2, "alts": ["turin", "torino", "turin, italy"], "dis": [[1612, 32952], [2404, 56589], [5279, 5938]]}, {"prop": "place of birth", "subj": "Brian Kerwin", "obj": "Chicago", "pop": 9451, "ids": [19562, 36239, 6329, 1523, 6052, 1377, 11241, 72], "a": 6, "n": 1, "alts": ["chicago", "chicago, illinois", "the windy city", "city by the lake", "the queen of the west", "chi-town", "chitown", "the second city", "city of chicago", "city of broad shoulders", "chi-raq", "chicago, il"], "dis": [[37301], [53823], [49320]]}, {"prop": "place of birth", "subj": "Marnus Labuschagne", "obj": "Klerksdorp", "pop": 9376, "ids": [1445, 2619, 1406, 13381, 1406, 1439, 32847, 1523, 6052, 1377, 1681, 3061, 1742, 20494, 59336, 72], "a": 10, "n": 5, "alts": ["klerksdorp"], "dis": [[1494, 2310, 59346, 1406, 1356], [21109, 59326, 1479, 19435, 59326], [30266, 1391, 1491, 15457, 38982]]}, {"prop": "place of birth", "subj": "William Osler", "obj": "Ontario", "pop": 9249, "ids": [9753, 16294, 3061, 1523, 6052, 1377, 16471, 72], "a": 6, "n": 1, "alts": ["ontario", "ont", "on"], "dis": [[15947], [42188], [36948]]}, {"prop": "place of birth", "subj": "Nikolai Kinski", "obj": "Paris", "pop": 8870, "ids": [22217, 1404, 3243, 1681, 38450, 1523, 6052, 1377, 11225, 72], "a": 8, "n": 1, "alts": ["paris", "city of light", "paris, france"], "dis": [[52541], [52063], [25485]]}, {"prop": "place of birth", "subj": "Julee Cruise", "obj": "Creston", "pop": 8179, "ids": [1609, 2393, 59321, 55434, 1523, 6052, 1377, 1419, 4939, 1350, 72], "a": 7, "n": 3, "alts": ["creston", "creston, iowa"], "dis": [[10665, 4764, 2232], [48030, 6181, 1403], [2861, 6812, 11561]]}, {"prop": "place of birth", "subj": "Yuriy Vernydub", "obj": "Zhytomyr", "pop": 7397, "ids": [1788, 10134, 59339, 48188, 3198, 1484, 1523, 6052, 1377, 57669, 2801, 4423, 59327, 72], "a": 9, "n": 4, "alts": ["zhytomyr", "zhitomir", "\u017eyt\u00f3myr", "\u017eit\u00f3mir", "\u017cytomierz", "\u017eitomir"], "dis": [[1495, 2874, 1509, 2305], [6551, 59324, 1667, 1687], [4544, 6778, 1448, 1639]]}, {"prop": "place of birth", "subj": "Trae tha Truth", "obj": "Houston", "pop": 5856, "ids": [18565, 59321, 1393, 59324, 44566, 1523, 6052, 1377, 17422, 72], "a": 8, "n": 1, "alts": ["houston", "houston, texas", "space city", "h-town", "city of houston", "ushou", "houston, tx"], "dis": [[58433], [37301], [32253]]}, {"prop": "place of birth", "subj": "Zara Turner", "obj": "Belfast", "pop": 5569, "ids": [2572, 5818, 34096, 1523, 6052, 1377, 53823, 72], "a": 6, "n": 1, "alts": ["belfast", "b\u00e9al feirste", "beal feirste", "belfast, ireland"], "dis": [[37198], [52563], [26835]]}, {"prop": "place of birth", "subj": "Jean Arp", "obj": "Strasbourg", "pop": 5307, "ids": [17657, 2548, 59336, 1523, 6052, 1377, 6066, 1382, 36625, 72], "a": 6, "n": 3, "alts": ["strasbourg", "stra\u00dfburg", "strassburg", "schdroosburi", "strossburi", "strossburig", "strosburi"], "dis": [[34835, 5546, 3106], [1452, 1821, 1350], [1681, 61843, 2642]]}, {"prop": "place of birth", "subj": "Richie Ashburn", "obj": "Tilden", "pop": 5130, "ids": [8213, 1778, 16013, 19079, 1523, 6052, 1377, 1408, 1735, 1356, 72], "a": 7, "n": 3, "alts": ["tilden", "tilden, nebraska"], "dis": [[3426, 59324, 1363], [1495, 23777, 2272], [1690, 1841, 28801]]}, {"prop": "place of birth", "subj": "Lucia de Berk", "obj": "The Hague", "pop": 4872, "ids": [14811, 1627, 1564, 8599, 59346, 1523, 6052, 1377, 1507, 1491, 3966, 72], "a": 8, "n": 3, "alts": ["the hague", "den haag", "'s-gravenhage", "haag"], "dis": [[17101, 6812, 1406], [34490, 9214, 46643], [3318, 1435, 5203]]}, {"prop": "place of birth", "subj": "Edward Drinker Cope", "obj": "Philadelphia", "pop": 4752, "ids": [18223, 4450, 51738, 1419, 2921, 1523, 6052, 1377, 19485, 72], "a": 8, "n": 1, "alts": ["philadelphia", "philly", "city of brotherly love", "cradle of liberty", "philadelphia, pennsylvania", "city of philadelphia", "philadelphia, pa"], "dis": [[30233], [56697], [8225]]}, {"prop": "place of birth", "subj": "Peter Philpott", "obj": "Manly", "pop": 4354, "ids": [11238, 7078, 59336, 3016, 1523, 6052, 1377, 5174, 1434, 72], "a": 7, "n": 2, "alts": ["manly", "manly, new south wales", "manly, new south wales, australia"], "dis": [[1493, 33990], [11043, 44533], [4516, 49579]]}, {"prop": "place of birth", "subj": "Moses the Black", "obj": "Egypt", "pop": 4295, "ids": [1445, 8569, 1358, 6782, 1523, 6052, 1377, 16750, 72], "a": 7, "n": 1, "alts": ["egypt", "republic of egypt", "eg", "egy", "\ud83c\uddea\ud83c\uddec", "arab republic of egypt", "arab rep. egypt", "rep. egypt"], "dis": [[15280], [34688], [46525]]}, {"prop": "place of birth", "subj": "Charles Joseph Bonaparte", "obj": "Baltimore", "pop": 3831, "ids": [12583, 15089, 16007, 1461, 48343, 1523, 6052, 1377, 28130, 72], "a": 8, "n": 1, "alts": ["baltimore", "baltimore, maryland", "city of baltimore", "baltimore city", "charm city", "b more"], "dis": [[46102], [34688], [45100]]}, {"prop": "place of birth", "subj": "Michelle Renaudddd", "obj": "Mexico City", "pop": 3821, "ids": [33382, 14666, 7839, 2429, 59332, 1523, 6052, 1377, 11198, 4599, 72], "a": 8, "n": 2, "alts": ["mexico city", "mexico d.f.", "ciudad de m\u00e9xico", "city of mexico", "mexico city, mexico", "cdmx", "mexico"], "dis": [[4516, 49579], [25459, 3889], [1420, 3458]]}, {"prop": "place of birth", "subj": "Diotima of Mantinea", "obj": "Mantineia", "pop": 3744, "ids": [7196, 1411, 11915, 1379, 53869, 22152, 1523, 6052, 1377, 1445, 39884, 1627, 72], "a": 9, "n": 3, "alts": ["mantineia", "mantinia", "ancient mantineia"], "dis": [[1529, 11583, 7362], [40389, 1505, 2157], [7654, 1808, 4270]]}, {"prop": "place of birth", "subj": "Erasmo Carlos", "obj": "Rio de Janeiro", "pop": 3720, "ids": [1529, 10039, 13011, 31264, 1523, 6052, 1377, 25416, 1564, 56030, 72], "a": 7, "n": 3, "alts": ["rio de janeiro", "rio", "rio de janeiro city", "r\u00edo", "r\u00edo de janeiro"], "dis": [[14726, 1576, 53647], [1452, 1359, 20366], [1469, 1470, 45724]]}, {"prop": "place of birth", "subj": "Ko Wen-je", "obj": "Hsinchu", "pop": 3585, "ids": [31858, 1495, 1356, 63, 14670, 1523, 6052, 1377, 1491, 3706, 1439, 59334, 72], "a": 8, "n": 4, "alts": ["hsinchu", "xinzhu"], "dis": [[26068, 7290, 1844, 2874], [1408, 5533, 59334, 4270], [1549, 59325, 24260, 1444]]}, {"prop": "place of birth", "subj": "Paris", "obj": "San Francisco", "pop": 3024, "ids": [11225, 1523, 6052, 1377, 6312, 14241, 72], "a": 4, "n": 2, "alts": ["san francisco", "san francisco, california", "the city by the bay", "sf", "sfo", "the golden city", "paris of the west", "baghdad by the bay", "san francisco county", "san francisco county, california", "san fran", "frisco", "city and county of san francisco"], "dis": [[11209, 12089], [23643, 40962], [38238, 19079]]}, {"prop": "place of birth", "subj": "Roman Shukhevych", "obj": "Lviv", "pop": 3019, "ids": [9486, 2419, 3776, 1351, 15794, 1439, 1523, 6052, 1377, 1494, 47115, 72], "a": 9, "n": 2, "alts": ["krakovets", "lviv", "lw\u00f3w", "lemberg", "lvov", "lwow", "l'viv", "lemberik", "l'vov", "leopol", "lemberg, austria"], "dis": [[9881, 46705], [7213, 42471], [1681, 20090]]}, {"prop": "place of birth", "subj": "Rahi Masoom Raza", "obj": "Ghazipur", "pop": 2930, "ids": [36835, 59325, 21401, 7619, 1490, 18300, 1523, 6052, 1377, 18660, 2215, 39874, 72], "a": 9, "n": 3, "alts": ["ghazipur", "ghazeepore", "gauspur", "ghazipour", "gadjipur", "ghazipoor", "gazipoor"], "dis": [[1413, 2233, 22901], [4002, 20170, 43073], [1413, 4343, 1371]]}, {"prop": "place of birth", "subj": "Joanna Elisabeth of Holstein-Gottorp", "obj": "Gottorf Castle", "pop": 2796, "ids": [14195, 13808, 5279, 1360, 16175, 1379, 6875, 14976, 63, 59394, 3016, 27315, 1523, 6052, 1377, 1527, 3016, 43653, 21500, 72], "a": 15, "n": 4, "alts": ["gottorf castle"], "dis": [[53641, 7550, 1381, 1412], [6620, 1657, 4137, 1363], [1493, 48328, 26599, 33167]]}, {"prop": "place of birth", "subj": "Jeanne de Casalis", "obj": "Basutoland", "pop": 2730, "ids": [17657, 1861, 1564, 15547, 29716, 1523, 6052, 1377, 12209, 1403, 1404, 1483, 72], "a": 8, "n": 4, "alts": ["basutoland"], "dis": [[16125, 1622, 1370, 5818], [1793, 21684, 8913, 34578], [1479, 20882, 18313, 39476]]}, {"prop": "place of birth", "subj": "Afonso II of Portugal", "obj": "Coimbra", "pop": 2551, "ids": [15236, 1350, 1656, 4401, 1379, 25720, 1523, 6052, 1377, 4154, 1400, 6457, 72], "a": 9, "n": 3, "alts": ["coimbra", "coimbra municipality"], "dis": [[28763, 5399, 56270], [5816, 2453, 6511], [3426, 2072, 5433]]}, {"prop": "place of birth", "subj": "Parnelli Jones", "obj": "Texarkana", "pop": 2519, "ids": [1452, 2619, 20414, 13523, 1523, 6052, 1377, 7654, 1808, 4270, 72], "a": 7, "n": 3, "alts": ["texarkana", "texarkana, arkansas"], "dis": [[15319, 1509, 3889], [15319, 3493, 10730], [10665, 4764, 2232]]}, {"prop": "place of birth", "subj": "Jaswant Singh of Marwar", "obj": "Jodhpur", "pop": 2478, "ids": [1609, 1382, 52893, 23689, 1379, 3318, 12711, 1523, 6052, 1377, 1609, 1438, 59331, 15189, 72], "a": 10, "n": 4, "alts": ["jodhpur"], "dis": [[1681, 4687, 39874, 1398], [2404, 24855, 1456, 32733], [1469, 2647, 8569, 1627]]}, {"prop": "place of birth", "subj": "Halloween", "obj": "Tijuana", "pop": 2360, "ids": [28920, 1523, 6052, 1377, 1408, 5533, 59334, 4270, 72], "a": 4, "n": 4, "alts": ["tijuana"], "dis": [[1445, 1514, 1603, 59343], [1495, 2874, 1509, 2305], [3474, 1356, 22901, 29225]]}, {"prop": "place of birth", "subj": "Martin Behaim", "obj": "Nuremberg", "pop": 2356, "ids": [12489, 3594, 59331, 4137, 1523, 6052, 1377, 1515, 1539, 3391, 59338, 72], "a": 7, "n": 4, "alts": ["nuremberg", "n\u00fcrnberg", "kreisfreie stadt n\u00fcrnberg", "n\u00fcremberg"], "dis": [[23115, 59334, 18937, 4481], [1452, 1511, 2564, 59324], [59320, 63323, 1466, 1431]]}, {"prop": "place of birth", "subj": "Eduardo Hurtado", "obj": "Esmeraldas", "pop": 2339, "ids": [56238, 21711, 1491, 7362, 5620, 1523, 6052, 1377, 14726, 1805, 15392, 1382, 72], "a": 8, "n": 4, "alts": ["esmeraldas", "esmeraldas (ecuador)", "esmeraldas, ecuador", "san mateo de las esmeraldas"], "dis": [[18779, 59346, 1447, 1350], [11209, 1445, 5329, 1360], [3015, 4423, 2643, 59324]]}, {"prop": "place of birth", "subj": "Sue Vertue", "obj": "Surrey", "pop": 2281, "ids": [45052, 28864, 1511, 1523, 6052, 1377, 49841, 72], "a": 6, "n": 1, "alts": ["surrey"], "dis": [[30276], [6991], [16016]]}, {"prop": "place of birth", "subj": "Tony Clark", "obj": "Newton", "pop": 2272, "ids": [22313, 24380, 1523, 6052, 1377, 21712, 72], "a": 5, "n": 1, "alts": ["newton", "newton, kansas"], "dis": [[37198], [47864], [53684]]}, {"prop": "place of birth", "subj": "Petar Stamboli\u0107", "obj": "Ivanjica", "pop": 2254, "ids": [8996, 1370, 1716, 3502, 15903, 61645, 1523, 6052, 1377, 36545, 59392, 3945, 72], "a": 9, "n": 3, "alts": ["ivanjica"], "dis": [[1529, 1539, 6046], [1681, 2875, 59346], [1413, 4343, 1371]]}, {"prop": "place of birth", "subj": "Hittman", "obj": "Los Angeles", "pop": 2234, "ids": [1491, 2349, 2450, 1523, 6052, 1377, 11209, 12089, 72], "a": 6, "n": 2, "alts": ["los angeles", "los angeles, california", "pink city", "the town of our lady the queen of the angels of the little portion", "la la land", "tinsel town", "city of angels", "city of los angeles", "la, california", "l.a.", "la", "double dubuque", "los \u00e1ngeles", "los angeles, ca"], "dis": [[1788, 13221], [33772, 1600], [12607, 45724]]}, {"prop": "place of birth", "subj": "Ge You", "obj": "Beijing", "pop": 2195, "ids": [4396, 2292, 1523, 6052, 1377, 26823, 72], "a": 5, "n": 1, "alts": ["beijing", "peking", "beiping", "peiping", "yanjing", "zhongdu", "khanbaliq", "bj", "shun tian fu", "pekin", "beijing"], "dis": [[22174], [7610], [51212]]}, {"prop": "place of birth", "subj": "Dave Marsh", "obj": "Detroit", "pop": 2144, "ids": [21758, 20065, 1523, 6052, 1377, 25216, 72], "a": 5, "n": 1, "alts": ["detroit", "motor city", "detroit, michigan", "detroiit", "detroit, mi", "the d"], "dis": [[25446], [24650], [31881]]}, {"prop": "place of birth", "subj": "Jiang Mianheng", "obj": "Shanghai", "pop": 2081, "ids": [56182, 1445, 1770, 17222, 1523, 6052, 1377, 32544, 72], "a": 7, "n": 1, "alts": ["shanghai", "hu", "shang hai", "shanghai, china"], "dis": [[16016], [27939], [25485]]}, {"prop": "place of birth", "subj": "Liz", "obj": "Tarzana", "pop": 1954, "ids": [51438, 1523, 6052, 1377, 29099, 59398, 4270, 72], "a": 4, "n": 3, "alts": ["tarzana", "tarzana, los angeles", "tarzana, california"], "dis": [[13133, 2549, 52279], [2270, 3834, 59339], [1445, 39884, 1627]]}, {"prop": "place of birth", "subj": "Tim Armstrong", "obj": "United States of America", "pop": 1953, "ids": [11033, 49740, 1523, 6052, 1377, 3532, 4025, 1379, 6263, 72], "a": 5, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[4544, 1456, 1382, 1382], [53641, 7550, 1381, 1412], [1408, 5533, 59334, 4270]]}, {"prop": "place of birth", "subj": "Ry\u016bzabur\u014d \u014ctomo", "obj": "Tokyo", "pop": 1898, "ids": [27650, 62258, 59398, 1456, 1407, 61843, 59320, 70025, 25652, 59326, 1523, 6052, 1377, 22174, 72], "a": 13, "n": 1, "alts": ["tokyo", "t\u014dky\u014d", "t\u00f4ky\u00f4", "tokyo-to", "tokyo metropolitan prefecture", "t\u014dky\u014d-to", "t\u00f4ky\u00f4-to", "tokyo metropolis", "tokio", "tokyo prefecture"], "dis": [[17422], [39406], [6098]]}, {"prop": "place of birth", "subj": "Roman Nose", "obj": "Colorado", "pop": 1891, "ids": [9486, 1515, 1721, 1523, 6052, 1377, 15947, 72], "a": 6, "n": 1, "alts": ["colorado", "co", "state of colorado", "colo."], "dis": [[52063], [25216], [27920]]}, {"prop": "place of birth", "subj": "Angela Leong", "obj": "Guangzhou", "pop": 1871, "ids": [50961, 3013, 1635, 1523, 6052, 1377, 45231, 35497, 72], "a": 6, "n": 2, "alts": ["guangzhou", "canton", "kwangchow", "kwongchow"], "dis": [[2404, 56589], [9332, 1347], [1690, 2984]]}, {"prop": "place of birth", "subj": "Khalid bin Barghash of Zanzibar", "obj": "Zanzibar", "pop": 1865, "ids": [14196, 5451, 12914, 5162, 1695, 2116, 1379, 2572, 16757, 1599, 1370, 1523, 6052, 1377, 2572, 16757, 1599, 1370, 72], "a": 14, "n": 4, "alts": ["zanzibar", "menuthias"], "dis": [[9824, 59324, 1484, 1363], [1452, 1511, 2564, 59324], [1716, 59324, 1444, 2232]]}, {"prop": "place of birth", "subj": "Blair Thornton", "obj": "Vancouver", "pop": 1844, "ids": [54730, 35689, 45126, 1523, 6052, 1377, 25485, 72], "a": 6, "n": 1, "alts": ["vancouver", "city of vancouver", "vancouver, bc", "vancouver, british columbia"], "dis": [[46197], [15947], [42188]]}, {"prop": "place of birth", "subj": "Brent Maddock", "obj": "United States of America", "pop": 1700, "ids": [42968, 57514, 1827, 1523, 6052, 1377, 3532, 4025, 1379, 6263, 72], "a": 6, "n": 4, "alts": ["united states of america", "the united states of america", "america", "u.s.a.", "usa", "u.s.", "us", "the us", "the usa", "us of a", "the united states", "u. s. a.", "u. s.", "the states", "the u.s.", "'merica", "u.s", "united states", "'murica"], "dis": [[1452, 1698, 1755, 36942], [1490, 3016, 1787, 1394], [23115, 59334, 18937, 4481]]}, {"prop": "place of birth", "subj": "Francisco Fern\u00e1ndez de la Cueva, 10th Duke of Alburquerque", "obj": "Genoa", "pop": 1678, "ids": [14241, 26892, 53778, 13059, 1564, 3452, 1419, 1511, 10682, 59342, 59320, 59345, 59344, 1440, 22045, 1379, 2270, 10337, 18937, 4481, 1523, 6052, 1377, 9840, 13149, 72], "a": 23, "n": 2, "alts": ["genoa", "genoa, italy", "genova, italy", "genova"], "dis": [[1515, 41604], [17774, 4238], [51493, 21500]]}, {"prop": "place of birth", "subj": "Teri DeSario", "obj": "Miami", "pop": 1673, "ids": [11622, 59325, 2748, 59352, 6716, 1523, 6052, 1377, 20847, 72], "a": 8, "n": 1, "alts": ["miami", "city of miami", "miami, florida", "miami, fl"], "dis": [[21712], [28130], [15466]]}, {"prop": "place of birth", "subj": "Karim Abdel Aziz", "obj": "Cairo", "pop": 1534, "ids": [12065, 1400, 4589, 9199, 21264, 1557, 1523, 6052, 1377, 58433, 72], "a": 9, "n": 1, "alts": ["cairo", "cairo, egypt"], "dis": [[45678], [56087], [20847]]}, {"prop": "place of birth", "subj": "Offer Nissim", "obj": "Tel Aviv", "pop": 1516, "ids": [47295, 1515, 1677, 1400, 1523, 6052, 1377, 19087, 9383, 1498, 72], "a": 7, "n": 3, "alts": ["tel aviv", "tel aviv-yafo", "tel-aviv", "tel aviv city", "tel aviv-jaffa", "tel-aviv", "tel aviv", "tel-aviv jaffa", "tel aviv jaffa", "tel aviv-jafo", "tel aviv, israel", "tlv"], "dis": [[1788, 1516, 3243], [3426, 59326, 1382], [18036, 1435, 6046]]}, {"prop": "place of birth", "subj": "Samantha James", "obj": "Los Angeles", "pop": 1501, "ids": [9137, 46349, 8266, 1523, 6052, 1377, 11209, 12089, 72], "a": 6, "n": 2, "alts": ["los angeles", "los angeles, california", "pink city", "the town of our lady the queen of the angels of the little portion", "la la land", "tinsel town", "city of angels", "city of los angeles", "la, california", "l.a.", "la", "double dubuque", "los \u00e1ngeles", "los angeles, ca"], "dis": [[4396, 58134], [23192, 1485], [33127, 11875]]}, {"prop": "producer", "subj": "Halloween", "obj": "John Carpenter", "pop": 1137224, "ids": [28920, 1523, 5908, 1520, 4801, 51589, 9744, 72], "a": 4, "n": 3, "alts": ["john carpenter", "john howard carpenter"], "dis": [[20652, 40422, 28283], [14881, 2518, 14348], [7821, 29379, 25742]]}, {"prop": "producer", "subj": "Marvel Cinematic Universe", "obj": "Kevin Feige", "pop": 490576, "ids": [34496, 25987, 34394, 34389, 1523, 5908, 1520, 20178, 4063, 17899, 72], "a": 7, "n": 3, "alts": ["kevin feige"], "dis": [[10073, 29620, 1349], [55751, 8250, 7662], [13765, 49460, 1352]]}, {"prop": "producer", "subj": "Game of Thrones", "obj": "David Benioff", "pop": 428774, "ids": [10164, 1379, 1632, 32461, 1523, 5908, 1520, 7524, 7370, 1839, 1595, 72], "a": 7, "n": 4, "alts": ["david benioff", "david friedman"], "dis": [[43250, 2232, 20989, 4238], [7341, 52465, 1646, 2206], [8049, 1549, 5093, 1349]]}, {"prop": "producer", "subj": "Titanic", "obj": "James Cameron", "pop": 344637, "ids": [41491, 1374, 1523, 5908, 1520, 8266, 36890, 72], "a": 5, "n": 2, "alts": ["james cameron", "james francis cameron", "jim cameron"], "dis": [[31235, 22393], [18223, 16015], [9114, 35146]]}, {"prop": "producer", "subj": "The Shining", "obj": "Stanley Kubrick", "pop": 326522, "ids": [1507, 2419, 5258, 1523, 5908, 1520, 31808, 1681, 1484, 13654, 72], "a": 6, "n": 4, "alts": ["stanley kubrick"], "dis": [[11388, 1765, 23347, 1827], [47952, 11140, 38509, 9723], [1408, 63, 10917, 1406]]}, {"prop": "producer", "subj": "Gladiator", "obj": "Ridley Scott", "pop": 326466, "ids": [8176, 15755, 2073, 1523, 5908, 1520, 49926, 3389, 9610, 72], "a": 6, "n": 3, "alts": ["ridley scott", "sir ridley scott"], "dis": [[4801, 4592, 2162], [43319, 21927, 1363], [12980, 27609, 1677]]}, {"prop": "producer", "subj": "Halloween III: Season of the Witch", "obj": "Debra Hill", "pop": 304403, "ids": [28920, 11223, 59358, 21559, 1379, 1358, 1495, 7212, 1523, 5908, 1520, 2748, 6457, 10082, 72], "a": 11, "n": 3, "alts": ["debra hill", "john carpenter", "john howard carpenter"], "dis": [[12209, 1394, 26507], [51830, 16949, 59334], [12980, 27494, 3389]]}, {"prop": "producer", "subj": "Friends", "obj": "David Crane", "pop": 294790, "ids": [27297, 1523, 5908, 1520, 7524, 5758, 2611, 72], "a": 4, "n": 3, "alts": ["marta kauffman", "marta fran kauffman", "david crane"], "dis": [[1494, 1659, 30617], [4801, 7302, 46137], [35037, 40339, 7662]]}, {"prop": "producer", "subj": "Alien", "obj": "David Giler", "pop": 273178, "ids": [2270, 7607, 1523, 5908, 1520, 7524, 1527, 9038, 72], "a": 5, "n": 3, "alts": ["david giler", "walter hill"], "dis": [[1469, 1402, 1625], [11238, 11318, 2552], [6483, 31495, 2536]]}, {"prop": "producer", "subj": "Avatar", "obj": "James Cameron", "pop": 237293, "ids": [9383, 12361, 1523, 5908, 1520, 8266, 36890, 72], "a": 5, "n": 2, "alts": ["james cameron", "james francis cameron", "jim cameron"], "dis": [[9753, 16032], [13738, 29253], [10518, 22045]]}, {"prop": "producer", "subj": "The Blair Witch Project", "obj": "Gregg Hale", "pop": 172292, "ids": [1507, 54730, 1495, 7212, 6900, 1523, 5908, 1520, 18432, 59338, 1491, 2160, 72], "a": 8, "n": 4, "alts": ["gregg hale"], "dis": [[34967, 7213, 1472, 3116], [44421, 1690, 1367, 2134], [17678, 1419, 1400, 1640]]}, {"prop": "producer", "subj": "The Terminal", "obj": "Steven Spielberg", "pop": 156195, "ids": [1507, 40975, 1523, 5908, 1520, 28204, 2870, 20090, 7662, 72], "a": 5, "n": 4, "alts": ["steven spielberg", "steven allan spielberg", "spielberg"], "dis": [[27739, 16162, 26729, 2037], [1612, 59378, 59370, 59344], [22313, 8729, 2103, 59325]]}, {"prop": "producer", "subj": "Back to the Future", "obj": "Bob Gale", "pop": 150710, "ids": [10203, 1385, 1358, 19186, 1523, 5908, 1520, 13187, 1527, 2160, 72], "a": 7, "n": 3, "alts": ["neil canton", "bob gale", "michael robert gale"], "dis": [[26507, 9553, 4140], [29099, 7133, 23689], [19378, 1778, 48856]]}, {"prop": "producer", "subj": "The Departed", "obj": "Brad Pitt", "pop": 137029, "ids": [1507, 5129, 1522, 1367, 1523, 5908, 1520, 18878, 1452, 2349, 72], "a": 7, "n": 3, "alts": ["brad pitt", "william bradley pitt", "brad jolie-pitt", "graham king"], "dis": [[7063, 16433, 59332], [4801, 2572, 2355], [33596, 13337, 1349]]}, {"prop": "producer", "subj": "Seven", "obj": "Arnold Kopelson", "pop": 136347, "ids": [28625, 1523, 5908, 1520, 39861, 1681, 1464, 21800, 72], "a": 4, "n": 4, "alts": ["arnold kopelson"], "dis": [[28860, 2079, 1469, 8650], [14972, 1382, 7209, 59324], [4908, 1347, 1861, 56972]]}, {"prop": "producer", "subj": "Let It Be", "obj": "Phil Spector", "pop": 134729, "ids": [2965, 1826, 3594, 1523, 5908, 1520, 7078, 21243, 1355, 72], "a": 6, "n": 3, "alts": ["phil spector", "harvey phillip spector", "phillip harvey spector", "00029306983 ipi"], "dis": [[11367, 26932, 9610], [9753, 3030, 1483], [37277, 4384, 58382]]}, {"prop": "producer", "subj": "Aliens", "obj": "Walter Hill", "pop": 132071, "ids": [22457, 1643, 1523, 5908, 1520, 28143, 10082, 72], "a": 5, "n": 2, "alts": ["david giler", "gale anne hurd", "valhalla motion pictures", "walter hill"], "dis": [[29151, 22701], [14628, 19163], [5430, 8902]]}, {"prop": "producer", "subj": "Carrie", "obj": "Brian De Palma", "pop": 131478, "ids": [3684, 9477, 1523, 5908, 1520, 19562, 2748, 7799, 2728, 72], "a": 5, "n": 4, "alts": ["brian de palma", "brian russell de palma"], "dis": [[1420, 1742, 56901, 34929], [1408, 51588, 5657, 14437], [11181, 2888, 14339, 59321]]}, {"prop": "producer", "subj": "19", "obj": "Jim Abbiss", "pop": 129345, "ids": [59320, 59345, 59373, 1523, 5908, 1520, 12980, 27609, 1677, 72], "a": 6, "n": 3, "alts": ["eg white", "00160488174 ipi", "francis anthony white", "jim abbiss"], "dis": [[4801, 1419, 2160], [43953, 1490, 2933], [12344, 3318, 3972]]}, {"prop": "producer", "subj": "The Social Network", "obj": "Ce\u00e1n Chaffin", "pop": 122369, "ids": [1507, 10847, 8516, 1523, 5908, 1520, 29279, 16693, 1793, 57487, 72], "a": 6, "n": 4, "alts": ["ce\u00e1n chaffin", "cean chaffin"], "dis": [[14881, 1469, 1481, 3243], [43250, 2232, 20989, 4238], [12583, 3317, 1558, 9747]]}, {"prop": "producer", "subj": "Toy Story", "obj": "Ralph Guggenheim", "pop": 120670, "ids": [23706, 19650, 1523, 5908, 1520, 42897, 1527, 2729, 1356, 22901, 72], "a": 5, "n": 5, "alts": ["ralph guggenheim"], "dis": [[1490, 3207, 21339, 1364, 2134], [4801, 45973, 63, 39184, 1349], [25690, 27266, 1419, 1994, 1558]]}, {"prop": "producer", "subj": "Flight", "obj": "Robert Zemeckis", "pop": 104318, "ids": [35755, 1523, 5908, 1520, 9812, 2572, 1454, 9725, 1360, 72], "a": 4, "n": 5, "alts": ["robert zemeckis", "robert l. zemeckis", "robert lee zemeckis"], "dis": [[1632, 59339, 30007, 2549, 1363], [5378, 1549, 59361, 13540, 48568], [50319, 10674, 1494, 2519, 59339]]}, {"prop": "producer", "subj": "Final Destination", "obj": "Glen Morgan", "pop": 102010, "ids": [17989, 26934, 3355, 1523, 5908, 1520, 33133, 24546, 72], "a": 6, "n": 2, "alts": ["glen morgan"], "dis": [[7063, 3013], [22769, 1578], [27128, 20119]]}, {"prop": "producer", "subj": "The Holiday", "obj": "Columbia Pictures", "pop": 101218, "ids": [1507, 34015, 1523, 5908, 1520, 16016, 28283, 72], "a": 5, "n": 2, "alts": ["columbia pictures", "cpii", "columbia pictures industries, inc.", "columbia pictures entertainment", "columbia pictures corporation", "nancy meyers", "nancy jane meyers"], "dis": [[13738, 29253], [45431, 22536], [11033, 55481]]}, {"prop": "producer", "subj": "The Prestige", "obj": "Christopher Nolan", "pop": 99894, "ids": [1507, 36568, 17899, 1523, 5908, 1520, 27971, 1515, 44003, 72], "a": 6, "n": 3, "alts": ["christopher nolan", "christopher jonathan james nolan", "nolan", "christopher edward nolan"], "dis": [[9812, 7868, 54941], [12344, 3318, 3972], [29099, 7133, 23689]]}, {"prop": "producer", "subj": "E.T. the Extra-Terrestrial", "obj": "Melissa Mathison", "pop": 97322, "ids": [1529, 72, 59351, 72, 1358, 33897, 63, 36864, 44231, 1523, 5908, 1520, 54080, 6424, 4125, 72], "a": 12, "n": 3, "alts": ["melissa mathison", "melissa marie mathison", "steven spielberg", "steven allan spielberg", "spielberg"], "dis": [[24987, 12836, 1742], [20987, 14339, 1568], [51830, 16949, 59334]]}, {"prop": "producer", "subj": "Ghost", "obj": "Bruce Joel Rubin", "pop": 88655, "ids": [40220, 1523, 5908, 1520, 27229, 46319, 15616, 1347, 72], "a": 4, "n": 4, "alts": ["bruce joel rubin"], "dis": [[2870, 4601, 14972, 3846], [23302, 1356, 18559, 2036], [19562, 1527, 5912, 1349]]}, {"prop": "producer", "subj": "The Dark Side of the Moon", "obj": "Pink Floyd", "pop": 84582, "ids": [1507, 19988, 21673, 1379, 1358, 21018, 1523, 5908, 1520, 33038, 58965, 72], "a": 9, "n": 2, "alts": ["pink floyd", "the pink floyd", "the pink floyd sound"], "dis": [[9753, 16032], [35219, 46899], [6512, 39635]]}, {"prop": "producer", "subj": "Rocky", "obj": "Irwin Winkler", "pop": 83985, "ids": [47864, 1523, 5908, 1520, 10255, 6329, 1495, 1837, 3061, 72], "a": 4, "n": 5, "alts": ["irwin winkler"], "dis": [[4801, 1445, 72, 59035, 1381], [20431, 1622, 1408, 5533, 13660], [13466, 59332, 3224, 1353, 1424]]}, {"prop": "producer", "subj": "Friday the 13th Part 2", "obj": "Steve Miner", "pop": 83503, "ids": [7502, 1358, 59320, 59345, 59362, 1440, 4470, 59320, 59349, 1523, 5908, 1520, 15816, 4119, 1349, 72], "a": 12, "n": 3, "alts": ["steve miner"], "dis": [[1493, 51234, 54094], [9610, 6717, 42313], [4801, 2572, 2355]]}, {"prop": "producer", "subj": "Into the Wild", "obj": "Sean Penn", "pop": 83157, "ids": [50324, 1358, 13337, 1523, 5908, 1520, 32392, 34970, 72], "a": 6, "n": 2, "alts": ["sean penn", "sean justin penn", "pappy pariah"], "dis": [[16162, 28653], [8313, 30536], [9812, 54897]]}, {"prop": "producer", "subj": "The Fly", "obj": "Mel Brooks", "pop": 82626, "ids": [1507, 26728, 1523, 5908, 1520, 10073, 44471, 72], "a": 5, "n": 2, "alts": ["mel brooks", "melvin brooks", "melvin kaminsky"], "dis": [[9114, 27816], [29536, 22716], [4801, 21942]]}, {"prop": "producer", "subj": "Green Lantern", "obj": "Greg Berlanti", "pop": 82433, "ids": [7868, 36391, 1880, 1523, 5908, 1520, 18432, 8599, 59329, 16655, 72], "a": 6, "n": 4, "alts": ["greg berlanti", "gregory berlanti", "berlanti productions"], "dis": [[19255, 1529, 1360, 2120], [37160, 1445, 2825, 1465], [22313, 8729, 2103, 59325]]}, {"prop": "producer", "subj": "The Birds", "obj": "Alfred Hitchcock", "pop": 79684, "ids": [1507, 57172, 1523, 5908, 1520, 34683, 1491, 7212, 47402, 72], "a": 5, "n": 4, "alts": ["alfred hitchcock", "alfred joseph hitchcock", "sir alfred hitchcock"], "dis": [[2944, 1370, 24081, 14603], [37160, 1445, 2825, 1465], [9753, 1420, 72, 54934]]}, {"prop": "producer", "subj": "The Graduate", "obj": "Lawrence Turman", "pop": 79381, "ids": [1507, 34712, 1523, 5908, 1520, 25747, 9332, 2450, 72], "a": 5, "n": 3, "alts": ["lawrence turman"], "dis": [[26159, 41990, 1431], [10677, 2655, 2162], [10073, 29620, 1349]]}, {"prop": "producer", "subj": "The Mist", "obj": "Frank Darabont", "pop": 76264, "ids": [1507, 49567, 1523, 5908, 1520, 10677, 15319, 1456, 1763, 72], "a": 5, "n": 4, "alts": ["frank darabont"], "dis": [[21861, 2534, 1400, 1778], [24634, 1479, 1969, 8650], [30087, 1469, 30560, 3668]]}, {"prop": "producer", "subj": "Argo", "obj": "Grant Heslov", "pop": 75862, "ids": [2548, 3128, 1523, 5908, 1520, 22976, 1491, 1359, 16659, 72], "a": 5, "n": 4, "alts": ["george clooney", "george timothy clooney", "ben affleck", "benjami g\u00e9za affleck", "benjamin g\u00e9za affleck-boldt", "benjami geza affleck", "benjamin geza affleck-boldt", "grant heslov"], "dis": [[1494, 1659, 13539, 59328], [4801, 1494, 1537, 4493], [36890, 5281, 59408, 2825]]}, {"prop": "producer", "subj": "The Imitation Game", "obj": "Ido Ostrowsky", "pop": 75669, "ids": [1507, 1417, 3368, 1409, 10164, 1523, 5908, 1520, 1417, 5095, 58571, 1773, 16872, 72], "a": 8, "n": 5, "alts": ["teddy schwarzman", "edward frank schwarzman", "edward schwarzman", "nora grossman", "ido ostrowsky"], "dis": [[11033, 2572, 9715, 1454, 2234], [3227, 23902, 2572, 3776, 1355], [5162, 6457, 22356, 1360, 1483]]}, {"prop": "producer", "subj": "Inside Out", "obj": "Jonas Rivera", "pop": 74976, "ids": [33044, 6249, 1523, 5908, 1520, 14972, 1382, 7209, 59324, 72], "a": 5, "n": 4, "alts": ["jonas rivera", "jonas hale rivera"], "dis": [[4286, 1386, 1347, 12917], [8467, 1479, 1643, 5399], [13344, 17432, 15191, 1370]]}, {"prop": "producer", "subj": "Wings", "obj": "Lucien Hubbard", "pop": 74878, "ids": [56490, 1523, 5908, 1520, 14811, 7607, 24961, 56188, 72], "a": 4, "n": 4, "alts": ["lucien hubbard"], "dis": [[4016, 59323, 21190, 22457], [5541, 8751, 18660, 3243], [44135, 3721, 1350, 1363]]}, {"prop": "producer", "subj": "Blade", "obj": "Wesley Snipes", "pop": 73446, "ids": [3474, 1958, 1523, 5908, 1520, 58878, 19208, 30831, 72], "a": 5, "n": 3, "alts": ["wesley snipes", "wesley trent snipes"], "dis": [[9812, 5098, 12650], [30087, 8250, 45756], [1490, 4095, 14976]]}, {"prop": "producer", "subj": "Clue", "obj": "John Landis", "pop": 72498, "ids": [2534, 1511, 1523, 5908, 1520, 4801, 10726, 1360, 72], "a": 5, "n": 3, "alts": ["debra hill", "john landis"], "dis": [[8710, 32591, 2232], [9595, 43359, 3742], [56087, 20460, 1861]]}, {"prop": "producer", "subj": "Her", "obj": "Spike Jonze", "pop": 70875, "ids": [6555, 1523, 5908, 1520, 2870, 4601, 14972, 3846, 72], "a": 4, "n": 4, "alts": ["spike jonze", "adam spiegel"], "dis": [[22430, 55470, 59336, 25201], [4498, 1655, 1370, 50176], [9595, 9383, 1622, 1770]]}, {"prop": "producer", "subj": "Crash", "obj": "Don Cheadle", "pop": 69270, "ids": [5758, 2116, 1523, 5908, 1520, 7063, 1419, 3106, 1378, 72], "a": 5, "n": 4, "alts": ["don cheadle", "paul haggis", "paul edward haggis"], "dis": [[47952, 11140, 38509, 9723], [3271, 2162, 9608, 1359], [53715, 1690, 1684, 1350]]}, {"prop": "producer", "subj": "The Happening", "obj": "Sam Mercer", "pop": 65800, "ids": [1507, 41877, 6612, 1523, 5908, 1520, 9137, 10541, 2497, 72], "a": 6, "n": 3, "alts": ["sam mercer", "barry mendel"], "dis": [[46390, 1360, 44810], [8266, 38289, 3846], [9812, 1495, 1998]]}, {"prop": "producer", "subj": "2012", "obj": "Roland Emmerich", "pop": 65303, "ids": [59320, 59349, 59344, 59345, 59349, 1523, 5908, 1520, 23020, 1483, 6503, 1805, 1620, 72], "a": 8, "n": 5, "alts": ["roland emmerich"], "dis": [[27634, 1374, 2572, 1391, 21692], [7846, 1363, 59334, 2518, 32414], [1420, 3368, 52682, 41832, 3165]]}, {"prop": "producer", "subj": "Taken", "obj": "Luc Besson", "pop": 64711, "ids": [45090, 1523, 5908, 1520, 14811, 1469, 1481, 1350, 72], "a": 4, "n": 4, "alts": ["luc besson", "luc paul maurice besson"], "dis": [[28860, 2079, 1469, 8650], [7341, 52465, 1646, 2206], [1495, 1400, 1495, 14596]]}, {"prop": "producer", "subj": "The Room", "obj": "Greg Sestero", "pop": 64610, "ids": [1507, 15980, 1523, 5908, 1520, 18432, 1413, 7434, 59326, 72], "a": 5, "n": 4, "alts": ["greg sestero", "gregory sestero", "tommy wiseau"], "dis": [[22313, 1527, 2619, 9747], [4801, 1527, 1353, 2233], [20236, 1363, 3141, 49547]]}, {"prop": "producer", "subj": "Waterworld", "obj": "Kevin Costner", "pop": 64581, "ids": [10613, 11262, 1523, 5908, 1520, 20178, 13821, 2120, 72], "a": 5, "n": 3, "alts": ["kevin costner", "kevin michael costner"], "dis": [[26750, 15821, 2120], [15816, 8978, 59322], [9114, 31482, 8573]]}, {"prop": "producer", "subj": "The Reader", "obj": "Anthony Minghella", "pop": 63712, "ids": [1507, 38157, 1523, 5908, 1520, 23978, 1445, 1371, 20015, 59324, 72], "a": 5, "n": 5, "alts": ["anthony minghella", "anthony mingella", "sydney pollack", "sydney irwin pollack"], "dis": [[13344, 17432, 39332, 2079, 1439], [28204, 1413, 9701, 1640, 1695], [27634, 1374, 2572, 1391, 21692]]}, {"prop": "producer", "subj": "Idiocracy", "obj": "Mike Judge", "pop": 63202, "ids": [1417, 7931, 5841, 3267, 1523, 5908, 1520, 13738, 26628, 72], "a": 7, "n": 2, "alts": ["mike judge", "judgemental films", "michael judge", "michael craig judge", "michael c. judge"], "dis": [[33686, 38466], [8439, 41034], [9114, 27816]]}, {"prop": "producer", "subj": "The Master", "obj": "Paul Thomas Anderson", "pop": 62076, "ids": [1507, 12419, 1523, 5908, 1520, 7821, 10804, 22393, 72], "a": 5, "n": 3, "alts": ["paul thomas anderson", "p. t. anderson", "paul anderson", "pta"], "dis": [[4801, 7335, 11371], [29099, 7133, 23689], [1452, 1657, 23016]]}, {"prop": "producer", "subj": "Office Space", "obj": "Mike Judge", "pop": 62042, "ids": [9489, 14289, 1523, 5908, 1520, 13738, 26628, 72], "a": 5, "n": 2, "alts": ["mike judge", "judgemental films", "michael judge", "michael craig judge", "michael c. judge"], "dis": [[7821, 21861], [4801, 17647], [12344, 28123]]}, {"prop": "producer", "subj": "Closer", "obj": "Mike Nichols", "pop": 62010, "ids": [2534, 30452, 1523, 5908, 1520, 13738, 25844, 18866, 72], "a": 5, "n": 3, "alts": ["mike nichols", "mikhail igorevich peschkowsky"], "dis": [[16991, 32058, 1558], [21165, 1793, 1375], [45265, 40152, 19381]]}, {"prop": "producer", "subj": "Signs", "obj": "Sam Mercer", "pop": 61883, "ids": [10347, 59328, 1523, 5908, 1520, 9137, 10541, 2497, 72], "a": 5, "n": 3, "alts": ["sam mercer"], "dis": [[10674, 1445, 6242], [19378, 21927, 1595], [13765, 2534, 2190]]}, {"prop": "producer", "subj": "Hugo", "obj": "Graham King", "pop": 61534, "ids": [52288, 1523, 5908, 1520, 27954, 6132, 72], "a": 4, "n": 2, "alts": ["johnny depp", "john christopher \"johnny\" depp ii", "john christopher depp ii", "john christopher depp", "martin scorsese", "martin skorseze", "martin charles scorsese", "graham king"], "dis": [[23604, 24987], [13187, 24380], [7524, 45182]]}, {"prop": "producer", "subj": "The Internship", "obj": "Vince Vaughn", "pop": 61374, "ids": [1507, 4387, 6073, 1523, 5908, 1520, 51170, 1690, 5590, 59322, 72], "a": 6, "n": 4, "alts": ["vince vaughn", "vincent anthony vaughn", "vincent vaughn"], "dis": [[4450, 72, 1479, 1352], [55256, 1851, 17141, 1350], [9549, 9519, 59329, 7290]]}, {"prop": "producer", "subj": "Vertigo", "obj": "Paramount Pictures", "pop": 59537, "ids": [28864, 17365, 1523, 5908, 1520, 4002, 15480, 28283, 72], "a": 5, "n": 3, "alts": ["paramount pictures", "paramount pictures corporation", "paramount", "paramount filmed entertainment", "paramount gallery & entertainment", "alfred hitchcock", "alfred joseph hitchcock", "sir alfred hitchcock"], "dis": [[58896, 1490, 4072], [9114, 31482, 8573], [1529, 2129, 1454]]}, {"prop": "producer", "subj": "Animal House", "obj": "Matty Simmons", "pop": 59225, "ids": [26510, 5562, 1523, 5908, 1520, 5271, 1961, 5098, 20036, 72], "a": 5, "n": 4, "alts": ["ivan reitman", "matty simmons", "martin gerald simmons"], "dis": [[1490, 8757, 23191, 1374], [34721, 3814, 38688, 2450], [1507, 6066, 1657, 11279]]}, {"prop": "producer", "subj": "9", "obj": "Jim Lemley", "pop": 56830, "ids": [59320, 59373, 1523, 5908, 1520, 12980, 27494, 3389, 72], "a": 5, "n": 3, "alts": ["jim lemley", "timur bekmambetov", "timur nurbakhitovich bekmambetov", "timur nuruakhitovich bekmambetov", "tim burton", "timothy burton", "timothy william burton", "timothy walter burton"], "dis": [[49412, 7302, 2112], [20987, 14339, 1568], [9812, 1495, 1998]]}, {"prop": "producer", "subj": "The Rock", "obj": "Jerry Bruckheimer", "pop": 56027, "ids": [1507, 10703, 1523, 5908, 1520, 32905, 3317, 3022, 1351, 7314, 72], "a": 5, "n": 5, "alts": ["jerry bruckheimer", "jerome leon bruckheimer", "don simpson", "donald clarence simpson", "donald clarence \"don\" simpson"], "dis": [[16013, 1607, 4220, 1897, 58681], [1609, 11010, 7341, 1416, 1850], [51466, 45215, 1491, 1447, 1350]]}, {"prop": "producer", "subj": "Fast & Furious 6", "obj": "Vin Diesel", "pop": 55637, "ids": [20602, 1930, 1493, 21251, 59320, 59381, 1523, 5908, 1520, 24541, 1479, 49546, 72], "a": 9, "n": 3, "alts": ["vin diesel", "mark sinclair vincent", "mark sinclair"], "dis": [[36890, 31831, 59321], [27128, 15191, 1809], [11033, 9308, 1483]]}, {"prop": "producer", "subj": "Bee Movie", "obj": "Jerry Seinfeld", "pop": 54894, "ids": [38934, 25517, 1523, 5908, 1520, 32905, 2989, 3498, 1892, 72], "a": 5, "n": 4, "alts": ["jerry seinfeld", "jerome allen seinfeld", "cable boy", "mr. steinfeld", "jerome allen \"jerry\" seinfeld"], "dis": [[1413, 30925, 13344, 3003], [59320, 59367, 59344, 3444], [4162, 1888, 32058, 1578]]}, {"prop": "producer", "subj": "A.I. Artificial Intelligence", "obj": "Bonnie Curtis", "pop": 54344, "ids": [1420, 72, 59355, 72, 46715, 28344, 1523, 5908, 1520, 16007, 21646, 54688, 72], "a": 9, "n": 3, "alts": ["bonnie curtis", "bonnie kathleen curtis"], "dis": [[14196, 2903, 30947], [44479, 23721, 1359], [14943, 1681, 8970]]}, {"prop": "producer", "subj": "Lincoln", "obj": "Steven Spielberg", "pop": 53410, "ids": [23423, 1523, 5908, 1520, 28204, 2870, 20090, 7662, 72], "a": 4, "n": 4, "alts": ["steven spielberg", "steven allan spielberg", "spielberg"], "dis": [[19255, 2534, 1461, 2232], [12980, 59325, 28370, 3406], [32962, 2572, 3210, 1347]]}, {"prop": "producer", "subj": "Do the Right Thing", "obj": "Spike Lee", "pop": 53051, "ids": [4498, 1358, 15815, 47849, 1523, 5908, 1520, 2870, 4601, 12917, 72], "a": 7, "n": 3, "alts": ["spike lee", "sheldon lee", "shelton j. lee", "shelton jackson lee", "mars blackmon"], "dis": [[39451, 1874, 28885], [48480, 15818, 1578], [9812, 8962, 59321]]}, {"prop": "producer", "subj": "Knowing", "obj": "Alex Proyas", "pop": 52858, "ids": [16180, 1371, 1523, 5908, 1520, 9855, 2078, 59339, 1382, 72], "a": 5, "n": 4, "alts": ["alex proyas", "alexander \"alex\" proyas", "alexander proyas"], "dis": [[3318, 13417, 30310, 59321], [1420, 1742, 56901, 34929], [30922, 1469, 1778, 1640]]}, {"prop": "producer", "subj": "Gravity", "obj": "David Heyman", "pop": 52804, "ids": [3509, 11876, 1523, 5908, 1520, 7524, 2063, 32647, 72], "a": 5, "n": 3, "alts": ["david heyman", "david jonathan heyman"], "dis": [[58022, 12051, 8650], [9114, 1419, 7290], [34881, 28817, 1603]]}, {"prop": "producer", "subj": "The Faculty", "obj": "Robert Rodriguez", "pop": 52202, "ids": [1507, 27759, 1523, 5908, 1520, 9812, 54897, 72], "a": 5, "n": 2, "alts": ["robert rodriguez", "robert anthony rodr\u00edguez", "robert anthony rodriguez"], "dis": [[24846, 39282], [19711, 12694], [32392, 34970]]}, {"prop": "producer", "subj": "Mission: Impossible \u2013 Rogue Nation", "obj": "Tom Cruise", "pop": 51933, "ids": [25861, 59358, 4016, 13137, 1991, 17545, 1511, 24141, 1523, 5908, 1520, 9549, 55434, 72], "a": 11, "n": 2, "alts": ["tom cruise", "thomas cruise mapother iv", "david ellison"], "dis": [[16162, 28653], [29536, 22716], [9137, 36578]]}, {"prop": "religion", "subj": "Elizabeth II", "obj": "Anglicanism", "pop": 1291432, "ids": [19594, 4401, 1410, 1348, 2185, 1349, 1379, 52175, 2583, 72], "a": 7, "n": 2, "alts": ["anglicanism", "anglicanism, anglican church"], "dis": [[39851, 2583], [19474, 2326], [21139, 1436]]}, {"prop": "religion", "subj": "Muhammad Ali", "obj": "Sunni Islam", "pop": 706284, "ids": [42820, 22457, 1410, 1348, 2185, 1349, 1379, 9826, 7376, 15213, 72], "a": 7, "n": 3, "alts": ["nation of islam", "noi", "sunni islam", "sunni", "sunnite muslims", "sunnite muslim", "sufism", "sufi", "sufi (person)"], "dis": [[5563, 56313, 2583], [52175, 6844, 1373], [15626, 9486, 22331]]}, {"prop": "religion", "subj": "Queen Victoria", "obj": "Anglicanism", "pop": 554097, "ids": [16039, 16689, 1410, 1348, 2185, 1349, 1379, 52175, 2583, 72], "a": 7, "n": 2, "alts": ["anglicanism", "anglicanism, anglican church"], "dis": [[56510, 2583], [33798, 2583], [54257, 47273]]}, {"prop": "religion", "subj": "Bill Clinton", "obj": "Baptists", "pop": 495298, "ids": [10518, 20355, 1410, 1348, 2185, 1349, 1379, 1469, 3520, 2308, 72], "a": 7, "n": 3, "alts": ["baptists", "baptist church", "baptist"], "dis": [[9893, 42098, 8458], [6717, 9878, 33893], [16225, 42098, 8458]]}, {"prop": "religion", "subj": "Marilyn Monroe", "obj": "Judaism", "pop": 487061, "ids": [3318, 2168, 59322, 53035, 1410, 1348, 2185, 1349, 1379, 13466, 59324, 2583, 72], "a": 9, "n": 3, "alts": ["judaism", "jewish", "jewish religion"], "dis": [[51633, 1516, 2583], [8458, 1379, 15280], [2419, 1627, 15213]]}, {"prop": "religion", "subj": "George VI", "obj": "Anglicanism", "pop": 420204, "ids": [9595, 31650, 1410, 1348, 2185, 1349, 1379, 52175, 2583, 72], "a": 7, "n": 2, "alts": ["anglicanism", "anglicanism, anglican church"], "dis": [[56510, 2583], [12978, 8458], [54257, 47273]]}, {"prop": "religion", "subj": "Muhammad", "obj": "Hanif", "pop": 415010, "ids": [42820, 1410, 1348, 2185, 1349, 1379, 21139, 1436, 72], "a": 6, "n": 2, "alts": ["hanif", "islam", "islamic religion", "mohammedanism", "muslim religion", "al-\u2019isl\u0101m", "religion of the muslims"], "dis": [[39851, 2583], [12978, 8458], [19474, 2326]]}, {"prop": "religion", "subj": "Abraham Lincoln", "obj": "Baptists", "pop": 394926, "ids": [46753, 23423, 1410, 1348, 2185, 1349, 1379, 1469, 3520, 2308, 72], "a": 7, "n": 3, "alts": ["baptists", "baptist church", "baptist"], "dis": [[8458, 1379, 18555], [1609, 1485, 2583], [1510, 1351, 2583]]}, {"prop": "religion", "subj": "Spain", "obj": "non-denominational", "pop": 377325, "ids": [13834, 1410, 1348, 2185, 1349, 1379, 2635, 63, 6110, 6813, 1988, 72], "a": 6, "n": 5, "alts": ["non-denominational", "nondenominational"], "dis": [[45947, 1770, 1690, 1438, 1375], [23049, 12978, 53653, 3850, 8458], [2572, 21749, 1606, 7906, 2583]]}, {"prop": "religion", "subj": "Vladimir Putin", "obj": "Russian Orthodox Church", "pop": 358542, "ids": [44000, 49739, 1410, 1348, 2185, 1349, 1379, 9893, 42098, 8458, 72], "a": 7, "n": 3, "alts": ["russian orthodox church"], "dis": [[2023, 26673, 2583], [6717, 9878, 33893], [2419, 1627, 15213]]}, {"prop": "religion", "subj": "Jesus", "obj": "Judaism", "pop": 353596, "ids": [13855, 1410, 1348, 2185, 1349, 1379, 13466, 59324, 2583, 72], "a": 6, "n": 3, "alts": ["judaism", "jewish", "jewish religion"], "dis": [[6717, 46810, 8458], [52175, 6844, 1373], [12135, 50325, 8458]]}, {"prop": "religion", "subj": "George Washington", "obj": "Episcopal Church", "pop": 350454, "ids": [9595, 8225, 1410, 1348, 2185, 1349, 1379, 12135, 50325, 8458, 72], "a": 7, "n": 3, "alts": ["episcopal church", "gu fakes", "guerra universitaria", "episcopalianism", "the episcopal church in the united states of america"], "dis": [[8458, 1379, 15280], [1413, 3232, 2583], [32106, 42098, 8458]]}, {"prop": "religion", "subj": "Francis", "obj": "Catholic Church", "pop": 286991, "ids": [10709, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 6, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[19474, 2326], [21139, 1436], [52175, 2583]]}, {"prop": "religion", "subj": "Gautama Buddha", "obj": "Buddhism", "pop": 270891, "ids": [1527, 2871, 4251, 53188, 5031, 1410, 1348, 2185, 1349, 1379, 33798, 2583, 72], "a": 10, "n": 2, "alts": ["buddhism", "buddha dharma"], "dis": [[2419, 25199], [1564, 2583], [12978, 8458]]}, {"prop": "religion", "subj": "Bhimrao Ramji Ambedkar", "obj": "Buddhism", "pop": 266712, "ids": [24161, 1400, 1765, 59326, 12339, 11140, 19458, 1367, 27101, 1410, 1348, 2185, 1349, 1379, 33798, 2583, 72], "a": 14, "n": 2, "alts": ["buddhism", "buddha dharma"], "dis": [[56510, 2583], [2419, 25199], [19474, 2326]]}, {"prop": "religion", "subj": "Thomas Jefferson", "obj": "deism", "pop": 254256, "ids": [10804, 38061, 1410, 1348, 2185, 1349, 1379, 1564, 2583, 72], "a": 7, "n": 2, "alts": ["deism", "deism"], "dis": [[2419, 25199], [19474, 2326], [12978, 8458]]}, {"prop": "religion", "subj": "Byzantine Empire", "obj": "Eastern Orthodox Church", "pop": 218905, "ids": [58951, 39884, 18747, 1410, 1348, 2185, 1349, 1379, 16225, 42098, 8458, 72], "a": 8, "n": 3, "alts": ["eastern orthodox church", "eastern church", "orthodox church", "orthodox catholic church", "greek church", "eastern orthodox communion", "church of the seven councils"], "dis": [[9893, 42098, 8458], [1510, 1351, 2583], [52175, 6844, 1373]]}, {"prop": "religion", "subj": "Kumail Nanjiani", "obj": "atheism", "pop": 209468, "ids": [1681, 1451, 1630, 24938, 59392, 49482, 1410, 1348, 2185, 1349, 1379, 1510, 1351, 2583, 72], "a": 11, "n": 3, "alts": ["atheism", "unbelief"], "dis": [[15626, 9486, 22331], [5563, 56313, 2583], [12135, 50325, 8458]]}, {"prop": "religion", "subj": "Catholic Church", "obj": "Catholicism", "pop": 195995, "ids": [12978, 8458, 1410, 1348, 2185, 1349, 1379, 12978, 2583, 72], "a": 7, "n": 2, "alts": ["catholicism", "catholic faith", "catholic religion", "catholic"], "dis": [[21139, 1436], [1564, 2583], [54257, 47273]]}, {"prop": "religion", "subj": "Anne of Great Britain", "obj": "Anglicanism", "pop": 156171, "ids": [24423, 1379, 7566, 15273, 1410, 1348, 2185, 1349, 1379, 52175, 2583, 72], "a": 9, "n": 2, "alts": ["anglicanism", "anglicanism, anglican church"], "dis": [[1564, 2583], [12978, 2583], [19474, 2326]]}, {"prop": "religion", "subj": "James VI and I", "obj": "Anglicanism", "pop": 151391, "ids": [8266, 31650, 1384, 1417, 1410, 1348, 2185, 1349, 1379, 52175, 2583, 72], "a": 9, "n": 2, "alts": ["anglicanism", "anglicanism, anglican church", "church of scotland", "cos", "the kirk"], "dis": [[12978, 8458], [33798, 2583], [39851, 2583]]}, {"prop": "religion", "subj": "Mark Twain", "obj": "atheism", "pop": 146014, "ids": [5378, 12239, 1485, 1410, 1348, 2185, 1349, 1379, 1510, 1351, 2583, 72], "a": 8, "n": 3, "alts": ["atheism", "unbelief"], "dis": [[20452, 1755, 2583], [5563, 56313, 2583], [44252, 42098, 8458]]}, {"prop": "religion", "subj": "Paul", "obj": "Judaism", "pop": 135900, "ids": [7821, 1410, 1348, 2185, 1349, 1379, 13466, 59324, 2583, 72], "a": 6, "n": 3, "alts": ["christianity", "christian faith", "judaism", "jewish", "jewish religion"], "dis": [[4544, 13417, 2583], [1417, 11103, 59325], [9826, 7376, 15213]]}, {"prop": "religion", "subj": "Aung San Suu Kyi", "obj": "Buddhism", "pop": 127882, "ids": [1420, 3300, 6312, 24051, 59334, 23069, 59325, 1410, 1348, 2185, 1349, 1379, 33798, 2583, 72], "a": 12, "n": 2, "alts": ["buddhism", "buddha dharma"], "dis": [[39851, 2583], [2419, 25199], [1564, 2583]]}, {"prop": "religion", "subj": "Empire of Japan", "obj": "Shinto", "pop": 126957, "ids": [18747, 1379, 5701, 1410, 1348, 2185, 1349, 1379, 2419, 25199, 72], "a": 8, "n": 2, "alts": ["shinto", "shintoism", "shint\u014d", "shindo"], "dis": [[12978, 8458], [1564, 2583], [21139, 1436]]}, {"prop": "religion", "subj": "Johann Sebastian Bach", "obj": "Lutheranism", "pop": 120538, "ids": [59147, 53213, 41832, 1410, 1348, 2185, 1349, 1379, 56510, 2583, 72], "a": 8, "n": 2, "alts": ["lutheranism", "lutheran church", "lutherans"], "dis": [[21139, 1436], [12978, 8458], [33798, 2583]]}, {"prop": "religion", "subj": "David", "obj": "Judaism", "pop": 111751, "ids": [7524, 1410, 1348, 2185, 1349, 1379, 13466, 59324, 2583, 72], "a": 6, "n": 3, "alts": ["judaism", "jewish", "jewish religion"], "dis": [[1609, 1485, 2583], [3532, 46810, 8458], [1510, 1351, 2583]]}, {"prop": "religion", "subj": "Roman Republic", "obj": "ancient Roman religion", "pop": 107065, "ids": [9486, 7121, 1410, 1348, 2185, 1349, 1379, 15626, 9486, 22331, 72], "a": 7, "n": 3, "alts": ["ancient roman religion", "roman religion", "religio romana", "roman polytheism", "ancient roman polytheism"], "dis": [[32106, 42098, 8458], [1510, 1351, 2583], [6948, 8822, 22331]]}, {"prop": "religion", "subj": "Rama", "obj": "Hinduism", "pop": 93777, "ids": [1490, 4251, 1410, 1348, 2185, 1349, 1379, 39851, 2583, 72], "a": 7, "n": 2, "alts": ["hinduism", "hindu dharma", "sanatan dharma", "hindu religion", "hindoo dharam", "\ud83d\uded5", "\u0950", "jainism", "nirgrantha", "jain dharma", "jainatva", "jaina", "jain mata", "jaini", "jainy", "jainas"], "dis": [[12978, 2583], [54257, 47273], [33798, 2583]]}, {"prop": "religion", "subj": "James Madison", "obj": "Episcopal Church", "pop": 93306, "ids": [8266, 32892, 1410, 1348, 2185, 1349, 1379, 12135, 50325, 8458, 72], "a": 7, "n": 3, "alts": ["episcopal church", "gu fakes", "guerra universitaria", "episcopalianism", "the episcopal church in the united states of america"], "dis": [[9826, 7376, 15213], [13466, 59324, 2583], [52175, 6844, 1373]]}, {"prop": "religion", "subj": "Carl Linnaeus", "obj": "Church of Sweden", "pop": 87289, "ids": [14881, 8351, 3003, 59321, 1406, 1410, 1348, 2185, 1349, 1379, 8458, 1379, 18555, 72], "a": 10, "n": 3, "alts": ["church of sweden", "svenska kyrkan"], "dis": [[44252, 42098, 8458], [1616, 58100, 2583], [51633, 1516, 2583]]}, {"prop": "religion", "subj": "Bertrand Russell", "obj": "agnosticism", "pop": 85602, "ids": [8599, 1586, 1483, 28123, 1410, 1348, 2185, 1349, 1379, 1845, 59322, 9059, 2583, 72], "a": 9, "n": 4, "alts": ["agnosticism"], "dis": [[3532, 8458, 1379, 4516], [50351, 53653, 3850, 8458], [24655, 1690, 2157, 4050]]}, {"prop": "religion", "subj": "L. Ron Hubbard", "obj": "Scientology", "pop": 84680, "ids": [1494, 72, 23604, 24961, 56188, 1410, 1348, 2185, 1349, 1379, 19474, 2326, 72], "a": 10, "n": 2, "alts": ["scientology"], "dis": [[56510, 2583], [1564, 2583], [12978, 2583]]}, {"prop": "religion", "subj": "William III of England", "obj": "Protestantism", "pop": 79983, "ids": [9753, 11223, 1379, 7610, 1410, 1348, 2185, 1349, 1379, 51633, 1516, 2583, 72], "a": 9, "n": 3, "alts": ["protestantism", "protestant religion", "protestant church", "christian protestantism", "protestant churches", "protestant christianity", "protestant", "protestants"], "dis": [[8458, 1379, 15280], [1417, 11103, 59325], [6717, 9878, 33893]]}, {"prop": "religion", "subj": "Jacob", "obj": "Judaism", "pop": 76739, "ids": [22200, 1410, 1348, 2185, 1349, 1379, 13466, 59324, 2583, 72], "a": 6, "n": 3, "alts": ["judaism", "jewish", "jewish religion"], "dis": [[2419, 1627, 15213], [3532, 46810, 8458], [12135, 50325, 8458]]}, {"prop": "religion", "subj": "T.\u00a0S. Eliot", "obj": "Unitarianism", "pop": 76305, "ids": [1408, 72, 1283, 1249, 59352, 72, 34974, 1411, 1410, 1348, 2185, 1349, 1379, 2023, 26673, 2583, 72], "a": 13, "n": 3, "alts": ["unitarianism", "rational dissent", "anglo-catholicism", "anglican catholicism", "catholic anglicanism"], "dis": [[52175, 6844, 1373], [13466, 59324, 2583], [38434, 1988, 8590]]}, {"prop": "religion", "subj": "Kingdom of England", "obj": "Catholic Church", "pop": 69770, "ids": [11690, 1379, 7610, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 8, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[56510, 2583], [19474, 2326], [1564, 2583]]}, {"prop": "religion", "subj": "Tang Empire", "obj": "Buddhism", "pop": 67196, "ids": [37202, 18747, 1410, 1348, 2185, 1349, 1379, 33798, 2583, 72], "a": 7, "n": 2, "alts": ["buddhism", "buddha dharma"], "dis": [[21139, 1436], [12978, 2583], [56510, 2583]]}, {"prop": "religion", "subj": "Austrian Empire", "obj": "Catholic Church", "pop": 56223, "ids": [33908, 18747, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 7, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[21139, 1436], [52175, 2583], [54257, 47273]]}, {"prop": "religion", "subj": "John Paul I", "obj": "Catholic Church", "pop": 56118, "ids": [4801, 7821, 1417, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 8, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[56510, 2583], [33798, 2583], [21139, 1436]]}, {"prop": "religion", "subj": "Tokugawa Shogunate", "obj": "Buddhism", "pop": 53238, "ids": [17596, 1719, 13109, 2419, 1514, 27007, 1410, 1348, 2185, 1349, 1379, 33798, 2583, 72], "a": 11, "n": 2, "alts": ["buddhism", "buddha dharma"], "dis": [[2419, 25199], [52175, 2583], [12978, 2583]]}, {"prop": "religion", "subj": "Georgetown University", "obj": "Catholic Church", "pop": 52990, "ids": [4396, 58134, 3063, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 8, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[2419, 25199], [54257, 47273], [21139, 1436]]}, {"prop": "religion", "subj": "John Calvin", "obj": "Protestantism", "pop": 51490, "ids": [4801, 4544, 13417, 1410, 1348, 2185, 1349, 1379, 51633, 1516, 2583, 72], "a": 8, "n": 3, "alts": ["protestantism", "protestant religion", "protestant church", "christian protestantism", "protestant churches", "protestant christianity", "protestant", "protestants"], "dis": [[8458, 1379, 18555], [15626, 9486, 22331], [5563, 56313, 2583]]}, {"prop": "religion", "subj": "Blaise Pascal", "obj": "jansenism", "pop": 51119, "ids": [3474, 59324, 1998, 20222, 5198, 1410, 1348, 2185, 1349, 1379, 1616, 58100, 2583, 72], "a": 10, "n": 3, "alts": ["jansenism", "jansenisme", "jansenists"], "dis": [[12135, 50325, 8458], [6948, 8822, 22331], [6717, 46810, 8458]]}, {"prop": "religion", "subj": "Mahavira", "obj": "Jainism", "pop": 50616, "ids": [14257, 1591, 11010, 1410, 1348, 2185, 1349, 1379, 1609, 1485, 2583, 72], "a": 8, "n": 3, "alts": ["jainism", "nirgrantha", "jain dharma", "jainatva", "jaina", "jain mata", "jaini", "jainy", "jainas"], "dis": [[15626, 9486, 22331], [1510, 1351, 2583], [12135, 50325, 8458]]}, {"prop": "religion", "subj": "Padre Pio", "obj": "Catholic Church", "pop": 49453, "ids": [26634, 1352, 1452, 1839, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 9, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[39851, 2583], [12978, 2583], [2419, 25199]]}, {"prop": "religion", "subj": "Claus Schenk Graf von Stauffenberg", "obj": "Catholic Church", "pop": 47789, "ids": [2534, 14476, 2732, 1668, 59346, 18692, 59337, 15585, 1716, 2874, 1595, 27811, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 17, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[56510, 2583], [2419, 25199], [12978, 2583]]}, {"prop": "religion", "subj": "Liberty University", "obj": "Evangelicalism", "pop": 46990, "ids": [37951, 3063, 1410, 1348, 2185, 1349, 1379, 8860, 17300, 1583, 2583, 72], "a": 7, "n": 4, "alts": ["evangelicalism", "evangelical christianity", "evangelical protestantism"], "dis": [[50351, 53653, 3850, 8458], [24655, 1690, 2157, 4050], [4516, 15540, 1738, 5948]]}, {"prop": "religion", "subj": "Armenians", "obj": "Armenian Apostolic Church", "pop": 45578, "ids": [30299, 5948, 1410, 1348, 2185, 1349, 1379, 50351, 53653, 3850, 8458, 72], "a": 7, "n": 4, "alts": ["armenian catholic church", "armenian church", "armenian apostolic church"], "dis": [[46094, 1363, 33798, 2583], [1845, 59322, 9059, 2583], [8860, 17300, 1583, 2583]]}, {"prop": "religion", "subj": "Saint Patrick", "obj": "Celtic Christianity", "pop": 43319, "ids": [14227, 21942, 1410, 1348, 2185, 1349, 1379, 54257, 47273, 72], "a": 7, "n": 2, "alts": ["celtic christianity", "celtic church", "insular christianity", "christianity", "christian faith"], "dis": [[19474, 2326], [56510, 2583], [2419, 25199]]}, {"prop": "religion", "subj": "Samuel Johnson", "obj": "Anglicanism", "pop": 36459, "ids": [30087, 14318, 1410, 1348, 2185, 1349, 1379, 52175, 2583, 72], "a": 7, "n": 2, "alts": ["anglicanism", "anglicanism, anglican church"], "dis": [[21139, 1436], [54257, 47273], [2419, 25199]]}, {"prop": "religion", "subj": "Maximilian I of Mexico", "obj": "Catholic Church", "pop": 34853, "ids": [10277, 1400, 55380, 1417, 1379, 11198, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 11, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[1564, 2583], [56510, 2583], [54257, 47273]]}, {"prop": "religion", "subj": "David Livingstone", "obj": "congregational church", "pop": 33550, "ids": [7524, 19596, 13332, 1410, 1348, 2185, 1349, 1379, 38434, 1988, 8590, 72], "a": 8, "n": 3, "alts": ["congregational church", "evangelical congregational church", "congregationalism"], "dis": [[20452, 1755, 2583], [1507, 1378, 2728], [51633, 1516, 2583]]}, {"prop": "religion", "subj": "Kingdom of Jerusalem", "obj": "Catholic Church", "pop": 31938, "ids": [11690, 1379, 34838, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 8, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[2419, 25199], [39851, 2583], [54257, 47273]]}, {"prop": "religion", "subj": "American University", "obj": "United Methodist Church", "pop": 31620, "ids": [3295, 3063, 1410, 1348, 2185, 1349, 1379, 3532, 46810, 8458, 72], "a": 7, "n": 3, "alts": ["united methodist church", "umc"], "dis": [[32106, 42098, 8458], [8458, 1379, 15280], [51633, 1516, 2583]]}, {"prop": "religion", "subj": "Bourbon Restoration", "obj": "Catholic Church", "pop": 31469, "ids": [28919, 5783, 12991, 8399, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 9, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[1564, 2583], [33798, 2583], [12978, 2583]]}, {"prop": "religion", "subj": "Francisco Pizarro", "obj": "Catholic Church", "pop": 30204, "ids": [14241, 1452, 28518, 1381, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 9, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[2419, 25199], [39851, 2583], [19474, 2326]]}, {"prop": "religion", "subj": "St. James the Elder, Apostle", "obj": "Catholic Church", "pop": 28599, "ids": [1716, 72, 8266, 1358, 34875, 1349, 59342, 53653, 1378, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 14, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[1564, 2583], [19474, 2326], [39851, 2583]]}, {"prop": "religion", "subj": "Evo Morales", "obj": "Catholic Church", "pop": 28015, "ids": [8860, 59326, 7213, 3337, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 9, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[1564, 2583], [56510, 2583], [39851, 2583]]}, {"prop": "religion", "subj": "Andrew the Apostle", "obj": "Catholic Church", "pop": 26745, "ids": [15054, 1358, 53653, 1378, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 9, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[12978, 2583], [54257, 47273], [21139, 1436]]}, {"prop": "religion", "subj": "Boston College", "obj": "Catholic Church", "pop": 25093, "ids": [12923, 5846, 1410, 1348, 2185, 1349, 1379, 12978, 8458, 72], "a": 7, "n": 2, "alts": ["catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[39851, 2583], [56510, 2583], [1564, 2583]]}, {"prop": "religion", "subj": "Assumption of Mary", "obj": "Anglican Communion", "pop": 23361, "ids": [3974, 45029, 1379, 8959, 1410, 1348, 2185, 1349, 1379, 52175, 6844, 1373, 72], "a": 9, "n": 3, "alts": ["anglican communion", "eastern orthodox church", "eastern church", "orthodox church", "orthodox catholic church", "greek church", "eastern orthodox communion", "church of the seven councils", "oriental orthodox churches", "oriental orthodox churches", "oriental orthodoxy", "catholic church", "roman catholic church", "church", "roman apostolic catholic church"], "dis": [[20452, 1755, 2583], [38434, 1988, 8590], [6717, 46810, 8458]]}, {"prop": "religion", "subj": "Sabine Baring-Gould", "obj": "Anglicanism", "pop": 23123, "ids": [17543, 1534, 1469, 3817, 63, 59394, 1619, 1410, 1348, 2185, 1349, 1379, 52175, 2583, 72], "a": 12, "n": 2, "alts": ["anglicanism", "anglicanism, anglican church"], "dis": [[54257, 47273], [12978, 2583], [56510, 2583]]}, {"prop": "religion", "subj": "synagogue", "obj": "Judaism", "pop": 22138, "ids": [7526, 1442, 28680, 1410, 1348, 2185, 1349, 1379, 13466, 59324, 2583, 72], "a": 8, "n": 3, "alts": ["judaism", "jewish", "jewish religion"], "dis": [[12135, 50325, 8458], [1609, 1485, 2583], [9826, 7376, 15213]]}, {"prop": "religion", "subj": "Loyalist", "obj": "Anglicanism", "pop": 21579, "ids": [1494, 7729, 1466, 1410, 1348, 2185, 1349, 1379, 52175, 2583, 72], "a": 8, "n": 2, "alts": ["anglicanism", "anglicanism, anglican church"], "dis": [[12978, 2583], [2419, 25199], [39851, 2583]]}, {"prop": "religion", "subj": "magi", "obj": "Zoroastrianism", "pop": 21156, "ids": [4633, 59325, 1410, 1348, 2185, 1349, 1379, 2572, 21749, 1606, 7906, 2583, 72], "a": 7, "n": 5, "alts": ["zoroastrianism", "zarathustraism", "mazdaism", "mazdanism", "magianism", "the good religion", "mazdayasna"], "dis": [[3327, 12637, 9033, 1379, 24391], [45947, 1770, 1690, 1438, 1375], [2635, 63, 6110, 6813, 1988]]}, {"prop": "religion", "subj": "First Bulgarian Empire", "obj": "Tengrism", "pop": 20493, "ids": [6212, 46483, 18747, 1410, 1348, 2185, 1349, 1379, 20452, 1755, 2583, 72], "a": 8, "n": 3, "alts": ["tengrism", "tengriism", "tengrianism", "tenggerism"], "dis": [[1609, 1485, 2583], [5563, 56313, 2583], [13466, 59324, 2583]]}, {"prop": "religion", "subj": "Silla", "obj": "Buddhism", "pop": 20252, "ids": [1413, 8034, 1410, 1348, 2185, 1349, 1379, 33798, 2583, 72], "a": 7, "n": 2, "alts": ["buddhism", "buddha dharma"], "dis": [[1564, 2583], [56510, 2583], [2419, 25199]]}, {"prop": "religion", "subj": "University of Evansville", "obj": "United Methodist Church", "pop": 19871, "ids": [3063, 1379, 30086, 7009, 1410, 1348, 2185, 1349, 1379, 3532, 46810, 8458, 72], "a": 9, "n": 3, "alts": ["united methodist church", "umc"], "dis": [[51633, 1516, 2583], [1408, 7022, 2583], [13466, 59324, 2583]]}, {"prop": "religion", "subj": "Hafez", "obj": "Sufism", "pop": 16925, "ids": [1491, 8557, 59398, 1410, 1348, 2185, 1349, 1379, 1413, 3232, 2583, 72], "a": 8, "n": 3, "alts": ["islam", "islamic religion", "mohammedanism", "muslim religion", "al-\u2019isl\u0101m", "religion of the muslims", "shia islam", "sh\u012ba islam", "shiism", "shiah islam", "shi\u02bfa islam", "shi'a islam", "sufism", "sufi", "sufi (person)"], "dis": [[3532, 46810, 8458], [1510, 1351, 2583], [24141, 1379, 15213]]}, {"prop": "screenwriter", "subj": "Halloween", "obj": "Debra Hill", "pop": 933638, "ids": [1507, 4925, 2532, 1379, 28920, 1523, 5336, 1520, 2748, 6457, 10082, 72], "a": 8, "n": 3, "alts": ["debra hill", "john carpenter", "john howard carpenter"], "dis": [[7524, 48471, 8573], [49412, 55899, 32350], [23978, 14642, 54104]]}, {"prop": "screenwriter", "subj": "The Shining", "obj": "Stanley Kubrick", "pop": 326522, "ids": [1507, 4925, 2532, 1379, 1507, 2419, 5258, 1523, 5336, 1520, 31808, 1681, 1484, 13654, 72], "a": 10, "n": 4, "alts": ["stanley kubrick", "stephen king", "stephen edwin king", "richard bachman", "john swithen", "diane johnson"], "dis": [[4801, 1609, 1464, 2162], [4016, 59323, 21190, 22457], [12583, 3317, 1558, 9747]]}, {"prop": "screenwriter", "subj": "The Matrix", "obj": "The Wachowskis", "pop": 291114, "ids": [1507, 4925, 2532, 1379, 1507, 23560, 1523, 5336, 1520, 1507, 1495, 1684, 2681, 59346, 1360, 72], "a": 9, "n": 6, "alts": ["the wachowskis", "wachowski brothers", "the wachowski brothers", "the wachowski starship", "wachowski sisters"], "dis": [[1413, 2021, 59331, 16358, 1677, 3003], [8454, 59324, 1681, 2874, 1595, 2450], [1527, 41109, 1549, 59361, 7072, 1355]]}, {"prop": "screenwriter", "subj": "The Thing", "obj": "Bill Lancaster", "pop": 195702, "ids": [1507, 4925, 2532, 1379, 1507, 47849, 1523, 5336, 1520, 10518, 55120, 72], "a": 9, "n": 2, "alts": ["bill lancaster", "william henry lancaster"], "dis": [[25287, 44471], [7524, 30396], [7821, 29580]]}, {"prop": "screenwriter", "subj": "Jurassic Park", "obj": "David Koepp", "pop": 184720, "ids": [1507, 4925, 2532, 1379, 44966, 54492, 5433, 1523, 5336, 1520, 7524, 31858, 1636, 59336, 72], "a": 10, "n": 4, "alts": ["michael crichton", "john michael crichton", "michael douglas", "jeffery hudson", "john lange", "david koepp"], "dis": [[14881, 1445, 1633, 59346], [8266, 1413, 1541, 1360], [9549, 2419, 2231, 2272]]}, {"prop": "screenwriter", "subj": "Pulp Fiction", "obj": "Roger Avary", "pop": 164826, "ids": [1507, 4925, 2532, 1379, 1452, 16227, 48431, 1523, 5336, 1520, 31088, 9383, 1629, 72], "a": 10, "n": 3, "alts": ["quentin tarantino", "tarantino", "quentin jerome tarantino", "roger avary", "roger roberts avary"], "dis": [[19488, 11891, 1861], [12344, 3318, 3972], [37277, 4384, 58382]]}, {"prop": "screenwriter", "subj": "Neon Genesis Evangelion", "obj": "Hideaki Anno", "pop": 160245, "ids": [1507, 4925, 2532, 1379, 5732, 1350, 9840, 6984, 8860, 17300, 1373, 1523, 5336, 1520, 46164, 14770, 2426, 3768, 72], "a": 14, "n": 4, "alts": ["y\u014dji enokido", "yoji enokido", "youji enokido", "hideaki anno", "anno hideaki"], "dis": [[21942, 9881, 1576, 1626], [21152, 2611, 3227, 3061], [27128, 1490, 3198, 43063]]}, {"prop": "screenwriter", "subj": "The Terminal", "obj": "Jeff Nathanson", "pop": 156195, "ids": [1507, 4925, 2532, 1379, 1507, 40975, 1523, 5336, 1520, 13457, 1515, 1598, 46192, 72], "a": 9, "n": 4, "alts": ["sacha gervasi", "alexander sacha simon gervasi", "jeff nathanson"], "dis": [[26403, 2232, 2278, 58265], [2534, 5940, 49047, 1364], [2426, 14965, 12180, 1350]]}, {"prop": "screenwriter", "subj": "Lost", "obj": "Damon Lindelof", "pop": 154447, "ids": [1507, 4925, 2532, 1379, 35739, 1523, 5336, 1520, 17101, 1350, 18857, 1391, 2382, 72], "a": 8, "n": 5, "alts": ["elizabeth sarnoff", "elizabeth \"liz\" sarnoff", "liz sarnoff", "adam horowitz", "damon lindelof", "damon laurence lindelof", "carlton cuse", "arthur carlton cuse", "edward kitsis", "edward \"eddy\" lawrence kitsis", "edward lawrence kitsis"], "dis": [[5732, 1349, 2549, 1690, 8390], [10073, 29591, 1491, 25775, 15248], [9137, 2162, 20746, 5031, 21800]]}, {"prop": "screenwriter", "subj": "Back to the Future", "obj": "Robert Zemeckis", "pop": 150710, "ids": [1507, 4925, 2532, 1379, 10203, 1385, 1358, 19186, 1523, 5336, 1520, 9812, 2572, 1454, 9725, 1360, 72], "a": 11, "n": 5, "alts": ["robert zemeckis", "robert l. zemeckis", "robert lee zemeckis", "bob gale", "michael robert gale"], "dis": [[28082, 49624, 8992, 59334, 3243], [9224, 59326, 1452, 14710, 59326], [14972, 1408, 7362, 5988, 1484]]}, {"prop": "screenwriter", "subj": "L\u00e9on: The Professional", "obj": "Luc Besson", "pop": 146310, "ids": [1507, 4925, 2532, 1379, 1494, 60025, 1350, 59358, 1507, 19567, 1523, 5336, 1520, 14811, 1469, 1481, 1350, 72], "a": 13, "n": 4, "alts": ["luc besson", "luc paul maurice besson"], "dis": [[21927, 1360, 15821, 1371], [28204, 2870, 20090, 7662], [18779, 6931, 18322, 1442]]}, {"prop": "screenwriter", "subj": "The Dark Knight Rises", "obj": "Christopher Nolan", "pop": 142251, "ids": [1507, 4925, 2532, 1379, 1507, 19988, 29813, 1490, 4844, 1523, 5336, 1520, 27971, 1515, 44003, 72], "a": 12, "n": 3, "alts": ["christopher nolan", "christopher jonathan james nolan", "nolan", "christopher edward nolan", "jonathan nolan"], "dis": [[24052, 2518, 1620], [4801, 28488, 1888], [9549, 8915, 1355]]}, {"prop": "screenwriter", "subj": "Seven", "obj": "Andrew Kevin Walker", "pop": 136347, "ids": [1507, 4925, 2532, 1379, 28625, 1523, 5336, 1520, 15054, 20178, 25321, 72], "a": 8, "n": 3, "alts": ["andrew kevin walker"], "dis": [[7524, 45253, 1541], [32932, 38688, 49492], [15504, 22774, 59329]]}, {"prop": "screenwriter", "subj": "The Ring", "obj": "K\u014dji Suzuki", "pop": 127283, "ids": [1507, 4925, 2532, 1379, 1507, 26578, 1523, 5336, 1520, 1681, 61843, 11140, 33205, 24750, 72], "a": 9, "n": 5, "alts": ["k\u014dji suzuki", "koji suzuki", "suzuki k\u014dji", "suzuki koji", "kouji suzuki"], "dis": [[33863, 6905, 13772, 1430, 1961], [39307, 1452, 1603, 8053, 59325], [13765, 2078, 2846, 59398, 7380]]}, {"prop": "screenwriter", "subj": "Toy Story", "obj": "Andrew Stanton", "pop": 120670, "ids": [1507, 4925, 2532, 1379, 23706, 19650, 1523, 5336, 1520, 15054, 1716, 40097, 72], "a": 9, "n": 3, "alts": ["john lasseter", "john alan lasseter", "john a. lasseter", "joss whedon", "joseph hill whedon", "andrew stanton", "andrew christopher stanton jr.", "andrew ayers stanton", "pete docter", "peter docter", "peter hans docter", "alec sokolow", "alec william sokolow", "joe ranft", "joseph henry ranft"], "dis": [[11033, 38901, 1370], [7524, 53641, 7662], [4801, 21152, 10253]]}, {"prop": "screenwriter", "subj": "Troy", "obj": "David Benioff", "pop": 118889, "ids": [1507, 4925, 2532, 1379, 50634, 1523, 5336, 1520, 7524, 7370, 1839, 1595, 72], "a": 8, "n": 4, "alts": ["david benioff", "david friedman"], "dis": [[4801, 51717, 1598, 1541], [5541, 8751, 27588, 4039], [1490, 22094, 25320, 37632]]}, {"prop": "screenwriter", "subj": "Thor", "obj": "Ashley Miller", "pop": 112482, "ids": [1507, 4925, 2532, 1379, 35689, 1523, 5336, 1520, 45248, 19583, 72], "a": 8, "n": 2, "alts": ["mark protosevich", "mark david protosevich", "ashley miller", "ashley edward miller", "zack stentz"], "dis": [[32392, 34970], [52171, 43909], [9137, 21861]]}, {"prop": "screenwriter", "subj": "Little Shop of Horrors", "obj": "Howard Ashman", "pop": 109716, "ids": [1507, 4925, 2532, 1379, 14404, 16008, 1379, 13993, 26054, 1523, 5336, 1520, 24987, 16013, 2450, 72], "a": 12, "n": 3, "alts": ["howard ashman", "howard elliott ashman"], "dis": [[34881, 18692, 1595], [4801, 12239, 1466], [24634, 1469, 9161]]}, {"prop": "screenwriter", "subj": "Cars", "obj": "John Lasseter", "pop": 107210, "ids": [1507, 4925, 2532, 1379, 44243, 1523, 5336, 1520, 4801, 1494, 1537, 4493, 72], "a": 8, "n": 4, "alts": ["john lasseter", "john alan lasseter", "john a. lasseter", "joe ranft", "joseph henry ranft"], "dis": [[9610, 5281, 12955, 44104], [29151, 1515, 2984, 3420], [10967, 9554, 1370, 12694]]}, {"prop": "screenwriter", "subj": "300", "obj": "Zack Snyder", "pop": 102685, "ids": [1507, 4925, 2532, 1379, 59320, 59362, 59344, 59344, 1523, 5336, 1520, 2572, 1558, 1413, 3362, 2134, 72], "a": 11, "n": 5, "alts": ["zack snyder", "zachary edward snyder", "zachary edward \"zack\" snyder"], "dis": [[7063, 1491, 45884, 33715, 59323], [57524, 1382, 48401, 1729, 2162], [19711, 24161, 6511, 1494, 2190]]}, {"prop": "screenwriter", "subj": "Final Destination", "obj": "Glen Morgan", "pop": 102010, "ids": [1507, 4925, 2532, 1379, 17989, 26934, 3355, 1523, 5336, 1520, 33133, 24546, 72], "a": 10, "n": 2, "alts": ["jeffrey reddick", "glen morgan"], "dis": [[40813, 27254], [32932, 37050], [24987, 26159]]}, {"prop": "screenwriter", "subj": "The Holiday", "obj": "Nancy Meyers", "pop": 101218, "ids": [1507, 4925, 2532, 1379, 1507, 34015, 1523, 5336, 1520, 37160, 1445, 2825, 1465, 72], "a": 9, "n": 4, "alts": ["nancy meyers", "nancy jane meyers"], "dis": [[48762, 1494, 45291, 2305], [37787, 1419, 72, 27816], [33306, 1402, 33306, 22632]]}, {"prop": "screenwriter", "subj": "The Fog", "obj": "John Carpenter", "pop": 100414, "ids": [1507, 4925, 2532, 1379, 1507, 1493, 1514, 1523, 5336, 1520, 4801, 51589, 9744, 72], "a": 10, "n": 3, "alts": ["john carpenter", "john howard carpenter"], "dis": [[20362, 55330, 19739], [23978, 37301, 1481], [13187, 38596, 21800]]}, {"prop": "screenwriter", "subj": "The Prestige", "obj": "Christopher Nolan", "pop": 99894, "ids": [1507, 4925, 2532, 1379, 1507, 36568, 17899, 1523, 5336, 1520, 27971, 1515, 44003, 72], "a": 10, "n": 3, "alts": ["christopher nolan", "christopher jonathan james nolan", "nolan", "christopher edward nolan", "christopher priest", "christopher mckenzie priest", "john luther novak", "colin wedgelock"], "dis": [[9595, 43359, 3742], [50878, 1793, 48178], [22313, 30310, 2232]]}, {"prop": "screenwriter", "subj": "The Omen", "obj": "David Seltzer", "pop": 99370, "ids": [1507, 4925, 2532, 1379, 1507, 1549, 8858, 1523, 5336, 1520, 7524, 1413, 6428, 6926, 72], "a": 10, "n": 4, "alts": ["david seltzer"], "dis": [[1490, 42967, 14196, 5451], [45573, 52531, 45167, 24661], [15089, 1445, 1673, 1668]]}, {"prop": "screenwriter", "subj": "E.T. the Extra-Terrestrial", "obj": "Melissa Mathison", "pop": 97322, "ids": [1507, 4925, 2532, 1379, 1529, 72, 59351, 72, 1358, 33897, 63, 36864, 44231, 1523, 5336, 1520, 54080, 6424, 4125, 72], "a": 16, "n": 3, "alts": ["melissa mathison", "melissa marie mathison"], "dis": [[51390, 7196, 42369], [14881, 8690, 2450], [12980, 3684, 7884]]}, {"prop": "screenwriter", "subj": "They Live", "obj": "John Carpenter", "pop": 90266, "ids": [1507, 4925, 2532, 1379, 3330, 13588, 1523, 5336, 1520, 4801, 51589, 9744, 72], "a": 9, "n": 3, "alts": ["john carpenter", "john howard carpenter"], "dis": [[52361, 57748, 8057], [23978, 14642, 54104], [1445, 1486, 10459]]}, {"prop": "screenwriter", "subj": "The Notebook", "obj": "Jan Sardi", "pop": 90029, "ids": [1507, 4925, 2532, 1379, 1507, 10469, 3629, 1523, 5336, 1520, 3868, 1413, 36989, 72], "a": 10, "n": 3, "alts": ["jan sardi", "jan vittorio sardi"], "dis": [[9812, 15143, 59323], [27954, 24235, 2225], [5174, 9001, 34929]]}, {"prop": "screenwriter", "subj": "Divergent", "obj": "Evan Daugherty", "pop": 88076, "ids": [1507, 4925, 2532, 1379, 1479, 3160, 8303, 1523, 5336, 1520, 55751, 1479, 5590, 18358, 72], "a": 10, "n": 4, "alts": ["vanessa taylor", "evan daugherty"], "dis": [[21165, 57357, 1637, 1391], [3684, 2325, 6953, 2450], [14666, 11140, 10998, 13086]]}, {"prop": "screenwriter", "subj": "Drive", "obj": "James Sallis", "pop": 87064, "ids": [1507, 4925, 2532, 1379, 19797, 1523, 5336, 1520, 8266, 1413, 1541, 1360, 72], "a": 8, "n": 4, "alts": ["hossein amini", "james sallis"], "dis": [[19255, 39637, 1472, 1770], [54279, 1493, 1402, 1568], [33205, 1778, 22350, 2232]]}, {"prop": "screenwriter", "subj": "Snow White and the Seven Dwarfs", "obj": "Merrill De Maris", "pop": 85762, "ids": [1507, 4925, 2532, 1379, 24746, 8080, 1384, 1358, 28625, 1479, 12711, 6894, 1523, 5336, 1520, 57075, 1487, 2748, 3318, 1360, 72], "a": 15, "n": 5, "alts": ["earl hurd", "merrill de maris", "ted sears", "edward sears"], "dis": [[2944, 17497, 1360, 23321, 59339], [14972, 1408, 7362, 5988, 1484], [1413, 15101, 1394, 59322, 16104]]}, {"prop": "screenwriter", "subj": "Scooby-Doo", "obj": "James Gunn", "pop": 85109, "ids": [1507, 4925, 2532, 1379, 2732, 59326, 52342, 63, 59371, 4050, 1523, 5336, 1520, 8266, 25144, 59322, 72], "a": 13, "n": 3, "alts": ["james gunn"], "dis": [[24052, 26338, 1389], [23604, 3362, 27954], [7370, 57514, 1424]]}, {"prop": "screenwriter", "subj": "Rocky", "obj": "Sylvester Stallone", "pop": 83985, "ids": [1507, 4925, 2532, 1379, 47864, 1523, 5336, 1520, 40596, 3269, 1443, 1716, 1541, 1639, 72], "a": 8, "n": 6, "alts": ["sylvester stallone", "sly stallone", "sylvester enzio stallone", "michael sylvester gardenzio stallone"], "dis": [[1445, 4623, 1429, 1716, 1778, 16065], [40344, 1783, 39694, 1370, 1591, 48349], [9753, 1445, 1606, 4940, 1400, 1639]]}, {"prop": "screenwriter", "subj": "Into the Wild", "obj": "Sean Penn", "pop": 83157, "ids": [1507, 4925, 2532, 1379, 50324, 1358, 13337, 1523, 5336, 1520, 32392, 34970, 72], "a": 10, "n": 2, "alts": ["sean penn", "sean justin penn", "pappy pariah"], "dis": [[5859, 48856], [31088, 30731], [12344, 41106]]}, {"prop": "screenwriter", "subj": "The Fly", "obj": "David Cronenberg", "pop": 82626, "ids": [1507, 4925, 2532, 1379, 1507, 26728, 1523, 5336, 1520, 7524, 57699, 27811, 72], "a": 9, "n": 3, "alts": ["david cronenberg", "david paul cronenberg", "david p. cronenberg", "david cronenburg"], "dis": [[12980, 3684, 7884], [9855, 31304, 1350], [21872, 10102, 1373]]}, {"prop": "screenwriter", "subj": "Up", "obj": "Pete Docter", "pop": 80458, "ids": [1507, 4925, 2532, 1379, 5816, 1523, 5336, 1520, 41138, 4498, 1387, 1349, 72], "a": 8, "n": 4, "alts": ["pete docter", "peter docter", "peter hans docter", "bob peterson", "robert peterson"], "dis": [[21942, 5281, 13906, 1598], [52171, 1495, 1485, 33603], [17856, 59333, 42730, 8573]]}, {"prop": "screenwriter", "subj": "The Birds", "obj": "Daphne du Maurier", "pop": 79684, "ids": [1507, 4925, 2532, 1379, 1507, 57172, 1523, 5336, 1520, 1479, 5212, 1861, 9249, 28860, 2230, 72], "a": 9, "n": 6, "alts": ["daphne du maurier", "dame daphne du maurier", "ed mcbain", "salvatore albert lombino", "evan hunter", "hunt collins", "curt cannon", "dean hudson", "richard marsten", "ezra hannon", "john abbott", "salvatore lombino"], "dis": [[1408, 1866, 1479, 2201, 59343, 15114], [2685, 1695, 1370, 14397, 59331, 15755], [21530, 1569, 59326, 1469, 9594, 19116]]}, {"prop": "screenwriter", "subj": "The Graduate", "obj": "Buck Henry", "pop": 79381, "ids": [1507, 4925, 2532, 1379, 1507, 34712, 1523, 5336, 1520, 27158, 14943, 72], "a": 9, "n": 2, "alts": ["charles webb", "charles richard webb", "calder willingham", "buck henry", "buck henry zuckerman", "henry zuckerman"], "dis": [[27816, 56522], [11238, 24546], [5929, 52904]]}, {"prop": "screenwriter", "subj": "Legend", "obj": "Brian Helgeland", "pop": 79235, "ids": [1507, 4925, 2532, 1379, 34842, 1523, 5336, 1520, 19562, 7164, 59338, 22312, 72], "a": 8, "n": 4, "alts": ["brian helgeland", "brian thomas helgeland"], "dis": [[52361, 1609, 5615, 1578], [47706, 2572, 1787, 1416], [21927, 1360, 15821, 1371]]}, {"prop": "screenwriter", "subj": "Soylent Green", "obj": "Harry Harrison", "pop": 77182, "ids": [1507, 4925, 2532, 1379, 1413, 46274, 1386, 7868, 1523, 5336, 1520, 20987, 37198, 72], "a": 11, "n": 2, "alts": ["harry harrison", "henry maxwell dempsey", "harry max harrison", "harry maxwell harrison", "felix boyd", "hank dempsey", "wade kaempfert"], "dis": [[45248, 19583], [11238, 25690], [32392, 46899]]}, {"prop": "screenwriter", "subj": "The Mist", "obj": "Stephen King", "pop": 76264, "ids": [1507, 4925, 2532, 1379, 1507, 49567, 1523, 5336, 1520, 19378, 6132, 72], "a": 9, "n": 2, "alts": ["stephen king", "stephen edwin king", "richard bachman", "john swithen"], "dis": [[20987, 37198], [41266, 26159], [13765, 30692]]}, {"prop": "screenwriter", "subj": "Inside Out", "obj": "Ronnie del Carmen", "pop": 74976, "ids": [1507, 4925, 2532, 1379, 33044, 6249, 1523, 5336, 1520, 23604, 21646, 2664, 31382, 1356, 72], "a": 9, "n": 5, "alts": ["bill hader", "william thomas hader jr.", "amy poehler", "amy meredith poehler", "pete docter", "peter docter", "peter hans docter", "ronnie del carmen", "ronaldo del carmen", "ronaldo del carmen"], "dis": [[51325, 5663, 1493, 1568, 7323], [1413, 15101, 1394, 59322, 16104], [22728, 59325, 1408, 1808, 46539]]}, {"prop": "screenwriter", "subj": "Come and See", "obj": "Elem Klimov", "pop": 74692, "ids": [1507, 4925, 2532, 1379, 21997, 1384, 5611, 1523, 5336, 1520, 1529, 19599, 1681, 4097, 1804, 72], "a": 10, "n": 5, "alts": ["elem klimov", "elem germanovich klimov", "ales adamovich"], "dis": [[10804, 1413, 2250, 2874, 59398], [17657, 63, 54440, 6274, 1578], [17373, 19290, 14763, 20233, 1839]]}, {"prop": "screenwriter", "subj": "Iron Man 3", "obj": "Drew Pearce", "pop": 72606, "ids": [1507, 4925, 2532, 1379, 22415, 5174, 59320, 59362, 1523, 5336, 1520, 48603, 22948, 1437, 72], "a": 11, "n": 3, "alts": ["drew pearce", "warren ellis", "warren girard ellis", "shane black"], "dis": [[57553, 50098, 1578], [35037, 2419, 1539], [8266, 8486, 1443]]}, {"prop": "screenwriter", "subj": "Clue", "obj": "John Landis", "pop": 72498, "ids": [1507, 4925, 2532, 1379, 2534, 1511, 1523, 5336, 1520, 4801, 10726, 1360, 72], "a": 9, "n": 3, "alts": ["john landis", "jonathan lynn", "jonathan lynn"], "dis": [[30542, 23405, 4140], [19378, 5967, 4696], [11238, 7672, 23676]]}, {"prop": "screenwriter", "subj": "Labyrinth", "obj": "Jim Henson", "pop": 71935, "ids": [1507, 4925, 2532, 1379, 13381, 6444, 35183, 1523, 5336, 1520, 12980, 1491, 39117, 72], "a": 10, "n": 3, "alts": ["terry jones", "terence graham parry jones", "jim henson", "james maury henson"], "dis": [[27971, 1515, 44003], [11367, 26932, 48242], [18432, 48819, 2968]]}, {"prop": "screenwriter", "subj": "Kids", "obj": "Larry Clark", "pop": 71906, "ids": [1507, 4925, 2532, 1379, 25384, 1523, 5336, 1520, 32932, 24380, 72], "a": 8, "n": 2, "alts": ["harmony korine", "larry clark", "lawrence donald clark", "larry clarke", "larry lewis legrand clark", "larry james clark"], "dis": [[54210, 37501], [10518, 35578], [44166, 54934]]}, {"prop": "screenwriter", "subj": "Her", "obj": "Spike Jonze", "pop": 70875, "ids": [1507, 4925, 2532, 1379, 6555, 1523, 5336, 1520, 2870, 4601, 14972, 3846, 72], "a": 8, "n": 4, "alts": ["spike jonze", "adam spiegel"], "dis": [[18878, 29337, 1439, 3776], [15528, 1363, 12836, 1509], [9114, 1491, 2611, 1509]]}, {"prop": "screenwriter", "subj": "My Neighbor Totoro", "obj": "Hayao Miyazaki", "pop": 67908, "ids": [1507, 4925, 2532, 1379, 4162, 5732, 32245, 1408, 16963, 59326, 1523, 5336, 1520, 19093, 7022, 1445, 11976, 2215, 14770, 72], "a": 13, "n": 6, "alts": ["hayao miyazaki", "miyazaki hayao", "tsutomu teruki"], "dis": [[10677, 1549, 59361, 59366, 1563, 1509], [11388, 1765, 1515, 9747, 1391, 48067], [5732, 1487, 3474, 1390, 59346, 2171]]}, {"prop": "screenwriter", "subj": "Monsters, Inc.", "obj": "Dan Gerson", "pop": 66713, "ids": [1507, 4925, 2532, 1379, 5246, 20053, 59342, 6107, 72, 1523, 5336, 1520, 14953, 1527, 2332, 72], "a": 12, "n": 3, "alts": ["andrew stanton", "andrew christopher stanton jr.", "andrew ayers stanton", "dan gerson", "daniel gerson", "daniel robert gerson"], "dis": [[9812, 1494, 16404], [9114, 31482, 8573], [9812, 26634, 25742]]}, {"prop": "screenwriter", "subj": "Cube", "obj": "Vincenzo Natali", "pop": 65696, "ids": [1507, 4925, 2532, 1379, 58184, 1523, 5336, 1520, 1690, 3357, 1356, 15622, 46390, 59325, 72], "a": 8, "n": 6, "alts": ["vincenzo natali"], "dis": [[3227, 7637, 59326, 2548, 49968, 1485], [31235, 10277, 52080, 63, 59394, 33716], [50124, 59326, 1527, 1606, 1364, 7931]]}, {"prop": "screenwriter", "subj": "2012", "obj": "Roland Emmerich", "pop": 65303, "ids": [1507, 4925, 2532, 1379, 59320, 59349, 59344, 59345, 59349, 1523, 5336, 1520, 23020, 1483, 6503, 1805, 1620, 72], "a": 12, "n": 5, "alts": ["roland emmerich"], "dis": [[38596, 6466, 2270, 22537, 59398], [56154, 4908, 59340, 20242, 59325], [43464, 2120, 6555, 59398, 1514]]}, {"prop": "screenwriter", "subj": "Taken", "obj": "Luc Besson", "pop": 64711, "ids": [1507, 4925, 2532, 1379, 45090, 1523, 5336, 1520, 14811, 1469, 1481, 1350, 72], "a": 8, "n": 4, "alts": ["robert mark kamen", "luc besson", "luc paul maurice besson"], "dis": [[14881, 1469, 1481, 3243], [20201, 1491, 4305, 1827], [9114, 1419, 13758, 5399]]}, {"prop": "screenwriter", "subj": "WALL\u00b7E", "obj": "Andrew Stanton", "pop": 64054, "ids": [1507, 4925, 2532, 1379, 1495, 8975, 59936, 59364, 1523, 5336, 1520, 15054, 1716, 40097, 72], "a": 11, "n": 3, "alts": ["jim reardon", "andrew stanton", "andrew christopher stanton jr.", "andrew ayers stanton"], "dis": [[29208, 26534, 1961], [12583, 25346, 1643], [31068, 41414, 5922]]}, {"prop": "screenwriter", "subj": "The Reader", "obj": "Bernhard Schlink", "pop": 63712, "ids": [1507, 4925, 2532, 1379, 1507, 38157, 1523, 5336, 1520, 15466, 15138, 5967, 4322, 72], "a": 9, "n": 4, "alts": ["david hare", "sir david hare", "bernhard schlink"], "dis": [[34721, 3814, 38688, 2450], [4915, 7374, 44420, 1364], [11622, 20614, 7341, 1809]]}, {"prop": "screenwriter", "subj": "Groundhog Day", "obj": "Harold Ramis", "pop": 63308, "ids": [1507, 4925, 2532, 1379, 28790, 58498, 6666, 1523, 5336, 1520, 50377, 12339, 1360, 72], "a": 10, "n": 3, "alts": ["harold ramis", "harold allen ramis", "danny rubin"], "dis": [[9812, 3509, 5398], [7524, 45253, 1541], [48603, 22948, 1437]]}, {"prop": "screenwriter", "subj": "The Town", "obj": "Ben Affleck", "pop": 63230, "ids": [1507, 4925, 2532, 1379, 1507, 8962, 1523, 5336, 1520, 7370, 11965, 1378, 2108, 72], "a": 9, "n": 4, "alts": ["ben affleck", "benjami g\u00e9za affleck", "benjamin g\u00e9za affleck-boldt", "benjami geza affleck", "benjamin geza affleck-boldt"], "dis": [[9549, 3721, 1350, 1363], [28204, 2419, 1485, 7662], [7196, 39449, 1419, 2329]]}, {"prop": "screenwriter", "subj": "Spotlight", "obj": "Josh Singer", "pop": 62360, "ids": [1507, 4925, 2532, 1379, 28609, 4891, 1523, 5336, 1520, 24846, 59064, 72], "a": 9, "n": 2, "alts": ["josh singer"], "dis": [[45514, 22393], [9137, 21861], [37160, 34967]]}, {"prop": "screenwriter", "subj": "Angels & Demons", "obj": "David Koepp", "pop": 62172, "ids": [1507, 4925, 2532, 1379, 47000, 1930, 6572, 2072, 1523, 5336, 1520, 7524, 31858, 1636, 59336, 72], "a": 11, "n": 4, "alts": ["david koepp", "akiva goldsman", "akiva j. goldsman"], "dis": [[21927, 1360, 15821, 1371], [7821, 40344, 1578, 10459], [41138, 4498, 1387, 1349]]}, {"prop": "screenwriter", "subj": "Office Space", "obj": "Mike Judge", "pop": 62042, "ids": [1507, 4925, 2532, 1379, 9489, 14289, 1523, 5336, 1520, 13738, 26628, 72], "a": 9, "n": 2, "alts": ["mike judge", "judgemental films", "michael judge", "michael craig judge", "michael c. judge"], "dis": [[12344, 41106], [29041, 36027], [18223, 52904]]}, {"prop": "screenwriter", "subj": "Closer", "obj": "Patrick Marber", "pop": 62010, "ids": [1507, 4925, 2532, 1379, 2534, 30452, 1523, 5336, 1520, 21942, 3318, 1640, 72], "a": 9, "n": 3, "alts": ["patrick marber", "patrick albert crispin marber"], "dis": [[42354, 8213, 2450], [53752, 34574, 3487], [5378, 24859, 4102]]}, {"prop": "screenwriter", "subj": "The Internship", "obj": "Vince Vaughn", "pop": 61374, "ids": [1507, 4925, 2532, 1379, 1507, 4387, 6073, 1523, 5336, 1520, 51170, 1690, 5590, 59322, 72], "a": 10, "n": 4, "alts": ["vince vaughn", "vincent anthony vaughn", "vincent vaughn"], "dis": [[27229, 1413, 2095, 2825], [23794, 1479, 1481, 1356], [8266, 1413, 1541, 1360]]}, {"prop": "screenwriter", "subj": "Casino", "obj": "Nicholas Pileggi", "pop": 59995, "ids": [1507, 4925, 2532, 1379, 35411, 1523, 5336, 1520, 39307, 1452, 1603, 8053, 59325, 72], "a": 8, "n": 5, "alts": ["martin scorsese", "martin skorseze", "martin charles scorsese", "nicholas pileggi", "nicolas pileggi"], "dis": [[5279, 1839, 25690, 1404, 7323], [9196, 1653, 1469, 1886, 2645], [4801, 1408, 7362, 1407, 1381]]}, {"prop": "screenwriter", "subj": "Silence", "obj": "Martin Scorsese", "pop": 58729, "ids": [1507, 4925, 2532, 1379, 9555, 1647, 1523, 5336, 1520, 12489, 2732, 1698, 4768, 72], "a": 9, "n": 4, "alts": ["jay cocks", "martin scorsese", "martin skorseze", "martin charles scorsese"], "dis": [[25214, 10541, 1498, 2160], [7821, 9519, 7214, 10431], [13738, 9391, 7196, 1454]]}, {"prop": "screenwriter", "subj": "Mission: Impossible \u2013 Ghost Protocol", "obj": "Josh Appelbaum", "pop": 57946, "ids": [1507, 4925, 2532, 1379, 25861, 59358, 4016, 13137, 1991, 40220, 31331, 1523, 5336, 1520, 24846, 5216, 1391, 54486, 72], "a": 14, "n": 4, "alts": ["josh appelbaum"], "dis": [[17101, 7607, 6572, 43170], [11238, 43445, 1847, 11467], [46319, 5929, 59338, 39735]]}, {"prop": "screenwriter", "subj": "Room", "obj": "Emma Donoghue", "pop": 56453, "ids": [1507, 4925, 2532, 1379, 15980, 1523, 5336, 1520, 40813, 7063, 57210, 1511, 72], "a": 8, "n": 4, "alts": ["emma donoghue"], "dis": [[1479, 6716, 17116, 59326], [1494, 1998, 3491, 1349], [7524, 31858, 1636, 59336]]}, {"prop": "screenwriter", "subj": "A.I. Artificial Intelligence", "obj": "Steven Spielberg", "pop": 54344, "ids": [1507, 4925, 2532, 1379, 1420, 72, 59355, 72, 46715, 28344, 1523, 5336, 1520, 28204, 2870, 20090, 7662, 72], "a": 13, "n": 4, "alts": ["ian watson", "stanley kubrick", "steven spielberg", "steven allan spielberg", "spielberg"], "dis": [[7821, 4063, 59392, 1472], [32502, 1406, 10677, 1356], [5929, 1429, 2814, 51986]]}, {"prop": "screenwriter", "subj": "Lucy", "obj": "Luc Besson", "pop": 53411, "ids": [1507, 4925, 2532, 1379, 51354, 1523, 5336, 1520, 14811, 1469, 1481, 1350, 72], "a": 8, "n": 4, "alts": ["luc besson", "luc paul maurice besson"], "dis": [[28653, 1490, 3152, 1487], [20201, 1491, 4305, 1827], [2426, 14965, 12180, 1350]]}, {"prop": "screenwriter", "subj": "Brave", "obj": "Brenda Chapman", "pop": 53071, "ids": [1507, 4925, 2532, 1379, 3317, 1543, 1523, 5336, 1520, 8405, 40040, 23405, 2450, 72], "a": 9, "n": 4, "alts": ["steve purcell", "steve ross purcell", "steven ross purcell", "brenda chapman", "irene mecchi"], "dis": [[9486, 4592, 1611, 7374], [24052, 1493, 2337, 59328], [9878, 1529, 1360, 2120]]}, {"prop": "screenwriter", "subj": "Do the Right Thing", "obj": "Spike Lee", "pop": 53051, "ids": [1507, 4925, 2532, 1379, 4498, 1358, 15815, 47849, 1523, 5336, 1520, 2870, 4601, 12917, 72], "a": 11, "n": 3, "alts": ["spike lee", "sheldon lee", "shelton j. lee", "shelton jackson lee", "mars blackmon"], "dis": [[37826, 16105, 1576], [21872, 32284, 39117], [17657, 26991, 7968]]}, {"prop": "sport", "subj": "Mr. Olympia", "obj": "bodybuilding", "pop": 316448, "ids": [6859, 72, 6908, 1627, 1410, 1348, 4956, 3795, 22210, 5499, 72], "a": 7, "n": 3, "alts": ["bodybuilding", "body building", "body-building"], "dis": [[7584, 59326, 5499], [35718, 1500, 5499], [50830, 40478, 5499]]}, {"prop": "sport", "subj": "James Rodr\u00edguez", "obj": "association football", "pop": 151571, "ids": [8266, 34817, 60577, 33582, 59398, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 8, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[46272, 16255, 5499], [7589, 51142, 5499], [2864, 15403, 5499]]}, {"prop": "sport", "subj": "Atl\u00e9tico de Madrid", "obj": "association football", "pop": 144999, "ids": [11851, 22866, 5663, 1564, 27118, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 8, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[3126, 49117, 5499], [2864, 15403, 5499], [7229, 18525, 5499]]}, {"prop": "sport", "subj": "St. Louis Cardinals", "obj": "baseball", "pop": 89017, "ids": [1716, 72, 10674, 50041, 1410, 1348, 4956, 16920, 5499, 72], "a": 7, "n": 2, "alts": ["baseball", "america's pastime", "\u26be"], "dis": [[16511, 5499], [34940, 5499], [49117, 5499]]}, {"prop": "sport", "subj": "Jason Kidd", "obj": "basketball", "pop": 75179, "ids": [24052, 1681, 3191, 1410, 1348, 4956, 12012, 5499, 72], "a": 6, "n": 2, "alts": ["basketball", "hoops", "b-ball", "basket ball", "bb", "basketball"], "dis": [[45919, 5499], [34940, 5499], [49117, 5499]]}, {"prop": "sport", "subj": "Coventry City F.C.", "obj": "association football", "pop": 50783, "ids": [22402, 9455, 4599, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 10, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[8445, 2626, 5499], [9614, 15403, 5499], [7584, 59326, 5499]]}, {"prop": "sport", "subj": "Birmingham City F.C.", "obj": "association football", "pop": 50643, "ids": [28731, 4599, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 9, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[4370, 26528, 5499], [35718, 1500, 5499], [2998, 50673, 5499]]}, {"prop": "sport", "subj": "Los Angeles Galaxy", "obj": "association football", "pop": 49982, "ids": [11209, 12089, 25065, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 6, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[50830, 40478, 5499], [2998, 50673, 5499], [24171, 4502, 5499]]}, {"prop": "sport", "subj": "David Coulthard", "obj": "Formula One", "pop": 38271, "ids": [7524, 52860, 1440, 1578, 1410, 1348, 4956, 24171, 4502, 5499, 72], "a": 7, "n": 3, "alts": ["formula one", "fia formula one world championship", "f1", "formula 1"], "dis": [[2998, 50673, 5499], [9614, 15403, 5499], [2782, 50673, 5499]]}, {"prop": "sport", "subj": "Pittsburgh Penguins", "obj": "ice hockey", "pop": 38191, "ids": [28623, 1452, 33749, 2233, 1410, 1348, 4956, 9614, 15403, 5499, 72], "a": 7, "n": 3, "alts": ["ice hockey"], "dis": [[35718, 1500, 5499], [19004, 12084, 5499], [12935, 34940, 5499]]}, {"prop": "sport", "subj": "Jerry West", "obj": "basketball", "pop": 35717, "ids": [32905, 5308, 1410, 1348, 4956, 12012, 5499, 72], "a": 5, "n": 2, "alts": ["basketball", "hoops", "b-ball", "basket ball", "bb", "basketball"], "dis": [[36756, 5499], [55224, 5499], [16511, 5499]]}, {"prop": "sport", "subj": "Jacob Eason", "obj": "American football", "pop": 34533, "ids": [22200, 1529, 2422, 1410, 1348, 4956, 3295, 3794, 5499, 72], "a": 6, "n": 3, "alts": ["american football", "gridiron football", "\ud83c\udfc8", "football", "american rules football"], "dis": [[19004, 12084, 5499], [5470, 51142, 5499], [2782, 50673, 5499]]}, {"prop": "sport", "subj": "Bryce Harper", "obj": "baseball", "pop": 32766, "ids": [21759, 1437, 44846, 1410, 1348, 4956, 16920, 5499, 72], "a": 6, "n": 2, "alts": ["baseball", "america's pastime", "\u26be"], "dis": [[12012, 5499], [15715, 5499], [34940, 5499]]}, {"prop": "sport", "subj": "tight end", "obj": "American football", "pop": 30557, "ids": [12884, 2164, 1410, 1348, 4956, 3295, 3794, 5499, 72], "a": 5, "n": 3, "alts": ["american football", "gridiron football", "\ud83c\udfc8", "football", "american rules football"], "dis": [[2998, 50673, 5499], [19004, 12084, 5499], [8445, 2626, 5499]]}, {"prop": "sport", "subj": "Indianapolis 500", "obj": "motorsport", "pop": 30423, "ids": [46197, 59320, 59367, 59344, 59344, 1410, 1348, 4956, 35718, 1500, 5499, 72], "a": 8, "n": 3, "alts": ["motorsport", "motorsports", "motor sport", "motor sports", "motor racing"], "dis": [[46272, 16255, 5499], [3295, 3794, 5499], [19004, 10539, 5499]]}, {"prop": "sport", "subj": "St. Louis Blues", "obj": "ice hockey", "pop": 29618, "ids": [1716, 72, 10674, 30421, 1410, 1348, 4956, 9614, 15403, 5499, 72], "a": 7, "n": 3, "alts": ["ice hockey"], "dis": [[21278, 34258, 5499], [12935, 34940, 5499], [4303, 19712, 5499]]}, {"prop": "sport", "subj": "linebacker", "obj": "American football", "pop": 29343, "ids": [2473, 2553, 1349, 1410, 1348, 4956, 3295, 3794, 5499, 72], "a": 6, "n": 3, "alts": ["american football", "gridiron football", "\ud83c\udfc8", "football", "american rules football"], "dis": [[5963, 48894, 5499], [7229, 18525, 5499], [4303, 19712, 5499]]}, {"prop": "sport", "subj": "Scuderia Toro Rosso", "obj": "Formula One", "pop": 28559, "ids": [2732, 1528, 7604, 9429, 59326, 17479, 1656, 1410, 1348, 4956, 24171, 4502, 5499, 72], "a": 10, "n": 3, "alts": ["formula one", "fia formula one world championship", "f1", "formula 1"], "dis": [[3795, 22210, 5499], [19004, 10539, 5499], [4370, 26528, 5499]]}, {"prop": "sport", "subj": "Toni Kuko\u010d", "obj": "basketball", "pop": 26301, "ids": [1408, 18602, 1681, 3776, 59326, 61757, 1410, 1348, 4956, 12012, 5499, 72], "a": 9, "n": 2, "alts": ["basketball", "hoops", "b-ball", "basket ball", "bb", "basketball"], "dis": [[16920, 5499], [56660, 5499], [19712, 5499]]}, {"prop": "sport", "subj": "Josef Mart\u00ednez", "obj": "association football", "pop": 25882, "ids": [11408, 59337, 8454, 60577, 55630, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 8, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[5470, 51142, 5499], [9614, 15403, 5499], [5963, 48894, 5499]]}, {"prop": "sport", "subj": "Leyton Orient F.C.", "obj": "association football", "pop": 25844, "ids": [59204, 2232, 1549, 18614, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 11, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[3295, 3794, 5499], [50830, 40478, 5499], [21278, 34258, 5499]]}, {"prop": "sport", "subj": "Cris Carter", "obj": "American football", "pop": 23539, "ids": [5758, 1360, 30692, 1410, 1348, 4956, 3295, 3794, 5499, 72], "a": 6, "n": 3, "alts": ["american football", "gridiron football", "\ud83c\udfc8", "football", "american rules football"], "dis": [[5963, 48894, 5499], [4370, 26528, 5499], [2998, 50673, 5499]]}, {"prop": "sport", "subj": "Vancouver Whitecaps FC", "obj": "association football", "pop": 21937, "ids": [25485, 23757, 1501, 3536, 13837, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 8, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[3295, 3794, 5499], [35718, 1500, 5499], [8445, 2626, 5499]]}, {"prop": "sport", "subj": "Katie Taylor", "obj": "boxing", "pop": 21488, "ids": [53722, 15248, 1410, 1348, 4956, 45919, 5499, 72], "a": 5, "n": 2, "alts": ["boxing"], "dis": [[16511, 5499], [16920, 5499], [12012, 5499]]}, {"prop": "sport", "subj": "Tranmere Rovers F.C.", "obj": "association football", "pop": 21475, "ids": [2278, 1363, 59335, 1486, 57261, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 12, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[7584, 59326, 5499], [19004, 10539, 5499], [35718, 1500, 5499]]}, {"prop": "sport", "subj": "\u00d6stersunds FK", "obj": "association football", "pop": 19815, "ids": [47533, 20053, 1660, 59328, 54103, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 8, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[9614, 15403, 5499], [35718, 1500, 5499], [3295, 3794, 5499]]}, {"prop": "sport", "subj": "King's Indian Defence", "obj": "chess", "pop": 19374, "ids": [6132, 59361, 59328, 6438, 38220, 1410, 1348, 4956, 36756, 5499, 72], "a": 8, "n": 2, "alts": ["chess", "international chess", "modern european chess", "western chess", "chess game"], "dis": [[19712, 5499], [15715, 5499], [56660, 5499]]}, {"prop": "sport", "subj": "Margaret Court", "obj": "tennis", "pop": 16158, "ids": [32740, 8845, 1410, 1348, 4956, 19712, 5499, 72], "a": 5, "n": 2, "alts": ["tennis", "lawn tennis", "lawntennis"], "dis": [[55224, 5499], [16920, 5499], [12012, 5499]]}, {"prop": "sport", "subj": "Doncaster Rovers F.C.", "obj": "association football", "pop": 15353, "ids": [7063, 35266, 57261, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 10, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[3295, 3794, 5499], [4303, 19712, 5499], [50830, 40478, 5499]]}, {"prop": "sport", "subj": "Anderson", "obj": "association football", "pop": 14892, "ids": [22393, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 4, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[19004, 12084, 5499], [35718, 1500, 5499], [2782, 50673, 5499]]}, {"prop": "sport", "subj": "Margarita Louis-Dreyfus", "obj": "association football", "pop": 13217, "ids": [20189, 1370, 8757, 10674, 63, 59371, 7884, 59337, 1406, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 12, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[8445, 2626, 5499], [3295, 3794, 5499], [9614, 15403, 5499]]}, {"prop": "sport", "subj": "Exeter City F.C.", "obj": "association football", "pop": 13120, "ids": [2387, 4493, 4599, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 10, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[19004, 10539, 5499], [2864, 15403, 5499], [12935, 34940, 5499]]}, {"prop": "sport", "subj": "Olympique Lyonnais F\u00e9minin", "obj": "association football", "pop": 12986, "ids": [6908, 3024, 16151, 38885, 1360, 1493, 60025, 2129, 1347, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 12, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[7229, 18525, 5499], [4370, 26528, 5499], [35718, 1500, 5499]]}, {"prop": "sport", "subj": "Boston United F.C.", "obj": "association football", "pop": 12930, "ids": [12923, 3532, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 9, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[3795, 22210, 5499], [8871, 11167, 5499], [7229, 18525, 5499]]}, {"prop": "sport", "subj": "earned run average", "obj": "baseball", "pop": 12609, "ids": [13938, 2399, 5090, 1410, 1348, 4956, 16920, 5499, 72], "a": 6, "n": 2, "alts": ["baseball", "america's pastime", "\u26be"], "dis": [[56660, 5499], [45919, 5499], [19712, 5499]]}, {"prop": "sport", "subj": "James Haskell", "obj": "rugby union", "pop": 12082, "ids": [8266, 57956, 1410, 1348, 4956, 19004, 12084, 5499, 72], "a": 5, "n": 3, "alts": ["rugby union", "rugby"], "dis": [[24171, 4502, 5499], [50830, 40478, 5499], [5470, 51142, 5499]]}, {"prop": "sport", "subj": "Houston Dynamo FC", "obj": "association football", "pop": 11557, "ids": [17422, 41515, 59326, 13837, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 7, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[35718, 1500, 5499], [46272, 16255, 5499], [4303, 19712, 5499]]}, {"prop": "sport", "subj": "Cheltenham Town F.C.", "obj": "association football", "pop": 11175, "ids": [26262, 50612, 8962, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 10, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[3126, 49117, 5499], [7584, 59326, 5499], [9614, 15403, 5499]]}, {"prop": "sport", "subj": "Joey Harrington", "obj": "American football", "pop": 10811, "ids": [14195, 2825, 16222, 5939, 1410, 1348, 4956, 3295, 3794, 5499, 72], "a": 7, "n": 3, "alts": ["american football", "gridiron football", "\ud83c\udfc8", "football", "american rules football"], "dis": [[5963, 48894, 5499], [7589, 51142, 5499], [19004, 10539, 5499]]}, {"prop": "sport", "subj": "run batted in", "obj": "softball", "pop": 9995, "ids": [2399, 1376, 19132, 1377, 1410, 1348, 4956, 8445, 2626, 5499, 72], "a": 7, "n": 3, "alts": ["softball", "baseball", "america's pastime", "\u26be"], "dis": [[19004, 12084, 5499], [5963, 48894, 5499], [4303, 19712, 5499]]}, {"prop": "sport", "subj": "Boreham Wood F.C.", "obj": "association football", "pop": 9786, "ids": [1469, 1512, 5399, 10714, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 11, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[5963, 48894, 5499], [19004, 10539, 5499], [7589, 51142, 5499]]}, {"prop": "sport", "subj": "Steve Coppell", "obj": "association football", "pop": 9236, "ids": [15816, 18030, 59336, 1568, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 7, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[2998, 50673, 5499], [3795, 22210, 5499], [7589, 51142, 5499]]}, {"prop": "sport", "subj": "penalty", "obj": "ice hockey", "pop": 9185, "ids": [24178, 1410, 1348, 4956, 9614, 15403, 5499, 72], "a": 4, "n": 3, "alts": ["ice hockey"], "dis": [[3126, 49117, 5499], [19004, 12084, 5499], [24171, 4502, 5499]]}, {"prop": "sport", "subj": "Bob Lanier", "obj": "basketball", "pop": 9081, "ids": [13187, 36391, 2230, 1410, 1348, 4956, 12012, 5499, 72], "a": 6, "n": 2, "alts": ["basketball", "hoops", "b-ball", "basket ball", "bb", "basketball"], "dis": [[16511, 5499], [49117, 5499], [16920, 5499]]}, {"prop": "sport", "subj": "Port Adelaide Football Club", "obj": "Australian rules football", "pop": 8894, "ids": [6551, 37711, 10775, 7888, 1410, 1348, 4956, 8853, 7488, 3794, 5499, 72], "a": 7, "n": 4, "alts": ["australian rules football", "football", "footy", "aussie rules", "australian football", "aussie rules football", "afl", "vfl"], "dis": [[53449, 63, 20030, 5499], [24145, 41527, 29423, 5499], [5800, 37718, 18525, 5499]]}, {"prop": "sport", "subj": "Dragons", "obj": "rugby union", "pop": 7568, "ids": [32585, 2072, 1410, 1348, 4956, 19004, 12084, 5499, 72], "a": 5, "n": 3, "alts": ["rugby union", "rugby"], "dis": [[8445, 2626, 5499], [5470, 51142, 5499], [50830, 40478, 5499]]}, {"prop": "sport", "subj": "Bernadette Sz\u0151cs", "obj": "table tennis", "pop": 6661, "ids": [15466, 1422, 8057, 34231, 63232, 5137, 1410, 1348, 4956, 4303, 19712, 5499, 72], "a": 9, "n": 3, "alts": ["table tennis", "ping pong", "wiff waff", "ping-pong"], "dis": [[19004, 10539, 5499], [4370, 26528, 5499], [3795, 22210, 5499]]}, {"prop": "sport", "subj": "Chuck Bednarik", "obj": "American football", "pop": 6638, "ids": [45254, 16023, 59322, 1920, 59346, 1410, 1348, 4956, 3295, 3794, 5499, 72], "a": 8, "n": 3, "alts": ["american football", "gridiron football", "\ud83c\udfc8", "football", "american rules football"], "dis": [[50830, 40478, 5499], [2998, 50673, 5499], [5963, 48894, 5499]]}, {"prop": "sport", "subj": "Australian Football League Draft", "obj": "Australian rules football", "pop": 6549, "ids": [8853, 10775, 5668, 29880, 1410, 1348, 4956, 8853, 7488, 3794, 5499, 72], "a": 7, "n": 4, "alts": ["australian rules football", "football", "footy", "aussie rules", "australian football", "aussie rules football", "afl", "vfl"], "dis": [[24145, 41527, 29423, 5499], [6413, 1514, 1778, 5499], [53449, 63, 20030, 5499]]}, {"prop": "sport", "subj": "FINA", "obj": "water polo", "pop": 6439, "ids": [1493, 40710, 1410, 1348, 4956, 3126, 49117, 5499, 72], "a": 5, "n": 3, "alts": ["open water swimming", "water polo", "diving"], "dis": [[12935, 34940, 5499], [2864, 15403, 5499], [7589, 51142, 5499]]}, {"prop": "sport", "subj": "Hanworth Villa F.C.", "obj": "association football", "pop": 5626, "ids": [21139, 15119, 34857, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 10, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[7589, 51142, 5499], [50830, 40478, 5499], [19004, 10539, 5499]]}, {"prop": "sport", "subj": "Welling United F.C.", "obj": "association football", "pop": 5135, "ids": [1495, 4538, 3532, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 10, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[3295, 3794, 5499], [46272, 16255, 5499], [19004, 10539, 5499]]}, {"prop": "sport", "subj": "Iowa Wild", "obj": "ice hockey", "pop": 4928, "ids": [18543, 13337, 1410, 1348, 4956, 9614, 15403, 5499, 72], "a": 5, "n": 3, "alts": ["ice hockey"], "dis": [[19004, 12084, 5499], [3295, 3794, 5499], [24171, 4502, 5499]]}, {"prop": "sport", "subj": "Eric Snow", "obj": "basketball", "pop": 4748, "ids": [19255, 24746, 1410, 1348, 4956, 12012, 5499, 72], "a": 5, "n": 2, "alts": ["basketball", "hoops", "b-ball", "basket ball", "bb", "basketball"], "dis": [[15715, 5499], [34940, 5499], [45919, 5499]]}, {"prop": "sport", "subj": "Madison", "obj": "track cycling", "pop": 4466, "ids": [32892, 1410, 1348, 4956, 4370, 26528, 5499, 72], "a": 4, "n": 3, "alts": ["track cycling", "track bike"], "dis": [[12935, 34940, 5499], [19004, 12084, 5499], [8566, 3794, 5499]]}, {"prop": "sport", "subj": "Kasperi Kapanen", "obj": "ice hockey", "pop": 4464, "ids": [39694, 1576, 59325, 1681, 5046, 1356, 1410, 1348, 4956, 9614, 15403, 5499, 72], "a": 9, "n": 3, "alts": ["ice hockey"], "dis": [[4303, 19712, 5499], [8566, 3794, 5499], [7589, 51142, 5499]]}, {"prop": "sport", "subj": "Deportivo Cali", "obj": "association football", "pop": 4439, "ids": [2748, 1500, 7003, 4544, 59325, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 8, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[2998, 50673, 5499], [3126, 49117, 5499], [19004, 10539, 5499]]}, {"prop": "sport", "subj": "David Campese", "obj": "rugby union", "pop": 4394, "ids": [7524, 10102, 4768, 1410, 1348, 4956, 19004, 12084, 5499, 72], "a": 6, "n": 3, "alts": ["rugby union", "rugby"], "dis": [[7589, 51142, 5499], [3126, 49117, 5499], [9614, 15403, 5499]]}, {"prop": "sport", "subj": "Crease", "obj": "cricket", "pop": 4202, "ids": [26794, 1582, 1410, 1348, 4956, 15715, 5499, 72], "a": 5, "n": 2, "alts": ["cricket"], "dis": [[56660, 5499], [45919, 5499], [16920, 5499]]}, {"prop": "sport", "subj": "Sporting News", "obj": "baseball", "pop": 4110, "ids": [19301, 1371, 8355, 1410, 1348, 4956, 16920, 5499, 72], "a": 6, "n": 2, "alts": ["baseball", "america's pastime", "\u26be"], "dis": [[12012, 5499], [19712, 5499], [55224, 5499]]}, {"prop": "sport", "subj": "Italy men's national volleyball team", "obj": "volleyball", "pop": 4010, "ids": [11840, 3797, 59361, 59328, 5354, 34940, 2840, 1410, 1348, 4956, 34940, 5499, 72], "a": 10, "n": 2, "alts": ["volleyball"], "dis": [[16920, 5499], [15715, 5499], [38623, 5499]]}, {"prop": "sport", "subj": "bonus", "obj": "basketball", "pop": 3886, "ids": [18723, 1410, 1348, 4956, 12012, 5499, 72], "a": 4, "n": 2, "alts": ["basketball", "hoops", "b-ball", "basket ball", "bb", "basketball"], "dis": [[38623, 5499], [16511, 5499], [49117, 5499]]}, {"prop": "sport", "subj": "Chertsey Town F.C.", "obj": "association football", "pop": 3769, "ids": [1793, 1822, 8429, 8962, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 11, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[4303, 19712, 5499], [12935, 34940, 5499], [9614, 15403, 5499]]}, {"prop": "sport", "subj": "Club Libertad", "obj": "association football", "pop": 3392, "ids": [7888, 57635, 1422, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 6, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[2998, 50673, 5499], [4303, 19712, 5499], [3126, 49117, 5499]]}, {"prop": "sport", "subj": "Grantham Town F.C.", "obj": "association football", "pop": 3359, "ids": [3509, 12555, 1398, 8962, 1493, 72, 59357, 72, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 11, "n": 3, "alts": ["association football", "football", "soccer"], "dis": [[12935, 34940, 5499], [3295, 3794, 5499], [21278, 34258, 5499]]}, {"prop": "sport", "subj": "antichess", "obj": "chess", "pop": 3330, "ids": [2703, 25509, 1885, 1410, 1348, 4956, 36756, 5499, 72], "a": 6, "n": 2, "alts": ["chess", "international chess", "modern european chess", "western chess", "chess game"], "dis": [[16920, 5499], [49117, 5499], [55224, 5499]]}, {"prop": "sport", "subj": "grip", "obj": "badminton", "pop": 3215, "ids": [33136, 1410, 1348, 4956, 5963, 48894, 5499, 72], "a": 4, "n": 3, "alts": ["badminton"], "dis": [[24171, 4502, 5499], [21278, 34258, 5499], [8566, 3794, 5499]]}, {"prop": "sport", "subj": "Buffalo Memorial Auditorium", "obj": "basketball", "pop": 3105, "ids": [35417, 20405, 15402, 59006, 1410, 1348, 4956, 12012, 5499, 72], "a": 7, "n": 2, "alts": ["basketball", "hoops", "b-ball", "basket ball", "bb", "basketball"], "dis": [[55224, 5499], [38623, 5499], [34940, 5499]]}, {"prop": "sport", "subj": "Ligue Magnus", "obj": "ice hockey", "pop": 3029, "ids": [1494, 15842, 30833, 1406, 1410, 1348, 4956, 9614, 15403, 5499, 72], "a": 7, "n": 3, "alts": ["ice hockey"], "dis": [[35718, 1500, 5499], [8871, 11167, 5499], [24171, 4502, 5499]]}, {"prop": "sport", "subj": "goalkeeper", "obj": "association football", "pop": 2992, "ids": [47942, 1410, 1348, 4956, 8566, 3794, 5499, 72], "a": 4, "n": 3, "alts": ["field hockey", "hockey", "international rules football", "bandy", "russian ice hockey", "lacrosse", "la crosse", "gaelic football", "floorball", "innebandy", "unihockey", "salibandy", "hurling", "association football", "football", "soccer", "ice hockey", "water polo", "handball", "olympic handball", "european team handball", "european handball", "borden ball", "pallammanu", "team handball"], "dis": [[24171, 4502, 5499], [21278, 34258, 5499], [2782, 50673, 5499]]}] \ No newline at end of file diff --git a/pilots/frontdec.log b/pilots/frontdec.log new file mode 100644 index 0000000000000000000000000000000000000000..dddf0a5b3d1dca6579d602861cc72bae6a8d2eab --- /dev/null +++ b/pilots/frontdec.log @@ -0,0 +1,22 @@ +EMA round 507 step 135150 +[blk8] The capital of France is ||| the city of Paris, located in the heart of the Mediterranean Sea. It is the largest city in the world and the largest city in the world. References External links Official website of Paris Cities of France Populated places in France Cities of Paris City of +[blk8] In the morning, the old fisherman walked down to ||| his house, swayed against the banks of the river, and began to make his way back home. He quickly sank into the water, and he tried to find a home. However, as the river flowed, he realized that he was not alone. Though he had been struggling to find his home, +[blk8] Photosynthesis is the process by which ||| plants convert light energy into chemical energy through the production of carbon dioxide. This process occurs when plants absorb sunlight, which is absorbed by their leaves, and release carbon dioxide into the atmosphere. As the carbon dioxide rises into the +[blk8] The main causes of the First World War were ||| the fall of the Ottoman Empire, the outbreak of the Second World War, and the outbreak of the First World War. The fall of the Ottoman Empire was the primary cause of the First World War. The outbreak of the First World War was primarily caused by the outbreak +RESULT blk8 distinct-2 0.776 rep4 0.100 tokens/pass 1.00 popqa 27/200 (13.5%) 65s +[ar1] The capital of France is ||| the city of Paris, the capital city of Guelph, the capital city of Monument, and the capital city of Le Bahé. The list of cities in France is as follows: The capital of France is the city ofovsk. The capital city of France is the city of Paris. +[ar1] In the morning, the old fisherman walked down to ||| his fisherman and caught the fish at the sea. He sat there for a while, watching the fish wryly. “Was you thinking about me?” he asked. “I’m so tired of my friends,” he said, “and I’m so sorry for your lifelessness. +[ar1] Photosynthesis is the process by which ||| plants convert food into energy through the process. This process occurs when plants absorb sunlight, which can be broken down into oxygen and carbon dioxide. When plants produce energy, these substances are used in the form of carbohydrates and water, which a +[ar1] The main causes of the First World War were ||| the diplomatic and political differences between France and Germany. The war was resolved by the Treaty of Strasbourg. The war ended in 1915, and the conflict was still ongoing. The outbreak of World War I was marked by the defeat of the Allies, including the +RESULT ar1 distinct-2 0.852 rep4 0.042 tokens/pass 1.00 popqa 27/200 (13.5%) 57s +[front] The capital of France is ||| the city of Paris, located in the heart of the country. The city is also known as the capital of Germany. See also List of cities in France List of cities of France References External links Capitals of France Populated places in Paris +[front] In the morning, the old fisherman walked down to ||| his family's backyard, watching his fish were feeding. One day, he noticed his own fish, Willie, who was really hungry. He asked his mom, who was also hungry, and she answered him. "Hey, I'm hungry, Willie," his mom said. "They're +[front] Photosynthesis is the process by which ||| plants convert sunlight into energy through the use of oxygen. This process occurs primarily in the leaves and involves the use of carbon dioxide, water, and light to generate energy. During photosynthesis, oxygen is released from the atmosphere to produce wat +[front] The main causes of the First World War were ||| the outbreak of World War I and the outbreak of World War II. In 1913, the Allies launched the invasion of Germany, which ended the war. In 1914, the Allies launched the invasion of France and the invasion of Italy. In 1915, the +RESULT front distinct-2 0.820 rep4 0.071 tokens/pass 1.13 popqa 25/200 (12.5%) 38s +[front_hi] The capital of France is ||| the city of Paris, located in the northern region of the country. The city is served by the city of Paris, which is the seat of the Republic of France. References External links Official website Capitals of France Capitals of Paris France +[front_hi] In the morning, the old fisherman walked down to ||| his family's backyard, watching his fish were feeding. One day, he noticed something strange - the fish were getting sick and dying! He decided to try some home remedies to help them recover. One day, while looking for remedies, he found a bunch of plants tha +[front_hi] Photosynthesis is the process by which ||| plants convert sunlight into energy through the use of oxygen. This process is essential in the survival of plants, their environment, and the development of life on Earth. Photosynthesis occurs when plants absorb sunlight from the sun, which is then converted +[front_hi] The main causes of the First World War were ||| the outbreak of the Second World War and the end of the First World War. The First World War was caused by the breakup of the First World War and the end of the Second World War. The First World War was the end of the Second World War and the beginning of the +RESULT front_hi distinct-2 0.814 rep4 0.072 tokens/pass 1.22 popqa 25/200 (12.5%) 37s +FRONTDEC DONE diff --git a/pilots/gen_gap0.txt b/pilots/gen_gap0.txt new file mode 100644 index 0000000000000000000000000000000000000000..a5c0ec2fc41a20654322ac8387aa34141217dfc0 --- /dev/null +++ b/pilots/gen_gap0.txt @@ -0,0 +1,70 @@ +calls=20 for 96 tokens (AR would need 96) + +PROMPT: The capital of France is +OUT: the sovere of a republic, with the Rébec-province of, the sovere of an agreement (France-the-ign of), and the capital of France is the theign of, with the the treaty of, the sovere ofnty of the frontier France, the veil of Marseille (France), the Paris (France), and the connois of, the capital of Marseau (the veil de lailleseFrance), with + +PROMPT: Water boils at a temperature of +OUT: 0°C and 150°C for one hour. At 160°C, water reaches its boiling point. A boiling point of 12.3°C can be achieved at at least 25% by weight. Only a boiling point of 4.5C can be achieved at 10–25%. 1. The active binding agent, the active binding agent, the binding agent, the active binding + +PROMPT: To make a simple omelette, first +OUT: add a half of it to the boil, then use as an ingredient in salad. Each slice is cooked in about 1/2 cup. Bell International Airport (also: Bell International Airport, L., Bic Air International, Bell International Airport, L., Bell, Inc., Italic International) is an international airline (EU) standard name with ISO No. 41 2158. +The EU standard name is: Bell International Airport + +PROMPT: The main difference between a virus and a bacterium is +OUT: that a virus can be spread, while a thermotolerant bacterium is weakened and invulnerable to viruses, making it less dangerous compared to a viral pathogens like a virus. IBM has served over Fortune 50,000 employees, and the answer may be unclear or unclear, but it is likely that the discussion will focus on the performance and influence of IBM’s most influential executives. In biologicalology, understanding the relationship between an organism and an organism is + +PROMPT: In the morning, the old fisherman walked down to +OUT: the open beach, rediscovering the sense of there— nowhere else to go to rest on the beach that day. The existence of large populations enables the general population, however, to place significant psychological and physiological demands on their health. When populations populations grow large enough, their stress levels malfunction and their ability to manage stress demands effectively. This imbalance results in individuals to experience failure even when facing severe emotional or physical challenges. The tendency to fail stem from elevated stress levels, + +PROMPT: Python is a programming language that +OUT: runss Python on the Python package, re-starts the Python installation process and simplifies the installation process. Python DirectVulia, which is the Python package of the Python installation process and installs DirectVulia, which executes the Python installation process rather than on the Python package, will installs DirectVulia. +- Numus is a Python programming language that also runs on the Python package and executes both the Python installation process and the Python installation. +- + +PROMPT: The three largest planets in the solar system are +OUT: Mars and Jupiter. The Ucladus and Jupiter orbit the solar system, reaching Pluto and returning to the Sun. - h-h-h-h-h-h-h-h-h j. destiny book j. The dest and the techniques of the 1905-06. It is evident, however, that these techniques were similar to those used in the literature. +- s j. destiny j. The + +PROMPT: She opened the letter and read that +OUT: letter again, her hand to her, she and again. She spoke, as long as she squeared, her letter to her. +What does she offer? +- Offers 10 cotton tapes and one pad +- 10 cotton pads, including two roll pads and one cotton pad +- Offers a cotton pad for those who wears a cotton pad +- Offers cotton pad only +- Offers cotton pads + +PROMPT: Before electricity, people lit their homes with +OUT: carbon dioxide or carbon dioxide, marking the burning of coal. The paleontological theory explains why coal has have burned as long as fossil fuels remained un-burned for millennia. Mineral sources such as spikes, pits, and high-speed spikes release trapped dust and gas, often resulting from ingested food. If humans relied on meat, fish, eggs, and vegetables, dust, gas, and debris may have accumulated over time. Over time, + +PROMPT: The reason metals conduct heat well is that +OUT: they attract other metals, such as the metals. The metal’s conductivity means they can conduct heat, and this conductivity is due to the heat stored by it and the temperature of the metal around it. This doesn’t mean that metals conduct heat as well, but the conductivity of the metals (where they conduct heat) is why the metals conduct heat well as well. +If you are stuck in a situation, it is important to seek help right away. At the, + +PROMPT: A good way to learn a new language is to +OUT: take three tests: ESL (ESL test), LSL (Modular language test), and TSL (grammatical language test). If these tests are not,, is it better to take them, in addition, during the learning process. + +When languages are interconnected and their foundational principles have been established, the knowledge is divided to three parts. The first is the knowledge, knowledge, and knowledge. the second is the knowledge. The third is + +PROMPT: The war ended when both sides agreed to +OUT: unify the union and extend the union’s membership to 15,010 members within following year. + +A groundbreaking approach enhances computational capabilities by creating a computationally efficient neural network with nearly the same power level as a classical computer. The process begins by defining a data featuream—a set of data points derived from a large set of values. This featuream is then processed by a a neural network designed to eliminate the need for a higher power, + +PROMPT: My grandmother always said that the secret to +OUT: the maid is the way the milk geese have had their own imbalances with the way in which the Milk and Vegetables have become imbalanced. “….unsl is the way of the imbal problems in the milk geese’s”. So. “….unsl is the way in which the proboscopies are attached to the external surface of the geese.”. It is part of “GFP…GFP- + +PROMPT: In a healthy diet, protein is important because +OUT: it contributes to growth and development, the and it plays an important role in growth and development, in a healthy diet, the body. +In brewer brevée, the (Ferra de la Révoyes de France," is commonly known by one of several regions (Ferra de la), located in the South-Columbia, of Mexico, of the Andayas, of the B Andayas, of the Caribbean, of the And + +PROMPT: The bridge collapsed because the engineers had +OUT: placed a hole in the it. + +Question: What feature, the bridge's width and cross track, help the five-thinking of preschoolers rely on the track to grow up? Answer: The bridge's width and the cross track of the iters help them grow up by providing stability, stability, and balance. In the examples described above, a, b, C, C, not preferably a, b, C, C, as are + +PROMPT: Scientists measure earthquakes using +OUT: seismic data +- Fast Fourier Transform Analysis +- Corointegration: Normal B Correlation and Richter Correlation +- Correlation: Normal B Correlation and Richter Correlation The signal is transmitted to the receiving source through refraction, which is the vibrational generated of the signal itself, and the vibration caused by this vibration, which is responsible for the vibration of from one signal to another. Through this vibration, the signal can be modulated and diff --git a/pilots/gen_gap2.txt b/pilots/gen_gap2.txt new file mode 100644 index 0000000000000000000000000000000000000000..aa235021a2e25f9b564711684a12ef3a50dfcb62 --- /dev/null +++ b/pilots/gen_gap2.txt @@ -0,0 +1,59 @@ +calls=20 for 96 tokens (AR would need 96) + +PROMPT: The capital of France is +OUT: a site of historical continuity, with Grand Vendilla-Muncony, the first 18th-century treaty to recognize the French, in 1723, adopted by the French government as a declaration of sovereignty. The capital, with both the Basque and the Basque, is also under Grand Vendilla-Muncony, Grand Powers of theikhail (modern Paris) and Mikhail, and Grand + +PROMPT: Water boils at a temperature of +OUT: 65°F/1.5°F at a temperature of 13.5°C and evaporation at boiling is 65°F/11.5°F at a boiling point of 65°F/1.5°F. +A mixture of water can be obtained at a concentration of 20% at boiling, 50% at a boiling point of 65°F, 10% + +PROMPT: To make a simple omelette, first +OUT: place it in a water-based pan and let it in medium heat for several minutes, which is known for the omelette recipe. Great Britain Air Airport, Farnes Air Airport, Farnes Fic. Air Airport, and Bees (C.A.). (Farnes) is an air travel facility located at Bell Air Airport in Great Britain, located in the area near the airport. The airport houses several large aircraft, including F F + +PROMPT: The main difference between a virus and a bacterium is +OUT: that a virus can not be bacteria. A highly effective biofilm that is bothoted and piloted targets bacteria itself, making it more likely to quickly evade detection of a virus. IBM began working on piloted targets in 2016, and the company now has redefined its role for IBM, meaning that a company can act on its targets. A virus or invader can only affect a specific organism—such as a specific host organism—while a avert + +PROMPT: In the morning, the old fisherman walked down to +OUT: the theft and rediscovered the ancient map of Scotland, where he placed the fish details on the map, using the digital camera to look up and appreciate the beauty and beauty of the place. Chronic and long-term mental health conditions and substance use disorders are associated with elevated rates of malnutrition across multiple and health domains, which may contribute to the development of individuals with eating disorders characterized by significant mental symptoms and substance use. An individual who transitions from primary care care to a + +PROMPT: Python is a programming language that +OUT: automatically installs runs the Python class, runs, initiates, runs, and execifies the Python installation. RunninggenicVulia is one of the classic install of Python install in Python that installs the Python class, and executes the Python installation noises one of the classic install of Python that in the Python class. RunninggenicVulia is a programming language class that automatically installs the Python class and runs, runs the Python class, and exec the Python installation noises + +PROMPT: The three largest planets in the solar system are +OUT: Uranus and Romulus. Venus and Jupiter are in the sky, around Pluto and orbiting the Sun. "Arts - ruined sins - a paintings in which the history of art seems to be unfolding, meticulously grounding its analysis in today's well-known case studies of art that were not widely recognized at the time. "Arts - the future" paintings of the past, destiny, and the + +PROMPT: She opened the letter and read that +OUT: one in. one red and the other in. one red or sticky stick. None of the guesses in. red or sticky stick. +|Grades 3 & 9: Grade 10: Color Books: Offended for kids who love the colors of the rainbow. Please let us know you can find canned color books for kindergarten and first graders that are red, purple and blue. Offered for toddlers, adults and preschoolers. Also + +PROMPT: Before electricity, people lit their homes with +OUT: carbon dioxide and water at all times. Charcoal was once the paleontological forms of crude oil and coal, but all modern fossil fuels are grain-sized by several millimeters in their fossil fossil counterparts. Mold spores and mildew can survive and penetrate spores and bacteria in drinking water without causing harm to ingested water. The Bayer, utility, plant, plant, plant, and wastewater systems played a crucial role in the early of fuel technology. Thousands + +PROMPT: The reason metals conduct heat well is that +OUT: they have many metals, such as thealt. Along with their many metals, metals also conduct heat in and out. The reason metals heat in and out is that it is the only element conducting heat. This doesn’t mean all metals conduct heat. For metals, the fact that many metals (all metals conduct heat) means that metals do conduct and heat as well. +If you’re reading a table or table, do you find it far better? After all, + +PROMPT: A good way to learn a new language is to +OUT: go through the following: Dutch (Dutch test), Ula (English), Uwe (English test), and Uwe (English test). Once they have been connected, additional learning is expected to take introduced and will be discussed in the next session. + +Once they are reconnected and the foundational knowledge has been established, further learning is expected in the next phase. However, there is no further learning outcome in the next session in which learning is expected will be + +PROMPT: The war ended when both sides agreed to +OUT: unify the union, expanding the union’s membership to nearly 1,000 members the following year. + +A groundbreaking method enhances computational efficiency by introducing a computationally intensive method designed to meet the demands of high-performance parallel processing. The process begins by identifying a known featuream—a collection or set of features from a large, diverse dataset. This featuream is then used to train models and algorithms designed to minimize the need for expensive parallel processing. + +PROMPT: My grandmother always said that the secret to +OUT: sleep sunsl is that the milk one one has had their own set of problems with the rest. My said one she wouldn’t have been able to sleep sure if the milk one had their set of different imbalances with the rest. My grandmother always said: +…. Sleep sunsl is a condition in which the capillaries are are unable to cover the surface of the breast tissue. In women, the combination of CGFD and CGFD + +PROMPT: In a healthy diet, protein is important because +OUT: it provides energy and helps supply the body what it needs for long-term well- health, making daily healthy eating easier and sustainable. Examples include Revina Ma Soup, "Male Soup," Revina Ma, and Revina Ma. It is commonly known as the West Coast, which differs significantly from its counterparts in North America. The West Coast, once a region shaped by centuries of civilization and obedience, is now vast and transformive, with + +PROMPT: The bridge collapsed because the engineers had +OUT: removed the it. + +Question: Compare why. The part of the bridge’s structure supports gravity- and chemical reactions? Which part? Why? Does the bridge allow wells to grow vertically? Answer: Why. Why? Why are the specific parts of the bridge allowing wells to grow vertically while still allowing for gravity- and chemical reactions? In the previously described configuration, a first portion of a 1-2-micron array (MGA) is positioned on the + +PROMPT: Scientists measure earthquakes using +OUT: a combination of these two measures: Shatter (S) detecting information by shatter Correlation and Richness Correlation (S)detecting information by shatter Correlation) . In this way, seismic data is being assimisolated away from being collected on the surface of Earth. Thus, to our knowledge, data collected in earthquakes is what is divided into two parts: data on magnetic field data and on magnetic field data, and data on seismic data diff --git a/pilots/gen_now.txt b/pilots/gen_now.txt new file mode 100644 index 0000000000000000000000000000000000000000..7dd3232062743b584d6fb505570e712466530976 --- /dev/null +++ b/pilots/gen_now.txt @@ -0,0 +1,127 @@ +The capital of France is rich in a wine and rich in beer and whiskey. D'uvre - the common in the society of the "people's" and the age of the - is the common household name of households,, in galais, interpreters, and, finally, in the café de l'Or. The procession the commotion, escorted by - the celibate, is dedicated to the redemption of the spirit of " ""uvre" and the end of "midnight." And, eleventh -, they're "att" to one another and their children, elders and friends." Theyre are at the heart of people's lives, at every time of life. Check out the doctor's, songs, discourses, observances, festivals of the day,, and the world. (We're got links to all of these). + +Water boils at a temperature of about 6 degrees, then turns clear to a liquid as the current travers the cliffs. The electric current, through the water, penetrates a thin layer of grass. Slightly, for several inches on the grass, still green, for several seconds. The grass,, has an uncured green color. The Ashtar or Ashtar Drate is ominous water, its color begins to be several hours after the water source has passed. The water from the Ashtar Drate system can be left on top of a water fertilizer for 10 days. + +To make a simple omelette, first . + +The main difference between a virus and a bacterium is that inoculation of this organism (that is to use virus or animal cells to cause infection or disease.) by the. or reagent treatment (pH of = 10) (e.g. the use of dyes, alcohol, or hydrochloric acid) is the first step in which the spores bind to the set of Receptors: D4, D5P and X (Fig.F. Figure 1)) Theseores are isolated from the surface of all these receptors. + +In the morning, the old fisherman walked down to the sea, evoking the “old age” of medicine, small and large, used by medical professionals. It’s time to end the noise, the acid rain. It’s a time, on-line, the history of the Pacific Northwest from the 970s, of an old son, from the 80s. But the real question: how can we, then, shape the understanding of the world, shaped by the narration of time? It undertakes the task of stepping into a world larger than we be imagined, present in time. The present, or “self,” is co-ordinated, and we engage with it, shaping the future through understanding and judging ourselves based on the present. This world has become a site of deep personal reflection, and a a to questions about globalization, and the pyramidal nature of time. + +Python is a programming language that makes it easier. Explore my comprehensive article on the basics of Python and the Coding Language in Python to support your learning. Additionally, read more articles on Python. I hope that all aspects of this article are helpful, not more so. If you can, consider appreciate your work, and future generations of Python will bring significant value to your development—in contrast, older versions often lack the depth and reliability. In my own experience with Python development, I rated the top-rated version at 10% and had no prior experience. + +The three largest planets in the solar system are ours. by Tumsun, these planets are in a complex three-dimensional ring, designed to be the the0 between the planets’ maximum and minimum moments. The rings are as delicate as the kind of rocks that attach to a planet’s surface—the planets in the system move very slowly. + +She opened the letter and read that my she had said so intensely that, for this reason I have been so intrigued, so all, that these letters have given me a reason to me, so fascinated by the beautiful things of all their different kinds, which I have, and that I, the meaning of the phrase "Beautiful one can see if it enumerates them". . + +Before electricity, people lit their homes with natural gas to draw from gas and sewer the sew into sew water. The was in the power lines often went unnoticed. The small supply of the sun – which can only be used by the photovoltaic panel – in the old power lines has rectified and can be utilised. The electricity grid – in the British Columbia area – has been rectified. It has opened up a new frontier for the world race – [a] new approach to the challenges of storms and floods, from hot to cold weather, and a new miracle that (indeed… – for decades ago was rife with disaster). + +The reason metals conduct heat well is that they should be able to work with the Concentrationrated Thermal Density (CPD), which is very important for poly-lithium as mentioned above, PFF, PDF, RDF, and other PTF materials where resistance to heat is related to the conductivity. At the time we started experimenting with metals, at that time they are very easy to work with. Materials that have good conductivity are used in electronics and electronics because of their good heat conductivity. – Nicholson, W. E. Lind, and P.S. Howe, Creative Commons Attribution. + +A good way to learn a new language is to write down the rules. You read and leg you and understand you too. + +The war ended when both sides agreed to cede troops to prevent British troops from the area. The park, now Wilberton National Park, features towering dandeliers, ancient trees, and the smoke’s aroma—its rich, grass-like aroma, evoking the scent of a sunflower. Steve Preacher stepped out of the park, called in from the fire at Belleuth Park. He expresseded with the scent, the smoke’s scent, and said it evokes the image of a folkloreal forest in the middle of Wilberton National Park. As he reached out at Belleuth, he turned to the station, where a new pilot arrived at the station. Preacher’s father responded by calling out 12th graders who brought themselves back to the park. On the morning, they reported using Belleuth National Park on BBC Radio, illustrating how modern technology can enhance everyday experiences like piloting aircraft. + +My grandmother always said that the secret to being able to make a sense of the” was “the indifference of 1940.” Thanks to his research and experience, we are now part of the world, in which he studies, in which beautiful, prosperous, and, are of interest. Gee, hamster, and leopard, cat, dog, and dog, and their geeks are more than, purposely ruthless, dichotomous primates entice with their very nature, ceases intellectively and very conceivable. Vernal their optimism, sacrifices their very nature, abuses their impotence and makes them morally weak. He tolerates and entice padds and irks their punting and misery, overemphasizes and haunts them, sustains a degree of restraint, stunts and weakens them, kills us, and so uselves. Surely, Ver + +In a healthy diet, protein is important because it helps mosquitoes.. + +The bridge collapsed because the engineers had drilled into it. However, the combination of lithography and nanoscale technology gave them a significant boost in confidence, and no one of them was uncertain—namely whether they’d proceed with the bridge on a much smaller scale. + +Scientists measure earthquakes using seismic pressure waves. + +The capital of France is Octavia. Carthage, the re- of France, is (ed 2) a recollection of a first protector. In the same year, she in France owes her commissary, and her St. Augustine, the eighteen of the twenty-aints, who who spent their lives in Vatica, were presented by the Pope’s in a public lecture at the University of Rome in 1951-6. For that reason, “ the name of God is. The it refers to to the scribe of the letters of the Book of the world.” the name of God is different from other parts of the world. If it is true that his name refers to what has been called “a ‘pseudo’ is written in the space between the letters of his name, “quite in the same way,” says. Religion is different—it is no religion in the world + +Water boils at a temperature of 0° C. or lower. + +To make a simple omelette, first roll the dice. + +The main difference between a virus and a bacterium is the level of host resistance (species). Viruses of nature, which can be classified as either in nature or not, are classified as bacteria rather than viruses. + +In the morning, the old fisherman walked down to Wolson’s boat, where he observed the fish, fish, and whales in the water. It had been watching for a days now, the fisherman was waking, and the fisherman had not been present. At a moment of relief, the fisherman did not see the fish hitch’s presence, but they were still alive. . . dolphin was hunted by him, fisherman, off the East Coast. Fortunately, Wolson,, accompanied by the fisherman, took the hitch away. While the dolphin had been hunted repeatedly over the years, the fisherman appears to have taken over it. Dolphin dout dolphins arrive too late in the Pacific, feeding on them, and dolphin dout dolphins should be found in California as soon as possible, though the presence or absence of a prey can still deter them. Edgard Jacobs, Jr. + +Python is a programming language that is compatible with AI languages and is used to write AI languages. The choice of the language is very important in the development. Therefore, it would be reckless to write an AI language in Python. We can choose whether language, in English, French, or English. The choice depends on the human hand. To change the familiarity of the Python language, we go to the test, and we can see that it can be easier. However, testing the Python language is very difficult. + +The three largest planets in the solar system are Earth. These planets are about 5% and about 10% more. + +She opened the letter and read that sentence. The 1 of 22 letters instead of 6 letters. I do not know if to was a casual one. The same letter as a gda nda was called a mafcken or a sub form of mafcken or mafcken. If you want an allowance to a, nda, or nar, to add a hyphen, or add a, nda, nda, or nella, a gda would called a gda. + +Before electricity, people lit their homes with a wood flame. The wood started burning the wood. Not enough heat to fire, but temperatures are getting warmer, requiring more heating for warmer and more comfortable homes. + +The reason metals conduct heat well is that booths are not exposed to the use of the heat-resistant materials layer (and example), so a grating session in the cold is ideal for a bo session). Heavy training sessions are much more intense, because booths are also lightweight, which means you will be able to do all levels of training at their best. The best booth is when it comes to soft wool, which is soft (which also makes it great for retaining heat); however, dyes are not able to resist overheating. "The most important of all soft wool" is that the warmth makes it ideal for restless sleep, even at night temperatures. + +A good way to learn a new language is to use the language of the target language naturally, there are games and games that will really help. You are also very likely to have a lot of language learning fun. which are all the non-fiction books, videos, quizzes, games and games available. In case of the toughest- or second-language learners learner, the rewards can be hard. + +The war ended when both sides agreed to use coal as an energy source in 1956, p. . ., June 15, and also thatposing pressure on fisheries, including tralers of fish. + +My grandmother always said that the secret to how the discovery of the element helium to diamond was the fact that the weight of the “ “atom” was now being transferred by the atoms and the space between them. But back in our days, diamonds were made from these atoms. The walls of atoms between that the atoms now had “the weight of the atom” and the “, and” is what Tenor calls the “atonic” diamond. + +In a healthy diet, protein is important because leucine is not absorbed by the body in large, unlike proteins from other amino sources, which the study says can cause problems. The authors say you don't really add protein to your child's diet. A small amount of protein. What's how much protein you child should diet? 0.5%. That 1% (out of the way) protein resists digestion. Or a 5% protein too, they say. + +The bridge collapsed because the engineers had failed to provide it in front of it. + +Scientists measure earthquakes using magnetic tomography in the subatomic world. + +The capital of France is indifferent to the French in terms of, culture and culture of its people. This difference is due to its close proximity to French culture. Like French, the French national spoken in Paris is well known, and French is the official language. France is home to about 4,300 people. In France about 5 million live. There are about 2,030 French's in the (countries) materiel. About 30 million live in France. This is because of the large French population. Like most of the countries, France has a large population, the banguard population. The French is the official language. The two variants of "French" are essentially the same national language. + +Water boils at a temperature of approximately0 degrees for several, allowing it to transform into the liquid. Wraping the liquid over a cloth, a man carrying a ball from the streets of Northern Ontario, Canada, captures the ball, where it is repositioned to form the letter Man. The meaning of the letter Man itself is unknown. The name features multiple instances of character C. The series titled C's the Sargentence, specifically Volume 2, Volume part of the same series, is also referenced. A purchase at a reduced price includes various types of electronic devices, including all household items that are in Ontario.. Canada may be allowed to consume up to two glasses of wine, with half of their own consumption permitted under Canadian law. They may ask if she wishes to return home, indicating mutual respect and desire toward the woman. The text references a substance, described as a non- mysterious substance whose physical properties cannot be predicted and depend solely on observation. + +To make a simple omelette, first add all ingredients of basil, rosemary, barbecue, and a pinch of the pepper. + +The main difference between a virus and a bacterium is how they are resistant. The Proceedings of the National Academy of Sciences recently published study, based on influenza strains, has shown that some levels of resistance are much lower enough than others. So, for example, L19L1 is only 0.5% resistant from A to A and A to B. If A-L1 is less than 1% and A-vaccines are resistant, then the virus will not re-enter the virus. + +In the morning, the old fisherman walked down to the station, and then the fisherman paused. The fish and J. Rec. run as the fisherman attaches a wooden berth to the station. McGill was driving away from the station when he saw it. The station’s sewage system is not yet functional, as it currently is difficult to access, and the area is not empty. In reality, the station serves as a remote site, one of the most remote and desolate regions in the United States, where it has been for nearly a year. Most people are familiar with the two unloading—carriage and sewage. While unloading offers greater benefit than either one, it can be used for practical, practical reasons. It is unfeasible in a urban States due to the proximity of a busy to street, or, other cases, the noise of a busy roadway. Go Kelley, if you want the J. Rec. station to go. + +Python is a programming language that includes both the Python module and its abstractions such as the Python library, Python functions, etc. Python is created interactively with two ways: adding a Python module and composing the module itself. Here, users and developers can add a module as minusculely of module (to module, next module, module, last module). By enforcing a Python add-on policy, end-of-line developers can control the behavior of the module, ensuring its stability and ease of use. The most important aspect of Python (sometimes overlooked) that is present is in the module itself, which itself includes components—specs, functions—used to provide interfaces between these components. (see,, subsystemes). These subsystemes, as mentioned above are of paramount importance when considering the limited feedback received by the module itself. Instead of addressing the implications of the recent release of the module on Flames, the focus should + +The three largest planets in the solar system are asteroids (in the Earth's solar system's rovers) and unidentified e.g. "un-observed asteroids." In addition to a few soaring, non-luminous giants, there are estimated 10 on the order of 2-5 e. . . In meteorite rovers the moon-a-the means "little or the " moon." In Earth-s solar system, the moon of Earth is as e-, Earth, south) in the sky e.g. for example, indicating that the solar system orbits the moon. In the 1960s. Unidentified the prefix moon-a-n, came to mean a moon-a-passing the western star of the halo. . James E. Lewis, M.D. is a registered U.S. federal intellectual property attorney in Washington, D. + +She opened the letter and read that book in HPS. + +Before electricity, people lit their homes with a lantern and the street was lit. Gradually, and in the morning there was a Sh of the. P.E., swirling around us. + +The reason metals conduct heat well is that metal has a low melting conductivity compared to solder, and also a high conductivity. Solder is also for electostruction . Other metals have high conductivity, too. Solder metals are strong enough to protect buildings and buildings from radiation and stratosphere . Other metals are high conductivity + +A good way to learn a new language is to read it. Words allude to knowledge, which can only be influenced by language, a common culture, a personal ego, a desire to be read by an interviewer as well as on a news article. + +The war ended when both sides agreed to a non-state peace, and a vote of 30 or1, or of, the end of the war. a peace free from fiscal constraints could be achieved by invoking a new form of government. Among the individuals who opposed Congress was Senator Wilson,, a senator and treasurer, who opposed the war, refused to inform Congress of his intentions. When Senator John Christie, a bank senator, incurred $7 million and $40 million in debt, the New York government imposed a crackdown on his office, labeling it as crazenness, which he described as “the “damn”” scandal, and attributed it to attack its former President Tina A. Grman, who admitted to mourning her. Ultimately, the Senate voted in favor of Congress, rather than the Senate, and Speaker of Congress,, Congressman Jorge Luis Ortiz,, in demanding a significant reduction in federal spending. + +My grandmother always said that the secret to what has happened is to be my son’s fate and destiny. + +In a healthy diet, protein is important because all the micronotrients are needed, for example. + +The bridge collapsed because the engineers had to using the bridge as a workpiece. + +Scientists measure earthquakes using magnetic field. “A small change in magnetic field can cause an earthquake or flood10.” Astronomers are studying how physical and astronomical processes influence the surface of both bodies and terrestrial planets in hot or cold regions of the sky where temperatures are different. Milo’s team has developed a new theory to help predict planetary processes. First, a planet sinks into seawater, forming ice (water, water, methane and snow). The ground can be covered with ice and create storms, such as thunderstorms. + +The capital of France is the “common-place” Paris, nestled in a façade wall, and the French inhabitants are descendants of it. The city is well-known . . It is a long-gotten acquaintance. The German physicist, Wilhelm Hertz, who published his work, visited Berlin, Germany, in the 1920s. Unlike many scientists whose works are well-known in Europe, it was not that Volton who was (clearly) a German author of Raleigh, (the synopsis of liberalism); rather, theism and the ideas of both Strauss and Haber have heavily influenced his work. Indeed, theismism has been the reason for an intellectual crisis in many countries. It continues to have an impact on the intellectual development of many Western countries. Who are the great thinkers of the history of science? (Guess who, do you already know)? + +Water boils at a temperature of 40 degrees, equivalent to the Protection Foot of the player's role in the game. The player bursts causes Water to be shaken into the water, a mythological event often interpreted as celebratory and magical. The key point is that Water, however, is not cooled by the Protection Foot after bursting. Once cool, it transforms into the water to be shaken. When Water reaches multipliers, it appears as water. However, the game—and the rest—is not completed. To complete the game, players must hold Water using a plunger, spoon, cans, sticks, or pellets. If a plunger fails to hold the water, players pour the water into a can, placing paws on it and force it into a can, akin to a river. + +To make a simple omelette, first use two wires. + +The main difference between a virus and a bacterium is the nature. Uses that are not very infectious. Do you find nuisance coming from the garbage? In a lot of cases it is common, but often it is not. Garbage tends to be insoluble, and with more people in them, the chances tend to increase. A combination of factors, heat, and,, due to the size and necromalacity of bacteria, a large number of infected person can develop an illness, especially if his body has exposed him to the bacterium for all his life. + +In the morning, the old fisherman walked down to work in Canson, Michigan. He sat at his doctor’s office. Then, all the time, the conversation stopped. Canson was the one who was in front of him. He went back to work. + +Python is a programming language that you can use to learn a Chinese speaker (Chinese) or English as a native speaker in Beijing, China. + +The three largest planets in the solar system are searching for life more than than the Earth; they are wrapped up around the sun just to protect it. Made of carbon dioxide. More than 1 million people on this planet (and people with) now live on the planet (and beyond). (See non-experts for Specialty, Explosive etc.). + +She opened the letter and read that in 2012, __k_@o_, _k_@os_@o_, __k_@o_ was on a letter to me. + +Before electricity, people lit their homes with a pot of water. Then, they illuminated the water, painted it color, and covered it with paint. Then a paraffin, and then a needles, and molded into pellets, etc. Honda’s? Honda, right? Yes, right. + +The reason metals conduct heat well is that they are smooth, flat, and allow glass surfaces to draw in in, reducing thermal loss. + +A good way to learn a new language is to ask yourself a few questions about yourself.Culture.com + +The war ended when both sides agreed to evacuate the world. The less developed world had to be reindustrialized ( (VII) , which would help to make the situation worse. + +My grandmother always said that the secret to Confucianism was to have a child with and giggle. Don't nay it out, because it isn't always the case; it is a mis-interpretation. + +In a healthy diet, protein is important because it serves as fuel. The high protein snakes are not very low in protein and, due to all the problems associated with them, they are not low in essential nutrients, such as B vitamins. The status of these animals is related to the status, however. Many of these animals are unable to survive the nutritional stress of the carnivorous relying on them. Herbarians and humans are capable of being healthier than the heterotrophic carnivores and do will feed these animals and give them their best. (to read this article). + +The bridge collapsed because the engineers had made their way to the gas gas. But before they could rebuild the bridge, they had used the gas plunge into the bridge. + +Scientists measure earthquakes using very high electrical amplifiers. For example, high-impeded ocean waves trigger tropical storms along Florida’s coast, while cyclones, one between two major shipping routes. diff --git a/pilots/heldout.log b/pilots/heldout.log new file mode 100644 index 0000000000000000000000000000000000000000..26329ebbb2a59e93507acf5825ab9df72846dc15 --- /dev/null +++ b/pilots/heldout.log @@ -0,0 +1,328 @@ +17:26:35 building held-out batch +17:28:29 batch [8, 2048] +17:28:42 extra slice wiki [4, 2048] +17:28:46 extra slice code [4, 2048] +17:29:12 round 757 step 181500 | indep 2.9499 | 5s +17:29:17 EXTRA round 757 step 181500 | wiki 3.5373 code 1.6672 +17:29:34 FACTS round 757 step 181500 | mc 0.5600 nll 5.8928 em - | 16s +17:29:55 EMA round 757 step 181500 n 136 | indep 2.9489 | 5s +17:30:00 EMA EXTRA round 757 step 181500 | wiki 3.5381 code 1.6656 +17:30:28 EMA FACTS round 757 step 181500 | mc 0.5590 nll 5.8936 em 0.0971 | 28s +17:34:50 round 758 step 181650 | indep 2.9500 | 5s +17:34:55 EXTRA round 758 step 181650 | wiki 3.5375 code 1.6651 +17:35:15 FACTS round 758 step 181650 | mc 0.5571 nll 5.8899 em - | 20s +17:35:37 EMA round 758 step 181650 n 137 | indep 2.9490 | 5s +17:35:42 EMA EXTRA round 758 step 181650 | wiki 3.5381 code 1.6654 +17:36:10 EMA FACTS round 758 step 181650 | mc 0.5590 nll 5.8932 em 0.0971 | 28s +17:44:32 round 759 step 181800 | indep 2.9505 | 5s +17:44:37 EXTRA round 759 step 181800 | wiki 3.5385 code 1.6650 +17:44:56 FACTS round 759 step 181800 | mc 0.5562 nll 5.8947 em - | 19s +17:45:16 EMA round 759 step 181800 n 138 | indep 2.9491 | 5s +17:45:21 EMA EXTRA round 759 step 181800 | wiki 3.5381 code 1.6653 +17:45:48 EMA FACTS round 759 step 181800 | mc 0.5590 nll 5.8933 em 0.0971 | 27s +17:54:09 round 760 step 181950 | indep 2.9495 | 5s +17:54:14 EXTRA round 760 step 181950 | wiki 3.5383 code 1.6640 +17:54:33 FACTS round 760 step 181950 | mc 0.5610 nll 5.8939 em - | 19s +17:54:54 EMA round 760 step 181950 n 139 | indep 2.9491 | 5s +17:54:59 EMA EXTRA round 760 step 181950 | wiki 3.5381 code 1.6651 +17:55:27 EMA FACTS round 760 step 181950 | mc 0.5600 nll 5.8933 em 0.0971 | 27s +18:03:48 round 761 step 182100 | indep 2.9497 | 5s +18:03:53 EXTRA round 761 step 182100 | wiki 3.5390 code 1.6650 +18:04:10 FACTS round 761 step 182100 | mc 0.5619 nll 5.8919 em - | 17s +18:04:33 EMA round 761 step 182100 n 140 | indep 2.9492 | 5s +18:04:38 EMA EXTRA round 761 step 182100 | wiki 3.5382 code 1.6649 +18:05:05 EMA FACTS round 761 step 182100 | mc 0.5600 nll 5.8931 em 0.0971 | 27s +18:13:27 round 762 step 182250 | indep 2.9494 | 5s +18:13:32 EXTRA round 762 step 182250 | wiki 3.5397 code 1.6652 +18:13:51 FACTS round 762 step 182250 | mc 0.5619 nll 5.8912 em - | 19s +18:14:14 EMA round 762 step 182250 n 141 | indep 2.9492 | 5s +18:14:19 EMA EXTRA round 762 step 182250 | wiki 3.5383 code 1.6648 +18:14:44 EMA FACTS round 762 step 182250 | mc 0.5600 nll 5.8929 em 0.0971 | 26s +18:25:13 round 763 step 182400 | indep 2.9501 | 5s +18:25:18 EXTRA round 763 step 182400 | wiki 3.5397 code 1.6679 +18:25:34 FACTS round 763 step 182400 | mc 0.5619 nll 5.8931 em - | 15s +18:25:58 EMA round 763 step 182400 n 142 | indep 2.9492 | 5s +18:26:03 EMA EXTRA round 763 step 182400 | wiki 3.5384 code 1.6649 +18:26:28 EMA FACTS round 763 step 182400 | mc 0.5600 nll 5.8929 em 0.0971 | 26s +18:36:54 round 764 step 182550 | indep 2.9509 | 5s +18:36:58 EXTRA round 764 step 182550 | wiki 3.5386 code 1.6641 +18:37:15 FACTS round 764 step 182550 | mc 0.5600 nll 5.8926 em - | 17s +18:37:39 EMA round 764 step 182550 n 143 | indep 2.9494 | 5s +18:37:44 EMA EXTRA round 764 step 182550 | wiki 3.5384 code 1.6647 +18:38:10 EMA FACTS round 764 step 182550 | mc 0.5619 nll 5.8928 em 0.0971 | 27s +18:46:35 round 765 step 182700 | indep 2.9502 | 5s +18:46:40 EXTRA round 765 step 182700 | wiki 3.5379 code 1.6649 +18:46:57 FACTS round 765 step 182700 | mc 0.5590 nll 5.8867 em - | 17s +18:47:19 EMA round 765 step 182700 n 144 | indep 2.9494 | 5s +18:47:24 EMA EXTRA round 765 step 182700 | wiki 3.5383 code 1.6645 +18:47:51 EMA FACTS round 765 step 182700 | mc 0.5610 nll 5.8921 em 0.0971 | 27s +18:56:15 round 766 step 182850 | indep 2.9506 | 5s +18:56:20 EXTRA round 766 step 182850 | wiki 3.5390 code 1.6658 +18:56:38 FACTS round 766 step 182850 | mc 0.5610 nll 5.8927 em - | 18s +18:56:57 EMA round 766 step 182850 n 145 | indep 2.9495 | 5s +18:57:02 EMA EXTRA round 766 step 182850 | wiki 3.5383 code 1.6645 +18:57:29 EMA FACTS round 766 step 182850 | mc 0.5610 nll 5.8922 em 0.0971 | 27s +19:05:53 round 767 step 183000 | indep 2.9499 | 5s +19:05:58 EXTRA round 767 step 183000 | wiki 3.5397 code 1.6630 +19:06:17 FACTS round 767 step 183000 | mc 0.5619 nll 5.8928 em - | 19s +19:06:37 EMA round 767 step 183000 n 146 | indep 2.9495 | 5s +19:06:42 EMA EXTRA round 767 step 183000 | wiki 3.5385 code 1.6641 +19:07:09 EMA FACTS round 767 step 183000 | mc 0.5600 nll 5.8921 em 0.0971 | 27s +19:15:30 round 768 step 183150 | indep 2.9498 | 5s +19:15:35 EXTRA round 768 step 183150 | wiki 3.5393 code 1.6645 +19:15:51 FACTS round 768 step 183150 | mc 0.5610 nll 5.8929 em - | 17s +19:16:11 EMA round 768 step 183150 n 147 | indep 2.9495 | 5s +19:16:15 EMA EXTRA round 768 step 183150 | wiki 3.5385 code 1.6640 +19:16:42 EMA FACTS round 768 step 183150 | mc 0.5600 nll 5.8922 em 0.0971 | 26s +19:25:06 round 769 step 183300 | indep 2.9491 | 5s +19:25:10 EXTRA round 769 step 183300 | wiki 3.5380 code 1.6637 +19:25:28 FACTS round 769 step 183300 | mc 0.5600 nll 5.8919 em - | 17s +19:25:49 EMA round 769 step 183300 n 148 | indep 2.9495 | 5s +19:25:54 EMA EXTRA round 769 step 183300 | wiki 3.5384 code 1.6638 +19:26:21 EMA FACTS round 769 step 183300 | mc 0.5600 nll 5.8921 em 0.0971 | 28s +19:34:47 round 770 step 183450 | indep 2.9497 | 5s +19:34:52 EXTRA round 770 step 183450 | wiki 3.5376 code 1.6661 +19:35:11 FACTS round 770 step 183450 | mc 0.5610 nll 5.8912 em - | 19s +19:35:35 EMA round 770 step 183450 n 149 | indep 2.9495 | 5s +19:35:39 EMA EXTRA round 770 step 183450 | wiki 3.5383 code 1.6639 +19:36:06 EMA FACTS round 770 step 183450 | mc 0.5610 nll 5.8919 em 0.0971 | 27s +19:46:27 round 771 step 183600 | indep 2.9502 | 5s +19:46:31 EXTRA round 771 step 183600 | wiki 3.5378 code 1.6644 +19:46:49 FACTS round 771 step 183600 | mc 0.5590 nll 5.8920 em - | 18s +19:47:08 EMA round 771 step 183600 n 150 | indep 2.9495 | 5s +19:47:13 EMA EXTRA round 771 step 183600 | wiki 3.5383 code 1.6638 +19:47:40 EMA FACTS round 771 step 183600 | mc 0.5600 nll 5.8919 em 0.0971 | 27s +19:58:02 round 772 step 183750 | indep 2.9500 | 5s +19:58:07 EXTRA round 772 step 183750 | wiki 3.5388 code 1.6646 +19:58:25 FACTS round 772 step 183750 | mc 0.5581 nll 5.8966 em - | 18s +19:58:46 EMA round 772 step 183750 n 151 | indep 2.9495 | 5s +19:58:51 EMA EXTRA round 772 step 183750 | wiki 3.5383 code 1.6637 +19:59:17 EMA FACTS round 772 step 183750 | mc 0.5600 nll 5.8923 em 0.0971 | 26s +20:09:39 round 773 step 183900 | indep 2.9500 | 5s +20:09:43 EXTRA round 773 step 183900 | wiki 3.5381 code 1.6624 +20:10:02 FACTS round 773 step 183900 | mc 0.5600 nll 5.8908 em - | 18s +20:10:23 EMA round 773 step 183900 n 152 | indep 2.9496 | 5s +20:10:28 EMA EXTRA round 773 step 183900 | wiki 3.5383 code 1.6635 +20:10:57 EMA FACTS round 773 step 183900 | mc 0.5600 nll 5.8922 em 0.0971 | 29s +20:19:18 round 774 step 184050 | indep 2.9498 | 5s +20:19:22 EXTRA round 774 step 184050 | wiki 3.5376 code 1.6634 +20:19:41 FACTS round 774 step 184050 | mc 0.5571 nll 5.8931 em - | 18s +20:20:04 EMA round 774 step 184050 n 153 | indep 2.9496 | 5s +20:20:08 EMA EXTRA round 774 step 184050 | wiki 3.5382 code 1.6634 +20:20:35 EMA FACTS round 774 step 184050 | mc 0.5590 nll 5.8923 em 0.0971 | 27s +20:29:02 round 775 step 184200 | indep 2.9500 | 5s +20:29:06 EXTRA round 775 step 184200 | wiki 3.5381 code 1.6629 +20:29:21 FACTS round 775 step 184200 | mc 0.5581 nll 5.8927 em - | 15s +20:29:43 EMA round 775 step 184200 n 154 | indep 2.9496 | 5s +20:29:48 EMA EXTRA round 775 step 184200 | wiki 3.5382 code 1.6632 +20:30:14 EMA FACTS round 775 step 184200 | mc 0.5581 nll 5.8923 em 0.0971 | 26s +20:38:38 round 776 step 184350 | indep 2.9495 | 5s +20:38:43 EXTRA round 776 step 184350 | wiki 3.5389 code 1.6645 +20:39:02 FACTS round 776 step 184350 | mc 0.5581 nll 5.8918 em - | 19s +20:39:22 EMA round 776 step 184350 n 155 | indep 2.9496 | 5s +20:39:27 EMA EXTRA round 776 step 184350 | wiki 3.5382 code 1.6632 +20:39:54 EMA FACTS round 776 step 184350 | mc 0.5581 nll 5.8922 em 0.0971 | 27s +20:48:12 round 777 step 184500 | indep 2.9498 | 2s +20:48:14 EXTRA round 777 step 184500 | wiki 3.5389 code 1.6643 +20:48:24 FACTS round 777 step 184500 | mc 0.5600 nll 5.8885 em - | 10s +20:50:46 EMA round 777 step 184500 n 156 | indep 2.9496 | 5s +20:50:51 EMA EXTRA round 777 step 184500 | wiki 3.5383 code 1.6632 +20:51:20 EMA FACTS round 777 step 184500 | mc 0.5590 nll 5.8918 em 0.0971 | 29s +20:59:40 round 778 step 184650 | indep 2.9499 | 5s +20:59:45 EXTRA round 778 step 184650 | wiki 3.5386 code 1.6632 +21:00:03 FACTS round 778 step 184650 | mc 0.5610 nll 5.8933 em - | 18s +21:00:24 EMA round 778 step 184650 n 157 | indep 2.9496 | 5s +21:00:28 EMA EXTRA round 778 step 184650 | wiki 3.5383 code 1.6632 +21:00:55 EMA FACTS round 778 step 184650 | mc 0.5581 nll 5.8920 em 0.0971 | 27s +21:09:18 round 779 step 184800 | indep 2.9500 | 5s +21:09:23 EXTRA round 779 step 184800 | wiki 3.5393 code 1.6639 +21:09:41 FACTS round 779 step 184800 | mc 0.5590 nll 5.8919 em - | 18s +21:10:00 EMA round 779 step 184800 n 158 | indep 2.9496 | 5s +21:10:05 EMA EXTRA round 779 step 184800 | wiki 3.5384 code 1.6631 +21:10:31 EMA FACTS round 779 step 184800 | mc 0.5581 nll 5.8920 em 0.0971 | 26s +21:18:51 round 780 step 184950 | indep 2.9502 | 5s +21:18:56 EXTRA round 780 step 184950 | wiki 3.5393 code 1.6648 +21:19:13 FACTS round 780 step 184950 | mc 0.5590 nll 5.8918 em - | 17s +21:19:35 EMA round 780 step 184950 n 159 | indep 2.9497 | 5s +21:19:40 EMA EXTRA round 780 step 184950 | wiki 3.5385 code 1.6632 +21:20:08 EMA FACTS round 780 step 184950 | mc 0.5590 nll 5.8918 em 0.0971 | 28s +21:28:28 round 781 step 185100 | indep 2.9508 | 5s +21:28:33 EXTRA round 781 step 185100 | wiki 3.5392 code 1.6651 +21:28:49 FACTS round 781 step 185100 | mc 0.5629 nll 5.8929 em - | 16s +21:29:11 EMA round 781 step 185100 n 160 | indep 2.9498 | 5s +21:29:16 EMA EXTRA round 781 step 185100 | wiki 3.5385 code 1.6634 +21:29:41 EMA FACTS round 781 step 185100 | mc 0.5590 nll 5.8920 em 0.0971 | 26s +21:38:07 round 782 step 185250 | indep 2.9499 | 5s +21:38:12 EXTRA round 782 step 185250 | wiki 3.5393 code 1.6624 +21:38:29 FACTS round 782 step 185250 | mc 0.5638 nll 5.8923 em - | 17s +21:38:49 EMA round 782 step 185250 n 161 | indep 2.9498 | 5s +21:38:54 EMA EXTRA round 782 step 185250 | wiki 3.5386 code 1.6632 +21:39:21 EMA FACTS round 782 step 185250 | mc 0.5590 nll 5.8920 em 0.0971 | 27s +21:47:41 round 783 step 185400 | indep 2.9502 | 5s +21:47:46 EXTRA round 783 step 185400 | wiki 3.5395 code 1.6633 +21:48:04 FACTS round 783 step 185400 | mc 0.5638 nll 5.8874 em - | 18s +21:48:26 EMA round 783 step 185400 n 162 | indep 2.9498 | 5s +21:48:31 EMA EXTRA round 783 step 185400 | wiki 3.5387 code 1.6631 +21:48:56 EMA FACTS round 783 step 185400 | mc 0.5590 nll 5.8916 em 0.0971 | 26s +21:57:17 round 784 step 185550 | indep 2.9503 | 5s +21:57:21 EXTRA round 784 step 185550 | wiki 3.5393 code 1.6626 +21:57:39 FACTS round 784 step 185550 | mc 0.5657 nll 5.8880 em - | 18s +21:57:59 EMA round 784 step 185550 n 163 | indep 2.9498 | 5s +21:58:04 EMA EXTRA round 784 step 185550 | wiki 3.5387 code 1.6630 +21:58:30 EMA FACTS round 784 step 185550 | mc 0.5590 nll 5.8911 em 0.0971 | 27s +22:06:55 round 785 step 185700 | indep 2.9500 | 5s +22:07:00 EXTRA round 785 step 185700 | wiki 3.5390 code 1.6606 +22:07:17 FACTS round 785 step 185700 | mc 0.5648 nll 5.8909 em - | 17s +22:07:41 EMA round 785 step 185700 n 164 | indep 2.9498 | 5s +22:07:46 EMA EXTRA round 785 step 185700 | wiki 3.5387 code 1.6627 +22:08:14 EMA FACTS round 785 step 185700 | mc 0.5610 nll 5.8911 em 0.0971 | 28s +22:16:37 round 786 step 185850 | indep 2.9499 | 5s +22:16:42 EXTRA round 786 step 185850 | wiki 3.5386 code 1.6617 +22:17:01 FACTS round 786 step 185850 | mc 0.5638 nll 5.8911 em - | 19s +22:17:19 EMA round 786 step 185850 n 165 | indep 2.9498 | 5s +22:17:24 EMA EXTRA round 786 step 185850 | wiki 3.5387 code 1.6625 +22:17:49 EMA FACTS round 786 step 185850 | mc 0.5600 nll 5.8911 em 0.0971 | 25s +22:26:14 round 787 step 186000 | indep 2.9500 | 5s +22:26:19 EXTRA round 787 step 186000 | wiki 3.5387 code 1.6621 +22:26:38 FACTS round 787 step 186000 | mc 0.5638 nll 5.8922 em - | 19s +22:27:00 EMA round 787 step 186000 n 166 | indep 2.9498 | 5s +22:27:04 EMA EXTRA round 787 step 186000 | wiki 3.5387 code 1.6624 +22:27:32 EMA FACTS round 787 step 186000 | mc 0.5610 nll 5.8911 em 0.0971 | 27s +22:41:54 round 788 step 186150 | indep 2.9499 | 5s +22:41:59 EXTRA round 788 step 186150 | wiki 3.5382 code 1.6654 +22:42:15 FACTS round 788 step 186150 | mc 0.5629 nll 5.8950 em - | 17s +22:42:36 EMA round 788 step 186150 n 167 | indep 2.9498 | 5s +22:42:41 EMA EXTRA round 788 step 186150 | wiki 3.5386 code 1.6626 +22:43:09 EMA FACTS round 788 step 186150 | mc 0.5610 nll 5.8915 em 0.0971 | 28s +22:53:31 round 789 step 186300 | indep 2.9499 | 5s +22:53:36 EXTRA round 789 step 186300 | wiki 3.5393 code 1.6637 +22:53:55 FACTS round 789 step 186300 | mc 0.5629 nll 5.8872 em - | 19s +22:54:16 EMA round 789 step 186300 n 168 | indep 2.9498 | 5s +22:54:21 EMA EXTRA round 789 step 186300 | wiki 3.5387 code 1.6626 +22:54:48 EMA FACTS round 789 step 186300 | mc 0.5619 nll 5.8910 em 0.0971 | 27s +23:05:15 round 790 step 186450 | indep 2.9502 | 5s +23:05:20 EXTRA round 790 step 186450 | wiki 3.5380 code 1.6640 +23:05:39 FACTS round 790 step 186450 | mc 0.5667 nll 5.8951 em - | 19s +23:05:59 EMA round 790 step 186450 n 169 | indep 2.9498 | 5s +23:06:04 EMA EXTRA round 790 step 186450 | wiki 3.5386 code 1.6626 +23:06:31 EMA FACTS round 790 step 186450 | mc 0.5619 nll 5.8914 em 0.0971 | 27s +23:16:53 round 791 step 186600 | indep 2.9507 | 5s +23:16:58 EXTRA round 791 step 186600 | wiki 3.5387 code 1.6641 +23:17:16 FACTS round 791 step 186600 | mc 0.5638 nll 5.8970 em - | 18s +23:17:36 EMA round 791 step 186600 n 170 | indep 2.9499 | 5s +23:17:41 EMA EXTRA round 791 step 186600 | wiki 3.5386 code 1.6626 +23:18:06 EMA FACTS round 791 step 186600 | mc 0.5619 nll 5.8919 em 0.0971 | 25s +23:28:26 round 792 step 186750 | indep 2.9501 | 5s +23:28:31 EXTRA round 792 step 186750 | wiki 3.5377 code 1.6669 +23:28:50 FACTS round 792 step 186750 | mc 0.5629 nll 5.8952 em - | 20s +23:29:12 EMA round 792 step 186750 n 171 | indep 2.9499 | 5s +23:29:16 EMA EXTRA round 792 step 186750 | wiki 3.5384 code 1.6629 +23:29:44 EMA FACTS round 792 step 186750 | mc 0.5629 nll 5.8922 em 0.0971 | 28s +23:40:04 round 793 step 186900 | indep 2.9500 | 5s +23:40:09 EXTRA round 793 step 186900 | wiki 3.5382 code 1.6649 +23:40:27 FACTS round 793 step 186900 | mc 0.5619 nll 5.8934 em - | 18s +23:40:48 EMA round 793 step 186900 n 172 | indep 2.9499 | 5s +23:40:53 EMA EXTRA round 793 step 186900 | wiki 3.5384 code 1.6629 +23:41:18 EMA FACTS round 793 step 186900 | mc 0.5638 nll 5.8922 em 0.0971 | 25s +23:51:41 round 794 step 187050 | indep 2.9495 | 5s +23:51:46 EXTRA round 794 step 187050 | wiki 3.5377 code 1.6624 +23:52:04 FACTS round 794 step 187050 | mc 0.5648 nll 5.8932 em - | 18s +23:52:26 EMA round 794 step 187050 n 173 | indep 2.9498 | 5s +23:52:31 EMA EXTRA round 794 step 187050 | wiki 3.5383 code 1.6627 +23:52:56 EMA FACTS round 794 step 187050 | mc 0.5638 nll 5.8922 em 0.0971 | 26s +00:03:22 round 795 step 187200 | indep 2.9503 | 5s +00:03:27 EXTRA round 795 step 187200 | wiki 3.5397 code 1.6642 +00:03:42 FACTS round 795 step 187200 | mc 0.5638 nll 5.8976 em - | 15s +00:04:04 EMA round 795 step 187200 n 174 | indep 2.9498 | 5s +00:04:09 EMA EXTRA round 795 step 187200 | wiki 3.5384 code 1.6627 +00:04:36 EMA FACTS round 795 step 187200 | mc 0.5638 nll 5.8928 em 0.0971 | 27s +00:15:00 round 796 step 187350 | indep 2.9500 | 5s +00:15:05 EXTRA round 796 step 187350 | wiki 3.5397 code 1.6671 +00:15:22 FACTS round 796 step 187350 | mc 0.5638 nll 5.9003 em - | 17s +00:15:43 EMA round 796 step 187350 n 175 | indep 2.9498 | 5s +00:15:47 EMA EXTRA round 796 step 187350 | wiki 3.5385 code 1.6630 +00:16:14 EMA FACTS round 796 step 187350 | mc 0.5638 nll 5.8935 em 0.0971 | 27s +00:32:38 round 797 step 187500 | indep 2.9501 | 5s +00:32:42 EXTRA round 797 step 187500 | wiki 3.5388 code 1.6661 +00:32:59 FACTS round 797 step 187500 | mc 0.5619 nll 5.9014 em - | 16s +00:33:28 EMA round 797 step 187500 n 176 | indep 2.9498 | 5s +00:33:33 EMA EXTRA round 797 step 187500 | wiki 3.5385 code 1.6631 +00:34:00 EMA FACTS round 797 step 187500 | mc 0.5638 nll 5.8942 em 0.0971 | 27s +00:44:25 round 798 step 187650 | indep 2.9507 | 5s +00:44:29 EXTRA round 798 step 187650 | wiki 3.5396 code 1.6657 +00:44:42 FACTS round 798 step 187650 | mc 0.5638 nll 5.8923 em - | 13s +00:45:03 EMA round 798 step 187650 n 177 | indep 2.9499 | 5s +00:45:07 EMA EXTRA round 798 step 187650 | wiki 3.5386 code 1.6631 +00:45:33 EMA FACTS round 798 step 187650 | mc 0.5638 nll 5.8939 em 0.0971 | 26s +00:56:00 round 799 step 187800 | indep 2.9505 | 5s +00:56:04 EXTRA round 799 step 187800 | wiki 3.5390 code 1.6665 +00:56:20 FACTS round 799 step 187800 | mc 0.5610 nll 5.8864 em - | 16s +00:56:43 EMA round 799 step 187800 n 178 | indep 2.9499 | 5s +00:56:48 EMA EXTRA round 799 step 187800 | wiki 3.5386 code 1.6632 +00:57:15 EMA FACTS round 799 step 187800 | mc 0.5629 nll 5.8930 em 0.0971 | 27s +01:07:46 round 800 step 187950 | indep 2.9511 | 5s +01:07:51 EXTRA round 800 step 187950 | wiki 3.5388 code 1.6648 +01:08:06 FACTS round 800 step 187950 | mc 0.5638 nll 5.8844 em - | 15s +01:08:29 EMA round 800 step 187950 n 179 | indep 2.9500 | 5s +01:08:34 EMA EXTRA round 800 step 187950 | wiki 3.5386 code 1.6631 +01:08:58 EMA FACTS round 800 step 187950 | mc 0.5629 nll 5.8921 em 0.0971 | 24s +01:19:25 round 801 step 188100 | indep 2.9507 | 5s +01:19:30 EXTRA round 801 step 188100 | wiki 3.5407 code 1.6659 +01:19:32 FACTS round 801 step 188100 | mc 0.5610 nll 5.8833 em - | 2s +01:20:11 EMA round 801 step 188100 n 180 | indep 2.9500 | 5s +01:20:15 EMA EXTRA round 801 step 188100 | wiki 3.5387 code 1.6632 +01:20:42 EMA FACTS round 801 step 188100 | mc 0.5629 nll 5.8911 em 0.0971 | 27s +01:33:05 round 802 step 188250 | indep 2.9516 | 5s +01:33:10 EXTRA round 802 step 188250 | wiki 3.5414 code 1.6691 +01:33:26 FACTS round 802 step 188250 | mc 0.5638 nll 5.8842 em - | 16s +01:33:50 EMA round 802 step 188250 n 181 | indep 2.9501 | 5s +01:33:55 EMA EXTRA round 802 step 188250 | wiki 3.5389 code 1.6635 +01:34:20 EMA FACTS round 802 step 188250 | mc 0.5619 nll 5.8903 em 0.0962 | 26s +01:44:43 round 803 step 188400 | indep 2.9510 | 5s +01:44:47 EXTRA round 803 step 188400 | wiki 3.5405 code 1.6646 +01:45:07 FACTS round 803 step 188400 | mc 0.5581 nll 5.8880 em - | 20s +01:45:30 EMA round 803 step 188400 n 182 | indep 2.9501 | 5s +01:45:34 EMA EXTRA round 803 step 188400 | wiki 3.5391 code 1.6634 +01:46:03 EMA FACTS round 803 step 188400 | mc 0.5619 nll 5.8900 em 0.0962 | 29s +01:56:27 round 804 step 188550 | indep 2.9505 | 5s +01:56:32 EXTRA round 804 step 188550 | wiki 3.5415 code 1.6695 +01:56:51 FACTS round 804 step 188550 | mc 0.5629 nll 5.8952 em - | 19s +01:57:11 EMA round 804 step 188550 n 183 | indep 2.9501 | 5s +01:57:16 EMA EXTRA round 804 step 188550 | wiki 3.5393 code 1.6637 +01:57:44 EMA FACTS round 804 step 188550 | mc 0.5629 nll 5.8904 em 0.0962 | 28s +02:08:09 round 805 step 188700 | indep 2.9512 | 5s +02:08:14 EXTRA round 805 step 188700 | wiki 3.5421 code 1.6650 +02:08:29 FACTS round 805 step 188700 | mc 0.5629 nll 5.8856 em - | 16s +02:08:51 EMA round 805 step 188700 n 184 | indep 2.9502 | 5s +02:08:55 EMA EXTRA round 805 step 188700 | wiki 3.5395 code 1.6636 +02:09:21 EMA FACTS round 805 step 188700 | mc 0.5629 nll 5.8898 em 0.0962 | 26s +02:19:52 round 806 step 188850 | indep 2.9507 | 5s +02:19:57 EXTRA round 806 step 188850 | wiki 3.5399 code 1.6639 +02:20:12 FACTS round 806 step 188850 | mc 0.5648 nll 5.8916 em - | 15s +02:20:34 EMA round 806 step 188850 n 185 | indep 2.9502 | 5s +02:20:38 EMA EXTRA round 806 step 188850 | wiki 3.5395 code 1.6635 +02:21:04 EMA FACTS round 806 step 188850 | mc 0.5629 nll 5.8900 em 0.0962 | 25s +02:31:31 round 807 step 189000 | indep 2.9504 | 5s +02:31:36 EXTRA round 807 step 189000 | wiki 3.5396 code 1.6652 +02:31:51 FACTS round 807 step 189000 | mc 0.5610 nll 5.8980 em - | 15s +02:32:12 EMA round 807 step 189000 n 186 | indep 2.9502 | 5s +02:32:17 EMA EXTRA round 807 step 189000 | wiki 3.5395 code 1.6634 +02:32:42 EMA FACTS round 807 step 189000 | mc 0.5629 nll 5.8907 em 0.0962 | 26s +02:43:07 round 808 step 189150 | indep 2.9506 | 5s +02:43:11 EXTRA round 808 step 189150 | wiki 3.5406 code 1.6674 +02:43:27 FACTS round 808 step 189150 | mc 0.5600 nll 5.8931 em - | 16s +02:43:49 EMA round 808 step 189150 n 187 | indep 2.9502 | 5s +02:43:54 EMA EXTRA round 808 step 189150 | wiki 3.5396 code 1.6637 +02:44:22 EMA FACTS round 808 step 189150 | mc 0.5629 nll 5.8909 em 0.0962 | 28s +02:54:54 round 809 step 189300 | indep 2.9510 | 5s +02:54:59 EXTRA round 809 step 189300 | wiki 3.5410 code 1.6658 +02:55:19 FACTS round 809 step 189300 | mc 0.5610 nll 5.8952 em - | 19s +02:55:40 EMA round 809 step 189300 n 188 | indep 2.9503 | 5s +02:55:45 EMA EXTRA round 809 step 189300 | wiki 3.5397 code 1.6637 +02:56:11 EMA FACTS round 809 step 189300 | mc 0.5629 nll 5.8912 em 0.0962 | 26s +03:06:41 round 810 step 189450 | indep 2.9521 | 5s +03:06:46 EXTRA round 810 step 189450 | wiki 3.5416 code 1.6665 +03:07:05 FACTS round 810 step 189450 | mc 0.5619 nll 5.8907 em - | 19s +03:07:26 EMA round 810 step 189450 n 189 | indep 2.9504 | 5s +03:07:31 EMA EXTRA round 810 step 189450 | wiki 3.5398 code 1.6638 +03:07:58 EMA FACTS round 810 step 189450 | mc 0.5629 nll 5.8911 em 0.0962 | 27s diff --git a/pilots/indep_1b.json b/pilots/indep_1b.json new file mode 100644 index 0000000000000000000000000000000000000000..8c6a1fdc8612ca33c6f9fb98d1eee14b53376f73 --- /dev/null +++ b/pilots/indep_1b.json @@ -0,0 +1,82 @@ +{ + "t0.9_K4_conf": [ + 0.1537, + 0.065, + 3.713 + ], + "t0.9_K4_spaced2": [ + 0.1455, + 0.0659, + 4.305 + ], + "t0.9_K4_vspaced2": [ + 0.1446, + 0.0669, + 4.446 + ], + "t0.9_K4_random": [ + 0.1326, + 0.056, + 4.369 + ], + "t0.9_K8_conf": [ + 0.1463, + 0.0668, + 4.288 + ], + "t0.9_K8_spaced2": [ + 0.1419, + 0.0682, + 4.733 + ], + "t0.9_K8_vspaced2": [ + 0.1401, + 0.0671, + 4.805 + ], + "t0.9_K8_random": [ + 0.1272, + 0.0573, + 4.894 + ], + "t0.9_K16_conf": [ + 0.1413, + 0.0674, + 4.687 + ], + "t0.9_K16_spaced2": [ + 0.1421, + 0.0699, + 4.823 + ], + "t0.9_K16_vspaced2": [ + 0.1409, + 0.0687, + 4.85 + ], + "t0.9_K16_random": [ + 0.1213, + 0.0546, + 5.268 + ], + "t0.9_K32_conf": [ + 0.1401, + 0.0684, + 4.844 + ], + "t0.9_K32_spaced2": [ + 0.1381, + 0.0686, + 5.042 + ], + "t0.9_K32_vspaced2": [ + 0.1399, + 0.0697, + 4.978 + ], + "t0.9_K32_random": [ + 0.1209, + 0.0556, + 5.377 + ] +} \ No newline at end of file diff --git a/pilots/indep_1b.log b/pilots/indep_1b.log new file mode 100644 index 0000000000000000000000000000000000000000..b30332598297d71269726c829a920b5217251cc7 --- /dev/null +++ b/pilots/indep_1b.log @@ -0,0 +1,19 @@ +01:38:31 === 1B global_b.pt round=28, 96 rows of 1024 === +01:40:29 mask rate 0.9 +01:40:34 K=4 conf tok=0.1537 pair=0.0650 joint/indep=3.713 +01:40:43 K=4 spaced2 tok=0.1455 pair=0.0659 joint/indep=4.305 +01:40:46 K=4 vspaced2 tok=0.1446 pair=0.0669 joint/indep=4.446 +01:40:50 K=4 random tok=0.1326 pair=0.0560 joint/indep=4.369 +01:40:58 K=8 conf tok=0.1463 pair=0.0668 joint/indep=4.288 +01:41:10 K=8 spaced2 tok=0.1419 pair=0.0682 joint/indep=4.733 +01:41:17 K=8 vspaced2 tok=0.1401 pair=0.0671 joint/indep=4.805 +01:41:25 K=8 random tok=0.1272 pair=0.0573 joint/indep=4.894 +01:41:40 K=16 conf tok=0.1413 pair=0.0674 joint/indep=4.687 +01:41:59 K=16 spaced2 tok=0.1421 pair=0.0699 joint/indep=4.823 +01:42:15 K=16 vspaced2 tok=0.1409 pair=0.0687 joint/indep=4.850 +01:42:30 K=16 random tok=0.1213 pair=0.0546 joint/indep=5.268 +01:43:00 K=32 conf tok=0.1401 pair=0.0684 joint/indep=4.844 +01:43:35 K=32 spaced2 tok=0.1381 pair=0.0686 joint/indep=5.042 +01:44:05 K=32 vspaced2 tok=0.1399 pair=0.0697 joint/indep=4.978 +01:44:36 K=32 random tok=0.1209 pair=0.0556 joint/indep=5.377 +01:44:36 published pilots/indep_1b.json diff --git a/pilots/judge_now.log b/pilots/judge_now.log new file mode 100644 index 0000000000000000000000000000000000000000..59576766ef2109fe1246c8df0bd7aff4de33bfd7 --- /dev/null +++ b/pilots/judge_now.log @@ -0,0 +1,9 @@ +16:11:26 seed=1 -> 16 samples +16:16:11 seed=2 -> 32 samples +16:20:52 seed=3 -> 48 samples +16:25:33 seed=4 -> 64 samples +16:25:33 text ready: 24020 chars +16:25:33 downloading judge runtime +16:25:37 downloading judge model +16:26:01 scoring on cpu +16:26:40 words=4177 doubled=0.263% rep-bigram=0.024% judge PPL 44.2074 +/- 2.89947 (was 35.10 +/- 2.09 at 6.4B) diff --git a/pilots/muonsw_pilot.json b/pilots/muonsw_pilot.json new file mode 100644 index 0000000000000000000000000000000000000000..e5042a9bcd16c4289bd4a40c7e84b6b2eb446ce4 --- /dev/null +++ b/pilots/muonsw_pilot.json @@ -0,0 +1,42 @@ +[ + { + "ce@0.3": 2.54304838180542, + "ce@0.6": 3.4121341705322266, + "pair": 0.16214335421016005, + "tok": 0.3898305084745763, + "ce@0.9": 5.0051655769348145, + "ce": 3.653449376424154, + "tag": "adamw", + "secs": 120.27560624699981 + }, + { + "ce@0.3": 2.5504543781280518, + "ce@0.6": 3.423912525177002, + "pair": 0.16109951287404314, + "tok": 0.38757062146892657, + "ce@0.9": 5.011563777923584, + "ce": 3.6619768937428794, + "tag": "muon0", + "secs": 161.68582208499993 + }, + { + "ce@0.3": 2.550332546234131, + "ce@0.6": 3.4235827922821045, + "pair": 0.1616214335421016, + "tok": 0.3881869542886492, + "ce@0.9": 5.011637210845947, + "ce": 3.661850849787394, + "tag": "muonw", + "secs": 161.75728613000047 + }, + { + "ce@0.3": 2.6604278087615967, + "ce@0.6": 3.5204432010650635, + "pair": 0.1440501043841336, + "tok": 0.37010785824345144, + "ce@0.9": 5.053725719451904, + "ce": 3.744865576426188, + "tag": "muon0_s01", + "secs": 152.3354180769993 + } +] \ No newline at end of file diff --git a/pilots/muonsw_pilot.log b/pilots/muonsw_pilot.log new file mode 100644 index 0000000000000000000000000000000000000000..e560ea514350c9180a1e01c4f9c7e78abfe1d175 --- /dev/null +++ b/pilots/muonsw_pilot.log @@ -0,0 +1,11 @@ +19:49:29 muon switch pilot: 4000 steps (switch at 2000), bs 64, lr 3e-04 +19:55:06 at switch ce 3.8762 +19:57:06 adamw ce 3.6534 | 0.3:2.5430 0.6:3.4121 0.9:5.0052 | pair 0.1621 tok 0.3898 | 120s +19:59:48 muon0 ce 3.6620 | 0.3:2.5505 0.6:3.4239 0.9:5.0116 | pair 0.1611 tok 0.3876 | 162s +20:02:30 muonw ce 3.6619 | 0.3:2.5503 0.6:3.4236 0.9:5.0116 | pair 0.1616 tok 0.3882 | 162s +20:05:03 muon0_s01 ce 3.7449 | 0.3:2.6604 0.6:3.5204 0.9:5.0537 | pair 0.1441 tok 0.3701 | 152s +20:05:03 --- versus adamw after the switch --- +20:05:03 adamw dCE +0.0000 dpair +0.0000 +20:05:03 muon0 dCE +0.0085 dpair -0.0010 +20:05:03 muonw dCE +0.0084 dpair -0.0005 +20:05:03 muon0_s01 dCE +0.0914 dpair -0.0181 diff --git a/pilots/objective.json b/pilots/objective.json new file mode 100644 index 0000000000000000000000000000000000000000..59421dcd9047b79f5e72d3d8b546fa8c0ceab987 --- /dev/null +++ b/pilots/objective.json @@ -0,0 +1,98 @@ +{ + "base_t0.5_K4": [ + 0.5338, + 0.3032 + ], + "base_t0.5_K8": [ + 0.5358, + 0.3135 + ], + "base_t0.5_K16": [ + 0.5335, + 0.3147 + ], + "base_t0.9_K4": [ + 0.1697, + 0.0663 + ], + "base_t0.9_K8": [ + 0.1623, + 0.0693 + ], + "base_t0.9_K16": [ + 0.1577, + 0.0707 + ], + "span_t0.5_K4": [ + 0.5325, + 0.3069 + ], + "span_t0.5_K8": [ + 0.5362, + 0.3198 + ], + "span_t0.5_K16": [ + 0.5341, + 0.3214 + ], + "span_t0.9_K4": [ + 0.1712, + 0.0683 + ], + "span_t0.9_K8": [ + 0.165, + 0.0716 + ], + "span_t0.9_K16": [ + 0.162, + 0.0731 + ], + "selfrep_t0.5_K4": [ + 0.5302, + 0.3011 + ], + "selfrep_t0.5_K8": [ + 0.5319, + 0.3084 + ], + "selfrep_t0.5_K16": [ + 0.5331, + 0.3183 + ], + "selfrep_t0.9_K4": [ + 0.164, + 0.0625 + ], + "selfrep_t0.9_K8": [ + 0.1595, + 0.0668 + ], + "selfrep_t0.9_K16": [ + 0.1543, + 0.0672 + ], + "both_t0.5_K4": [ + 0.5345, + 0.3094 + ], + "both_t0.5_K8": [ + 0.5367, + 0.3201 + ], + "both_t0.5_K16": [ + 0.5342, + 0.3242 + ], + "both_t0.9_K4": [ + 0.1688, + 0.0652 + ], + "both_t0.9_K8": [ + 0.1592, + 0.067 + ], + "both_t0.9_K16": [ + 0.1572, + 0.0705 + ] +} \ No newline at end of file diff --git a/pilots/objective.log b/pilots/objective.log new file mode 100644 index 0000000000000000000000000000000000000000..f5f9ddacd9048ebb366fe1cd79d568576295a830 --- /dev/null +++ b/pilots/objective.log @@ -0,0 +1,18 @@ +03:15:29 === objectives: base / span / selfrepair / both === +03:19:16 base 3000 loss=4.3888 (219s) +03:23:08 base 6000 loss=3.4443 (451s) +03:23:14 base t=0.5 K=4 tok=0.5338 pair=0.3032 K=8 tok=0.5358 pair=0.3135 K=16 tok=0.5335 pair=0.3147 +03:23:14 base t=0.9 K=4 tok=0.1697 pair=0.0663 K=8 tok=0.1623 pair=0.0693 K=16 tok=0.1577 pair=0.0707 +03:27:00 span 3000 loss=4.4491 (226s) +03:30:34 span 6000 loss=3.4527 (439s) +03:30:39 span t=0.5 K=4 tok=0.5325 pair=0.3069 K=8 tok=0.5362 pair=0.3198 K=16 tok=0.5341 pair=0.3214 +03:30:40 span t=0.9 K=4 tok=0.1712 pair=0.0683 K=8 tok=0.1650 pair=0.0716 K=16 tok=0.1620 pair=0.0731 +03:34:22 selfrep 3000 loss=4.2847 (222s) +03:37:58 selfrep 6000 loss=3.2314 (438s) +03:38:03 selfrep t=0.5 K=4 tok=0.5302 pair=0.3011 K=8 tok=0.5319 pair=0.3084 K=16 tok=0.5331 pair=0.3183 +03:38:04 selfrep t=0.9 K=4 tok=0.1640 pair=0.0625 K=8 tok=0.1595 pair=0.0668 K=16 tok=0.1543 pair=0.0672 +03:41:50 both 3000 loss=4.3134 (226s) +03:45:33 both 6000 loss=3.2216 (448s) +03:45:38 both t=0.5 K=4 tok=0.5345 pair=0.3094 K=8 tok=0.5367 pair=0.3201 K=16 tok=0.5342 pair=0.3242 +03:45:39 both t=0.9 K=4 tok=0.1688 pair=0.0652 K=8 tok=0.1592 pair=0.0670 K=16 tok=0.1572 pair=0.0705 +03:45:39 published pilots/objective.json diff --git a/pilots/opt_pilot.json b/pilots/opt_pilot.json new file mode 100644 index 0000000000000000000000000000000000000000..3f4b1dbaaad35c0abddd75df7eb002fbb2e6bc79 --- /dev/null +++ b/pilots/opt_pilot.json @@ -0,0 +1,52 @@ +[ + { + "ce@0.3": 2.541595935821533, + "ce@0.6": 3.409759521484375, + "pair": 0.16301322199025747, + "tok": 0.3892141756548536, + "ce@0.9": 5.002624988555908, + "ce": 3.651326815287272, + "tag": "base", + "secs": 415.9052699919994 + }, + { + "ce@0.3": 2.5441536903381348, + "ce@0.6": 3.4071030616760254, + "pair": 0.1638830897703549, + "tok": 0.3901386748844376, + "ce@0.9": 5.003111839294434, + "ce": 3.6514561971028647, + "tag": "plain", + "secs": 444.69129400699967 + }, + { + "ce@0.3": 2.2862162590026855, + "ce@0.6": 3.17996883392334, + "pair": 0.19119693806541405, + "tok": 0.4156137647663071, + "ce@0.9": 4.896960258483887, + "ce": 3.454381783803304, + "tag": "cautious", + "secs": 458.3112234720011 + }, + { + "ce@0.3": 2.5429396629333496, + "ce@0.6": 3.406092643737793, + "pair": 0.16440501043841335, + "tok": 0.3896250642013354, + "ce@0.9": 5.002017974853516, + "ce": 3.6503500938415527, + "tag": "cwd", + "secs": 420.9203538700003 + }, + { + "ce@0.3": 2.2803192138671875, + "ce@0.6": 3.174450159072876, + "pair": 0.18910925539318024, + "tok": 0.4164355418592707, + "ce@0.9": 4.895169258117676, + "ce": 3.449979543685913, + "tag": "cautious+cwd", + "secs": 464.8913762250013 + } +] \ No newline at end of file diff --git a/pilots/opt_pilot.log b/pilots/opt_pilot.log new file mode 100644 index 0000000000000000000000000000000000000000..7a614930ef5186b2295fe2030f568c70d4954c19 --- /dev/null +++ b/pilots/opt_pilot.log @@ -0,0 +1,12 @@ +14:39:06 opt pilot: 4000 steps, bs 64, lr 3e-04, arms base,plain,cautious,cwd,cautious+cwd +14:46:04 base ce 3.6513 | 0.3:2.5416 0.6:3.4098 0.9:5.0026 | pair 0.1630 tok 0.3892 | 416s +14:53:34 plain ce 3.6515 | 0.3:2.5442 0.6:3.4071 0.9:5.0031 | pair 0.1639 tok 0.3901 | 445s +15:01:17 cautious ce 3.4544 | 0.3:2.2862 0.6:3.1800 0.9:4.8970 | pair 0.1912 tok 0.4156 | 458s +15:08:24 cwd ce 3.6504 | 0.3:2.5429 0.6:3.4061 0.9:5.0020 | pair 0.1644 tok 0.3896 | 421s +15:16:14 cautious+cwd ce 3.4500 | 0.3:2.2803 0.6:3.1745 0.9:4.8952 | pair 0.1891 tok 0.4164 | 465s +15:16:19 --- versus base --- +15:16:19 base dCE +0.0000 dpair +0.0000 time x1.00 +15:16:19 plain dCE +0.0001 dpair +0.0009 time x1.07 +15:16:19 cautious dCE -0.1969 dpair +0.0282 time x1.10 +15:16:19 cwd dCE -0.0010 dpair +0.0014 time x1.01 +15:16:19 cautious+cwd dCE -0.2013 dpair +0.0261 time x1.12 diff --git a/pilots/outer_ortho_0.log b/pilots/outer_ortho_0.log new file mode 100644 index 0000000000000000000000000000000000000000..10d06c2b15c73b3a8414ec12198d26ebfde53b10 --- /dev/null +++ b/pilots/outer_ortho_0.log @@ -0,0 +1,39 @@ +21:14:43 outer-orthogonalisation pilot: 4 workers x 50 local steps x 40 rounds, bs 32, lr 6e-04, seed 0, arms ['avg', 'scale', 'orth', 'orth_m', 'sync'] +21:15:48 avg round 5: ce 5.9686 (64s) +21:16:19 avg round 10: ce 5.9250 (95s) +21:16:49 avg round 15: ce 5.8884 (125s) +21:17:18 avg round 20: ce 5.7672 (154s) +21:17:46 avg round 25: ce 5.6918 (183s) +21:18:15 avg round 30: ce 5.5897 (211s) +21:18:44 avg round 35: ce 5.4288 (241s) +21:19:19 avg round 40: ce 5.2819 (275s) +21:19:39 RESULT avg r5:5.9686 r10:5.9250 r15:5.8884 r20:5.7672 r25:5.6918 r30:5.5897 r35:5.4288 r40:5.2819 +21:20:42 scale round 5: ce 5.9686 (62s) +21:21:21 scale round 10: ce 5.9351 (101s) +21:22:01 scale round 15: ce 5.8983 (140s) +21:22:40 scale round 20: ce 5.8756 (179s) +21:23:19 scale round 25: ce 5.8483 (218s) +21:23:58 scale round 30: ce 5.7557 (257s) +21:24:37 scale round 35: ce 5.7163 (296s) +21:25:16 scale round 40: ce 5.6876 (336s) +21:25:36 RESULT scale r5:5.9686 r10:5.9351 r15:5.8983 r20:5.8756 r25:5.8483 r30:5.7557 r35:5.7163 r40:5.6876 +21:26:38 orth round 5: ce 6.1813 (62s) +21:27:18 orth round 10: ce 6.0847 (101s) +21:27:58 orth round 15: ce 6.0074 (141s) +21:28:38 orth round 20: ce 5.8833 (181s) +21:29:17 orth round 25: ce 5.3913 (221s) +21:29:57 orth round 30: ce 4.6464 (260s) +21:30:37 orth round 35: ce 4.3250 (300s) +21:31:16 orth round 40: ce 4.1799 (339s) +21:31:36 RESULT orth r5:6.1813 r10:6.0847 r15:6.0074 r20:5.8833 r25:5.3913 r30:4.6464 r35:4.3250 r40:4.1799 +21:32:38 orth_m round 5: ce 6.1436 (61s) +21:33:17 orth_m round 10: ce 6.0515 (101s) +21:33:57 orth_m round 15: ce 5.9338 (140s) +21:34:37 orth_m round 20: ce 5.5953 (180s) +21:35:16 orth_m round 25: ce 4.5903 (220s) +21:35:56 orth_m round 30: ce 4.1725 (260s) +21:36:36 orth_m round 35: ce 3.9704 (299s) +21:37:16 orth_m round 40: ce 3.8688 (339s) +21:37:36 RESULT orth_m r5:6.1436 r10:6.0515 r15:5.9338 r20:5.5953 r25:4.5903 r30:4.1725 r35:3.9704 r40:3.8688 +21:42:22 RESULT sync r5:5.9627 r10:5.9194 r15:5.8448 r20:5.6808 r25:5.0786 r30:4.6793 r35:4.5249 r40:4.4359 +21:42:23 OUTER ORTHO DONE diff --git a/pilots/outer_ortho_lr3.log b/pilots/outer_ortho_lr3.log new file mode 100644 index 0000000000000000000000000000000000000000..d40e9239e3bdf419f75e48f6a7e6d54a7cbc4254 --- /dev/null +++ b/pilots/outer_ortho_lr3.log @@ -0,0 +1,30 @@ +21:39:02 outer-orthogonalisation pilot: 4 workers x 50 local steps x 60 rounds, bs 32, lr 3e-04, seed 0, arms ['avg', 'orth_m', 'sync'] +22:07:12 outer-orthogonalisation pilot: 4 workers x 50 local steps x 60 rounds, bs 32, lr 3e-04, seed 0, arms ['avg', 'orth_m', 'sync'] +22:08:22 avg round 5: ce 5.9614 (70s) +22:09:00 avg round 10: ce 5.9158 (108s) +22:09:38 avg round 15: ce 5.8033 (146s) +22:10:17 avg round 20: ce 5.6751 (184s) +22:10:55 avg round 25: ce 5.3911 (222s) +22:11:33 avg round 30: ce 4.8188 (261s) +22:12:11 avg round 35: ce 4.4835 (299s) +22:12:49 avg round 40: ce 4.2860 (337s) +22:13:27 avg round 45: ce 4.1618 (375s) +22:14:05 avg round 50: ce 4.0850 (413s) +22:14:44 avg round 55: ce 4.0395 (451s) +22:15:22 avg round 60: ce 4.0046 (489s) +22:15:42 RESULT avg r5:5.9614 r10:5.9158 r15:5.8033 r20:5.6751 r25:5.3911 r30:4.8188 r35:4.4835 r40:4.2860 r45:4.1618 r50:4.0850 r55:4.0395 r60:4.0046 +22:16:51 orth_m round 5: ce 6.1033 (63s) +22:17:31 orth_m round 10: ce 6.0159 (103s) +22:18:10 orth_m round 15: ce 5.9713 (142s) +22:18:50 orth_m round 20: ce 5.9098 (181s) +22:19:29 orth_m round 25: ce 5.4955 (221s) +22:20:09 orth_m round 30: ce 4.7066 (261s) +22:20:49 orth_m round 35: ce 4.3041 (301s) +22:21:29 orth_m round 40: ce 4.0816 (340s) +22:22:08 orth_m round 45: ce 3.9294 (380s) +22:22:48 orth_m round 50: ce 3.8395 (420s) +22:23:28 orth_m round 55: ce 3.7885 (460s) +22:24:08 orth_m round 60: ce 3.7502 (500s) +22:24:28 RESULT orth_m r5:6.1033 r10:6.0159 r15:5.9713 r20:5.9098 r25:5.4955 r30:4.7066 r35:4.3041 r40:4.0816 r45:3.9294 r50:3.8395 r55:3.7885 r60:3.7502 +22:31:27 RESULT sync r5:5.9234 r10:5.8368 r15:5.5260 r20:4.4893 r25:4.0672 r30:3.8381 r35:3.6922 r40:3.5837 r45:3.5116 r50:3.4614 r55:3.4260 r60:3.4072 +22:31:28 OUTER ORTHO DONE diff --git a/pilots/outer_ortho_s1.log b/pilots/outer_ortho_s1.log new file mode 100644 index 0000000000000000000000000000000000000000..5f26d2013eb5279c9739e1fd6e5656a3dbfae470 --- /dev/null +++ b/pilots/outer_ortho_s1.log @@ -0,0 +1,29 @@ +21:39:02 outer-orthogonalisation pilot: 4 workers x 50 local steps x 40 rounds, bs 32, lr 6e-04, seed 1, arms ['avg', 'orth_m'] +22:31:38 outer-orthogonalisation pilot: 4 workers x 50 local steps x 60 rounds, bs 32, lr 3e-04, seed 1, arms ['avg', 'orth_m'] +22:32:45 avg round 5: ce 5.9558 (67s) +22:33:23 avg round 10: ce 5.9074 (105s) +22:34:01 avg round 15: ce 5.7903 (143s) +22:34:39 avg round 20: ce 5.6727 (181s) +22:35:17 avg round 25: ce 5.3556 (219s) +22:35:55 avg round 30: ce 4.8327 (257s) +22:36:34 avg round 35: ce 4.4443 (295s) +22:37:12 avg round 40: ce 4.2427 (334s) +22:37:50 avg round 45: ce 4.1228 (372s) +22:38:28 avg round 50: ce 4.0516 (410s) +22:39:06 avg round 55: ce 4.0056 (448s) +22:39:44 avg round 60: ce 3.9739 (486s) +22:40:04 RESULT avg r5:5.9558 r10:5.9074 r15:5.7903 r20:5.6727 r25:5.3556 r30:4.8327 r35:4.4443 r40:4.2427 r45:4.1228 r50:4.0516 r55:4.0056 r60:3.9739 +22:41:05 orth_m round 5: ce 6.1046 (60s) +22:41:44 orth_m round 10: ce 6.0236 (100s) +22:42:24 orth_m round 15: ce 5.9695 (140s) +22:43:04 orth_m round 20: ce 5.9371 (179s) +22:43:43 orth_m round 25: ce 5.7434 (219s) +22:44:23 orth_m round 30: ce 5.0180 (258s) +22:45:03 orth_m round 35: ce 4.4659 (298s) +22:45:42 orth_m round 40: ce 4.2188 (338s) +22:46:22 orth_m round 45: ce 4.0661 (377s) +22:47:02 orth_m round 50: ce 3.9653 (417s) +22:47:41 orth_m round 55: ce 3.9008 (457s) +22:48:21 orth_m round 60: ce 3.8520 (496s) +22:48:41 RESULT orth_m r5:6.1046 r10:6.0236 r15:5.9695 r20:5.9371 r25:5.7434 r30:5.0180 r35:4.4659 r40:4.2188 r45:4.0661 r50:3.9653 r55:3.9008 r60:3.8520 +22:48:42 OUTER ORTHO DONE diff --git a/pilots/outer_ortho_s2.log b/pilots/outer_ortho_s2.log new file mode 100644 index 0000000000000000000000000000000000000000..3db19d89b663965dadac190a1372ca23f289487c --- /dev/null +++ b/pilots/outer_ortho_s2.log @@ -0,0 +1,29 @@ +21:39:01 outer-orthogonalisation pilot: 4 workers x 50 local steps x 40 rounds, bs 32, lr 6e-04, seed 2, arms ['avg', 'orth_m'] +22:48:52 outer-orthogonalisation pilot: 4 workers x 50 local steps x 60 rounds, bs 32, lr 3e-04, seed 2, arms ['avg', 'orth_m'] +22:49:54 avg round 5: ce 5.9678 (62s) +22:50:32 avg round 10: ce 5.9197 (100s) +22:51:10 avg round 15: ce 5.8414 (138s) +22:51:49 avg round 20: ce 5.7274 (176s) +22:52:27 avg round 25: ce 5.6307 (214s) +22:53:04 avg round 30: ce 5.2859 (251s) +22:53:42 avg round 35: ce 4.8262 (289s) +22:54:20 avg round 40: ce 4.5854 (327s) +22:54:58 avg round 45: ce 4.4533 (366s) +22:55:36 avg round 50: ce 4.3690 (404s) +22:56:14 avg round 55: ce 4.3202 (442s) +22:56:52 avg round 60: ce 4.2842 (480s) +22:57:12 RESULT avg r5:5.9678 r10:5.9197 r15:5.8414 r20:5.7274 r25:5.6307 r30:5.2859 r35:4.8262 r40:4.5854 r45:4.4533 r50:4.3690 r55:4.3202 r60:4.2842 +22:58:15 orth_m round 5: ce 6.0966 (62s) +22:58:55 orth_m round 10: ce 6.0155 (102s) +22:59:35 orth_m round 15: ce 5.9707 (141s) +23:00:14 orth_m round 20: ce 5.9272 (181s) +23:00:54 orth_m round 25: ce 5.7275 (220s) +23:01:34 orth_m round 30: ce 5.0126 (260s) +23:02:13 orth_m round 35: ce 4.4409 (300s) +23:02:53 orth_m round 40: ce 4.1717 (339s) +23:03:33 orth_m round 45: ce 4.0101 (379s) +23:04:12 orth_m round 50: ce 3.9108 (418s) +23:04:51 orth_m round 55: ce 3.8536 (457s) +23:05:30 orth_m round 60: ce 3.8113 (497s) +23:05:50 RESULT orth_m r5:6.0966 r10:6.0155 r15:5.9707 r20:5.9272 r25:5.7275 r30:5.0126 r35:4.4409 r40:4.1717 r45:4.0101 r50:3.9108 r55:3.8536 r60:3.8113 +23:05:51 OUTER ORTHO DONE diff --git a/pilots/phrase_pilot.log b/pilots/phrase_pilot.log new file mode 100644 index 0000000000000000000000000000000000000000..de98d29b852c654909e5dfd770139f3b64abd6a5 --- /dev/null +++ b/pilots/phrase_pilot.log @@ -0,0 +1,26 @@ +08:52:11 docs: 130000 train, 400 eval; eval bytes 279389 +08:52:16 bpe 8190 trained in 4s +08:52:49 phrases on bpe 8190: 1536 + 512 merges in 3s; e.g. _of_the, _in_the, _such_as, _following_best, _best_describes, _Which_of, _the_following, _the_text, _What_is, _can_be, _to_the, _as_a, _Which_of_the_following, _the_following_best_describes, _according_to_the_text, _the_following_is_NOT, _Is_it_true_that, _What_is_the_main, _mentioned_in_the_text, _in_the_text +08:53:05 bpe: vocab 8192, 0.242 tokens/byte, 406276 rows (need 384000), eval truncated 16 +08:53:05 bpe+phr: vocab 10240, 0.221 tokens/byte, 371783 rows (need 384000), eval truncated 16 +08:53:09 bpe 6142 trained in 4s +08:53:38 phrases on bpe 6142: 1536 + 512 merges in 3s; e.g. _of_the, _in_the, _such_as, _following_best, _best_describes, _Which_of, _the_following, _the_text, _What_is, _to_the, _can_be, _as_a, _Which_of_the_following, _the_following_best_describes, _according_to_the_text, _the_following_is_NOT, _Is_it_true_that, _What_is_the_main, _mentioned_in_the_text, _in_the_text +08:53:53 phr: vocab 8192, 0.234 tokens/byte, 392795 rows (need 384000), eval truncated 17 +08:54:30 bpe step 1000 text seen 68 MB bpb 2.5253 (33s) +08:55:03 bpe step 2000 text seen 136 MB bpb 2.1461 (66s) +08:55:46 bpe step 3000 text seen 203 MB bpb 1.9985 (109s) +08:57:01 bpe step 4000 text seen 271 MB bpb 1.9206 (184s) +08:58:15 bpe step 5000 text seen 339 MB bpb 1.8818 (258s) +08:59:29 bpe step 6000 text seen 407 MB bpb 1.8596 (332s) +09:00:48 phr step 1000 text seen 70 MB bpb 2.5659 (75s) +09:02:03 phr step 2000 text seen 140 MB bpb 2.2337 (149s) +09:03:17 phr step 3000 text seen 210 MB bpb 2.0466 (223s) +09:04:32 phr step 4000 text seen 280 MB bpb 1.9622 (298s) +09:05:46 phr step 5000 text seen 351 MB bpb 1.9137 (373s) +09:07:00 phr step 6000 text seen 421 MB bpb 1.8909 (447s) +09:08:23 bpe+phr step 1000 text seen 74 MB bpb 2.4725 (79s) +09:09:41 bpe+phr step 2000 text seen 148 MB bpb 2.1510 (157s) +09:10:59 bpe+phr step 3000 text seen 222 MB bpb 2.0054 (235s) +09:12:18 bpe+phr step 4000 text seen 296 MB bpb 1.9296 (314s) +09:13:36 bpe+phr step 5000 text seen 370 MB bpb 1.8905 (392s) +09:14:54 bpe+phr step 6000 text seen 444 MB bpb 1.8709 (470s) diff --git a/pilots/pkm1b.log b/pilots/pkm1b.log new file mode 100644 index 0000000000000000000000000000000000000000..32a31ee308429d632c8a1aedba9874fe66679fa4 --- /dev/null +++ b/pilots/pkm1b.log @@ -0,0 +1,27 @@ +15:40:21 pkm test from global round 518 step 136800: 1000 steps x 64 rows, arms ['ctrl'], slots 1000000 x 256 +15:55:08 data ready in 886s +15:57:55 ctrl step 50 loss 3.3026 (158s) +16:00:31 ctrl step 100 loss 3.7679 (314s) +16:03:08 ctrl step 150 loss 3.1679 (470s) +16:05:44 ctrl step 200 loss 3.3044 (626s) +16:08:20 ctrl step 250 loss 2.9798 (782s) +16:08:23 EVAL ctrl step 250 | facts mc 0.5457 nll 6.0348 em 0.0762 +16:10:59 ctrl step 300 loss 2.9517 (942s) +16:13:35 ctrl step 350 loss 3.3515 (1098s) +16:16:12 ctrl step 400 loss 3.2505 (1254s) +16:18:48 ctrl step 450 loss 3.2101 (1410s) +16:21:24 ctrl step 500 loss 2.9018 (1566s) +16:21:25 EVAL ctrl step 500 | facts mc 0.5410 nll 6.0856 em 0.0743 +16:24:02 ctrl step 550 loss 3.9067 (1724s) +16:26:38 ctrl step 600 loss 3.0002 (1880s) +16:29:14 ctrl step 650 loss 3.0659 (2037s) +16:31:51 ctrl step 700 loss 2.9223 (2193s) +16:34:27 ctrl step 750 loss 3.5296 (2349s) +16:34:28 EVAL ctrl step 750 | facts mc 0.5486 nll 6.0934 em 0.0676 +16:37:05 ctrl step 800 loss 4.0751 (2507s) +16:39:41 ctrl step 850 loss 3.5520 (2663s) +16:42:17 ctrl step 900 loss 2.9435 (2819s) +16:44:53 ctrl step 950 loss 3.4863 (2976s) +16:47:30 ctrl step 1000 loss 3.0002 (3132s) +16:47:32 EVAL ctrl step 1000 | facts mc 0.5429 nll 6.0695 em 0.0752 long 2.9042 short 4.0073 +16:47:33 PKM DONE diff --git a/pilots/pkm1b_mem.log b/pilots/pkm1b_mem.log new file mode 100644 index 0000000000000000000000000000000000000000..c0d32c921f8888d42fe46bbe315812340dac841c --- /dev/null +++ b/pilots/pkm1b_mem.log @@ -0,0 +1,28 @@ +16:48:49 pkm test from global round 518 step 136800: 1000 steps x 64 rows, arms ['mem'], slots 1000000 x 256 +17:02:47 data ready in 838s +17:03:02 pkm arm: 2097152 slots (537M value params), 3.9M query/out params, read before blocks [3, 9, 14], top 16 of 8^3 +17:05:48 mem step 50 loss 3.3027 (167s) |wo| 2.055,2.111,1.961 slots used/batch 583,1100,20065 +17:08:33 mem step 100 loss 3.7679 (332s) |wo| 3.153,3.208,3.123 slots used/batch 99,266,9236 +17:11:18 mem step 150 loss 3.1679 (497s) |wo| 3.924,3.936,3.945 slots used/batch 86,182,1814 +17:14:03 mem step 200 loss 3.3043 (661s) |wo| 4.569,4.547,4.574 slots used/batch 90,136,1208 +17:16:48 mem step 250 loss 2.9800 (826s) |wo| 5.080,5.018,5.083 slots used/batch 47,111,847 +17:16:51 EVAL mem step 250 | facts mc 0.5448 nll 6.0324 em 0.0762 +17:19:36 mem step 300 loss 2.9518 (994s) |wo| 5.541,5.411,5.525 slots used/batch 40,89,468 +17:22:21 mem step 350 loss 3.3515 (1159s) |wo| 5.937,5.829,5.917 slots used/batch 33,82,446 +17:25:05 mem step 400 loss 3.2505 (1324s) |wo| 6.266,6.175,6.252 slots used/batch 49,82,400 +17:27:50 mem step 450 loss 3.2103 (1488s) |wo| 6.614,6.447,6.512 slots used/batch 43,87,435 +17:30:34 mem step 500 loss 2.9020 (1653s) |wo| 6.912,6.771,6.819 slots used/batch 54,112,458 +17:30:36 EVAL mem step 500 | facts mc 0.5429 nll 6.0810 em 0.0724 +17:33:21 mem step 550 loss 3.9059 (1819s) |wo| 7.229,7.068,7.094 slots used/batch 53,88,448 +17:36:06 mem step 600 loss 3.0003 (1984s) |wo| 7.506,7.353,7.352 slots used/batch 66,128,638 +17:38:50 mem step 650 loss 3.0657 (2149s) |wo| 7.836,7.652,7.636 slots used/batch 55,92,458 +17:41:35 mem step 700 loss 2.9226 (2314s) |wo| 8.153,7.872,7.871 slots used/batch 48,88,392 +17:44:20 mem step 750 loss 3.5297 (2479s) |wo| 8.390,8.109,8.107 slots used/batch 60,107,456 +17:44:22 EVAL mem step 750 | facts mc 0.5514 nll 6.0899 em 0.0657 +17:47:07 mem step 800 loss 4.0750 (2645s) |wo| 8.597,8.357,8.349 slots used/batch 55,153,1022 +17:49:52 mem step 850 loss 3.5521 (2810s) |wo| 8.866,8.613,8.595 slots used/batch 55,136,934 +17:52:36 mem step 900 loss 2.9434 (2975s) |wo| 9.109,8.845,8.850 slots used/batch 55,135,541 +17:55:21 mem step 950 loss 3.4859 (3140s) |wo| 9.354,9.070,9.060 slots used/batch 36,148,726 +17:58:06 mem step 1000 loss 3.0010 (3305s) |wo| 9.549,9.312,9.289 slots used/batch 52,163,1029 +17:58:08 EVAL mem step 1000 | facts mc 0.5438 nll 6.0484 em 0.0790 long 2.9043 short 4.0056 +17:58:09 PKM DONE diff --git a/pilots/pkm2_mem_ctrl.log b/pilots/pkm2_mem_ctrl.log new file mode 100644 index 0000000000000000000000000000000000000000..5d2dc0a9b5b4498aef0f98bdfef561fa5ebf4339 --- /dev/null +++ b/pilots/pkm2_mem_ctrl.log @@ -0,0 +1,52 @@ +19:43:45 pkm2 (query BN, keys lr 1e-04) test from global round 537 step 140250: 1000 steps x 64 rows, arms ['mem', 'ctrl'], 2097152 slots +19:57:06 data ready in 801s +19:57:20 pkm arm: 2097152 slots (537M value params), 3.9M query/out params, read before blocks [3, 9, 14], top 16 of 8^3 +20:00:13 mem step 50 loss 3.3036 (173s) |wo| 1.955,1.914,1.888 slots used/batch 194944,286423,285886 +20:02:58 mem step 100 loss 3.7698 (338s) |wo| 3.134,3.135,3.059 slots used/batch 145583,255082,267721 +20:05:42 mem step 150 loss 3.1698 (502s) |wo| 3.965,3.990,3.883 slots used/batch 96545,209817,253652 +20:08:27 mem step 200 loss 3.3056 (667s) |wo| 4.639,4.692,4.556 slots used/batch 75128,181667,200741 +20:11:11 mem step 250 loss 2.9771 (831s) |wo| 5.227,5.300,5.142 slots used/batch 89016,186179,212295 +20:11:19 EVAL mem step 250 | facts mc 0.5543 nll 6.0017 em 0.0800 +20:14:04 mem step 300 loss 2.9544 (1004s) |wo| 5.753,5.812,5.681 slots used/batch 71559,149147,193290 +20:16:48 mem step 350 loss 3.3512 (1168s) |wo| 6.230,6.311,6.164 slots used/batch 77739,164676,202873 +20:19:33 mem step 400 loss 3.2505 (1333s) |wo| 6.660,6.761,6.596 slots used/batch 80859,150350,184488 +20:22:17 mem step 450 loss 3.2095 (1497s) |wo| 7.057,7.181,6.997 slots used/batch 78922,139819,186551 +20:25:02 mem step 500 loss 2.9035 (1662s) |wo| 7.432,7.579,7.378 slots used/batch 73173,137114,184838 +20:25:03 EVAL mem step 500 | facts mc 0.5476 nll 6.1074 em 0.0762 +20:27:48 mem step 550 loss 3.9067 (1828s) |wo| 7.795,7.955,7.747 slots used/batch 67600,121021,165555 +20:30:33 mem step 600 loss 3.0004 (1993s) |wo| 8.142,8.298,8.090 slots used/batch 78596,138816,184164 +20:33:18 mem step 650 loss 3.0662 (2158s) |wo| 8.474,8.626,8.425 slots used/batch 71319,125133,178872 +20:36:02 mem step 700 loss 2.9204 (2322s) |wo| 8.785,8.941,8.740 slots used/batch 68821,112064,161948 +20:38:47 mem step 750 loss 3.5282 (2487s) |wo| 9.092,9.253,9.030 slots used/batch 67685,107322,150310 +20:38:48 EVAL mem step 750 | facts mc 0.5438 nll 6.1373 em 0.0629 +20:41:34 mem step 800 loss 4.0744 (2654s) |wo| 9.376,9.549,9.312 slots used/batch 72349,109279,148882 +20:44:18 mem step 850 loss 3.5503 (2818s) |wo| 9.662,9.831,9.592 slots used/batch 80672,118638,161610 +20:47:03 mem step 900 loss 2.9436 (2983s) |wo| 9.917,10.105,9.865 slots used/batch 83581,125061,167604 +20:49:48 mem step 950 loss 3.4850 (3148s) |wo| 10.190,10.381,10.130 slots used/batch 76186,124659,149468 +20:52:32 mem step 1000 loss 2.9999 (3312s) |wo| 10.454,10.644,10.391 slots used/batch 75856,123367,151795 +20:52:35 EVAL mem step 1000 | facts mc 0.5410 nll 6.1012 em 0.0752 long 2.9045 short 4.0013 +20:55:21 ctrl step 50 loss 3.3033 (156s) +20:57:57 ctrl step 100 loss 3.7699 (311s) +21:00:32 ctrl step 150 loss 3.1699 (467s) +21:03:08 ctrl step 200 loss 3.3056 (623s) +21:05:44 ctrl step 250 loss 2.9774 (778s) +21:05:45 EVAL ctrl step 250 | facts mc 0.5543 nll 6.0053 em 0.0781 +21:08:21 ctrl step 300 loss 2.9543 (936s) +21:10:57 ctrl step 350 loss 3.3510 (1091s) +21:13:32 ctrl step 400 loss 3.2504 (1247s) +21:16:08 ctrl step 450 loss 3.2093 (1403s) +21:18:43 ctrl step 500 loss 2.9035 (1558s) +21:18:45 EVAL ctrl step 500 | facts mc 0.5457 nll 6.1106 em 0.0743 +21:21:21 ctrl step 550 loss 3.9069 (1716s) +21:23:57 ctrl step 600 loss 3.0005 (1872s) +21:26:33 ctrl step 650 loss 3.0659 (2027s) +21:29:08 ctrl step 700 loss 2.9203 (2183s) +21:31:44 ctrl step 750 loss 3.5284 (2339s) +21:31:45 EVAL ctrl step 750 | facts mc 0.5457 nll 6.1399 em 0.0629 +21:34:21 ctrl step 800 loss 4.0746 (2496s) +21:36:57 ctrl step 850 loss 3.5510 (2652s) +21:39:33 ctrl step 900 loss 2.9436 (2808s) +21:42:09 ctrl step 950 loss 3.4849 (2963s) +21:44:44 ctrl step 1000 loss 3.0001 (3119s) +21:44:46 EVAL ctrl step 1000 | facts mc 0.5381 nll 6.3159 em 0.0590 long 2.9046 short 4.0030 +21:44:47 PKM DONE diff --git a/pilots/pmi_pilot.json b/pilots/pmi_pilot.json new file mode 100644 index 0000000000000000000000000000000000000000..ca14a9e647ef7f41f0439aeade24deb3d82bdb98 --- /dev/null +++ b/pilots/pmi_pilot.json @@ -0,0 +1,14 @@ +[ + { + "ce@0.3": 2.584545850753784, + "ce@0.6": 3.4313249588012695, + "pair": 0.15744606819763396, + "tok": 0.38243451463790445, + "ce@0.9": 5.008693695068359, + "ce": 3.674854834874471, + "ce_units@0.6": 3.476053476333618, + "tag": "pmi", + "secs": 428.62712072900104, + "seg_secs": 15.076797405028628 + } +] \ No newline at end of file diff --git a/pilots/pmi_pilot.log b/pilots/pmi_pilot.log new file mode 100644 index 0000000000000000000000000000000000000000..3edc7b1792e88d773fe9dd3efa73276666a7ea47 --- /dev/null +++ b/pilots/pmi_pilot.log @@ -0,0 +1,74 @@ +16:36:12 pmi table: 20000 bigrams (pmi >= 2.71), 10000 trigrams (pmi >= 4.53) from 20M tokens in 66s; 50.9% of tokens fall inside a multi-token unit +16:36:12 sample units: ĠprettiestĠdress | ĠrubĠit | ĠheĠremembered | ?".ĠHer | WhenĠSarah | ?"ĠMom | Ġwrong? | ĠSnowfl | ĠtoĠmelt | ĠanotherĠvoice | "Now | ĠmeĠcarry | ĠJim.ĠJim | Hey,Ġstop | ĠuntilĠsheĠfinally | ĠLittleĠbyĠlittle | ĠbeĠscaredĠsometimes | ĠcakeĠwithĠcandles | ĠontoĠtheĠswing | ĠsticksĠandĠleaves +16:36:12 pmi pilot: 4000 steps, bs 64, lr 3e-04, span 0.50, arms base,pmi,pmi_all +16:38:09 base 1000 train=5.9347 (115s, segmentation 0s) +16:39:59 base 2000 train=4.7405 (226s, segmentation 0s) +16:41:51 base 3000 train=4.1446 (337s, segmentation 0s) +16:43:24 base 4000 train=4.0345 (430s, segmentation 0s) +16:43:24 base ce 3.7481 | 0.3:2.7062 0.6:3.5129 0.9:5.0252 | pair 0.1505 tok 0.3764 | units@0.6 3.9870 | 430s +16:45:01 pmi 1000 train=5.9620 (92s, segmentation 5s) +16:46:47 pmi 2000 train=4.7988 (198s, segmentation 10s) +16:48:34 pmi 3000 train=4.2890 (305s, segmentation 14s) +16:50:22 pmi 4000 train=4.1798 (413s, segmentation 19s) +16:50:22 pmi ce 3.7682 | 0.3:2.7165 0.6:3.5404 0.9:5.0477 | pair 0.1477 tok 0.3689 | units@0.6 3.9363 | 413s +16:52:23 pmi_all 1000 train=5.9729 (116s, segmentation 5s) +16:54:11 pmi_all 2000 train=5.0327 (224s, segmentation 9s) +16:55:59 pmi_all 3000 train=4.4833 (332s, segmentation 14s) +16:57:46 pmi_all 4000 train=4.3726 (438s, segmentation 19s) +16:57:46 pmi_all ce 4.1455 | 0.3:3.2513 0.6:3.9815 0.9:5.2037 | pair 0.0974 tok 0.2977 | units@0.6 3.9802 | 438s +16:57:51 --- versus base (base must read 3.7481 to match the earlier pilots) --- +16:57:51 base dCE +0.0000 dpair +0.0000 dtok +0.0000 dunits +0.0000 time x1.00 +16:57:51 pmi dCE +0.0201 dpair -0.0028 dtok -0.0075 dunits -0.0508 time x0.96 +16:57:51 pmi_all dCE +0.3974 dpair -0.0531 dtok -0.0787 dunits -0.0068 time x1.02 +16:57:52 published +16:57:52 PMI PILOT DONE +17:05:51 pmi table: 2000 bigrams (pmi >= 4.12), 300 trigrams (pmi >= 6.83) from 20M tokens in 66s; 14.4% of tokens fall inside a multi-token unit +17:05:51 sample units: ĠatĠschool | OfĠcourse | OneĠday | ĠsomeĠjuice | ĠgoodĠhelper | GoĠaway | ĠkeptĠgoing | ĠsmelledĠsomething | ĠcalledĠJane | ĠEveryĠmorning | ĠdarkĠclouds | Ġ"Now | ĠAfterĠaĠwhile | ĠpulledĠandĠpulled | ĠpullĠandĠtug | ĠsolveĠtheĠproblem | AfterĠtheyĠfinished | ĠheldĠitĠtight | ĠlearnedĠherĠlesson | ĠintoĠsmallĠpieces +17:05:51 pmi pilot: 4000 steps, bs 64, lr 3e-04, span 0.50, arms pmi +17:07:44 pmi 1000 train=5.9563 (112s, segmentation 4s) +17:09:30 pmi 2000 train=4.6271 (217s, segmentation 8s) +17:10:49 pmi 3000 train=4.1409 (297s, segmentation 12s) +17:12:02 pmi 4000 train=4.0344 (369s, segmentation 16s) +17:12:02 pmi ce 3.7158 | 0.3:2.6455 0.6:3.4788 0.9:5.0232 | pair 0.1533 tok 0.3777 | units@0.6 3.6105 | 369s +17:12:08 published +17:12:08 PMI PILOT DONE +17:33:35 pmi table: 700 bigrams (pmi >= 4.91), 100 trigrams (pmi >= 7.44) from 20M tokens in 67s; 7.3% of tokens fall inside a multi-token unit +17:33:35 sample units: ĠextraĠcareful | ThatĠnight | OfĠcourse | ĠhigherĠthan | ĠloudĠroar | OneĠday | ĠgoodĠhelper | ĠsayĠhello | ĠfellĠdown | GoĠaway | ĠmanyĠtimes | ĠoldĠman | AfterĠtheyĠfinished | ĠcanââĤ¬âĦ¢t | ĠgustĠofĠwind | ĠblackĠandĠwhite | ĠMamaĠandĠPapa | You'reĠwelcome | Ġthree-year | ĠhideĠandĠseek +17:33:35 pmi pilot: 4000 steps, bs 64, lr 3e-04, span 0.50, arms pmi +17:35:17 pmi 1000 train=5.9515 (101s, segmentation 4s) +17:37:03 pmi 2000 train=4.5015 (207s, segmentation 8s) +17:38:49 pmi 3000 train=4.0625 (313s, segmentation 11s) +17:40:35 pmi 4000 train=3.9647 (419s, segmentation 15s) +17:40:35 pmi ce 3.6701 | 0.3:2.5718 0.6:3.4318 0.9:5.0068 | pair 0.1588 tok 0.3826 | units@0.6 3.5045 | 419s +17:40:41 published +17:40:41 PMI PILOT DONE +17:42:01 pmi table: 5000 bigrams (pmi >= 3.42), 1200 trigrams (pmi >= 5.75) from 20M tokens in 69s; 25.4% of tokens fall inside a multi-token unit +17:42:01 sample units: ĠgoesĠback | ĠbestĠgift | ĠpineĠcone | OneĠday | ĠgoodĠhelper | ĠwithĠhappiness | ĠgetĠhurt | ĠdarkĠclouds | Ġ"Now | Ġslipping | Ġwrong? | MeĠtoo | ĠMr.ĠSmith | ĠsharingĠisĠimportant | 'mĠnotĠmad | ĠallĠnightĠlong | ĠhasĠmanyĠcolors | Hey,Ġstop | ĠMr.ĠSnuggles | ĠintoĠsmallĠpieces +17:42:01 pmi pilot: 4000 steps, bs 64, lr 3e-04, span 0.50, arms pmi +17:43:58 pmi 1000 train=5.9580 (115s, segmentation 4s) +17:45:44 pmi 2000 train=4.6413 (221s, segmentation 9s) +17:47:21 pmi 3000 train=4.1742 (319s, segmentation 13s) +17:48:33 pmi 4000 train=4.0698 (391s, segmentation 17s) +17:48:33 pmi ce 3.7219 | 0.3:2.6537 0.6:3.4900 0.9:5.0221 | pair 0.1534 tok 0.3756 | units@0.6 3.7149 | 391s +17:48:39 published +17:48:39 PMI PILOT DONE +17:58:29 pmi table: 20000 bigrams (pmi >= 2.71), 10000 trigrams (pmi >= 4.53) from 20M tokens in 69s; 50.9% of tokens fall inside a multi-token unit +17:58:29 sample units: ĠprettiestĠdress | ĠrubĠit | ĠheĠremembered | ?".ĠHer | WhenĠSarah | ?"ĠMom | Ġwrong? | ĠSnowfl | ĠtoĠmelt | ĠanotherĠvoice | "Now | ĠmeĠcarry | ĠJim.ĠJim | Hey,Ġstop | ĠuntilĠsheĠfinally | ĠLittleĠbyĠlittle | ĠbeĠscaredĠsometimes | ĠcakeĠwithĠcandles | ĠontoĠtheĠswing | ĠsticksĠandĠleaves +17:58:29 pmi pilot: 4000 steps, bs 64, lr 3e-04, span 0.00, arms base +18:00:15 base 1000 train=5.9471 (105s, segmentation 0s) +18:01:46 base 2000 train=4.4912 (195s, segmentation 0s) +18:03:27 base 3000 train=4.0242 (296s, segmentation 0s) +18:05:07 base 4000 train=3.9248 (397s, segmentation 0s) +18:05:07 base ce 3.6513 | 0.3:2.5416 0.6:3.4098 0.9:5.0026 | pair 0.1630 tok 0.3892 | units@0.6 3.9470 | 397s +18:05:12 --- versus base (base must read 3.7481 to match the earlier pilots) --- +18:05:12 base dCE +0.0000 dpair +0.0000 dtok +0.0000 dunits +0.0000 time x1.00 +18:05:13 published +18:05:13 PMI PILOT DONE +18:06:31 pmi table: 250 bigrams (pmi >= 5.30), 30 trigrams (pmi >= 7.64) from 20M tokens in 67s; 3.7% of tokens fall inside a multi-token unit +18:06:31 sample units: 'mĠglad | ĠitsĠmouth | CanĠwe | Ġwasn't | ĠfightĠover | OfĠcourse | ĠhappilyĠever | ĠbrightĠlight | ĠyearsĠold | ĠsayĠhello | ĠbaldĠman | GoĠaway | ĠhappilyĠeverĠafter | Ow,Ġow | ĠwaggedĠhisĠtail | ĠdidnââĤ¬âĦ¢t | Ġx-ray | Don'tĠworry | ĠspentĠtheĠrest | What'sĠwrong +18:06:31 pmi pilot: 4000 steps, bs 64, lr 3e-04, span 0.50, arms pmi +18:08:25 pmi 1000 train=5.9485 (113s, segmentation 4s) +18:10:11 pmi 2000 train=4.5163 (219s, segmentation 8s) +18:11:57 pmi 3000 train=4.0615 (325s, segmentation 11s) +18:13:41 pmi 4000 train=3.9603 (429s, segmentation 15s) +18:13:41 pmi ce 3.6749 | 0.3:2.5845 0.6:3.4313 0.9:5.0087 | pair 0.1574 tok 0.3824 | units@0.6 3.4761 | 429s diff --git a/pilots/sc_pilot.json b/pilots/sc_pilot.json new file mode 100644 index 0000000000000000000000000000000000000000..8f4ba1ce271f3937b04647dca8509aa91c207e4e --- /dev/null +++ b/pilots/sc_pilot.json @@ -0,0 +1,52 @@ +[ + { + "ce@0.3": 2.7062206268310547, + "ce@0.6": 3.5128703117370605, + "pair": 0.15048712595685457, + "tok": 0.3763739085772984, + "ce@0.9": 5.025173664093018, + "ce": 3.7480882008870444, + "tag": "base/1pass", + "secs": 454.6690887469995 + }, + { + "ce@0.3": 2.711622476577759, + "ce@0.6": 3.5273077487945557, + "pair": 0.14578983994432845, + "tok": 0.370005136106831, + "ce@0.9": 5.060877799987793, + "ce": 3.7666026751200357, + "tag": "sc/1pass", + "secs": 444.1969652539992 + }, + { + "ce@0.3": 2.6729958057403564, + "ce@0.6": 3.448431968688965, + "pair": 0.1671885873347251, + "tok": 0.38613251155624034, + "ce@0.9": 5.019032001495361, + "ce": 3.713486591974894, + "tag": "sc/2pass", + "secs": 444.1969652539992 + }, + { + "ce@0.3": 2.5927491188049316, + "ce@0.6": 3.459599733352661, + "pair": 0.15187891440501045, + "tok": 0.377914740626605, + "ce@0.9": 5.055078983306885, + "ce": 3.7024759451548257, + "tag": "sc+elbo/1pass", + "secs": 355.8079646189981 + }, + { + "ce@0.3": 2.543799877166748, + "ce@0.6": 3.369680166244507, + "pair": 0.17832289491997216, + "tok": 0.3922958397534669, + "ce@0.9": 5.002950668334961, + "ce": 3.6388102372487388, + "tag": "sc+elbo/2pass", + "secs": 355.8079646189981 + } +] \ No newline at end of file diff --git a/pilots/sc_pilot.log b/pilots/sc_pilot.log new file mode 100644 index 0000000000000000000000000000000000000000..4b792113e56e0ce2bc0f2c1850417e659012252a --- /dev/null +++ b/pilots/sc_pilot.log @@ -0,0 +1,26 @@ +18:33:10 sc pilot: 4000 steps, bs 64, lr 3e-04, span 0.50, arms base,sc,sc+elbo +18:35:10 base 1000 train=5.9347 (118s) +18:37:01 base 2000 train=4.7405 (229s) +18:38:53 base 3000 train=4.1446 (341s) +18:40:47 base 4000 train=4.0345 (455s) +18:40:47 base/1pass ce 3.7481 | 0.3:2.7062 0.6:3.5129 0.9:5.0252 | pair 0.1505 tok 0.3764 | 455s +18:43:01 sc 1000 train=5.9331 (130s) +18:45:05 sc 2000 train=4.7393 (253s) +18:47:03 sc 3000 train=4.1419 (371s) +18:48:16 sc 4000 train=4.0267 (444s) +18:48:16 sc/1pass ce 3.7666 | 0.3:2.7116 0.6:3.5273 0.9:5.0609 | pair 0.1458 tok 0.3700 | 444s +18:48:16 sc/2pass ce 3.7135 | 0.3:2.6730 0.6:3.4484 0.9:5.0190 | pair 0.1672 tok 0.3861 | 444s +18:48:16 sc projection norm 2.6691 (starts at 0) +18:50:34 sc+elbo 1000 train=5.9087 (132s) +18:51:48 sc+elbo 2000 train=4.4299 (207s) +18:53:03 sc+elbo 3000 train=3.8391 (282s) +18:54:17 sc+elbo 4000 train=3.7182 (356s) +18:54:17 sc+elbo/1pass ce 3.7025 | 0.3:2.5927 0.6:3.4596 0.9:5.0551 | pair 0.1519 tok 0.3779 | 356s +18:54:17 sc+elbo/2pass ce 3.6388 | 0.3:2.5438 0.6:3.3697 0.9:5.0030 | pair 0.1783 tok 0.3923 | 356s +18:54:17 sc+elbo projection norm 2.7896 (starts at 0) +18:54:22 --- versus base/1pass --- +18:54:22 base/1pass dCE +0.0000 dpair +0.0000 dtok +0.0000 time x1.00 +18:54:22 sc/1pass dCE +0.0185 dpair -0.0047 dtok -0.0064 time x0.98 +18:54:22 sc/2pass dCE -0.0346 dpair +0.0167 dtok +0.0098 time x0.98 +18:54:22 sc+elbo/1pass dCE -0.0456 dpair +0.0014 dtok +0.0015 time x0.78 +18:54:22 sc+elbo/2pass dCE -0.1093 dpair +0.0278 dtok +0.0159 time x0.78 diff --git a/pilots/short1b.log b/pilots/short1b.log new file mode 100644 index 0000000000000000000000000000000000000000..ab211d80dc398dd7782fa88156cb39f5b02e0cb9 --- /dev/null +++ b/pilots/short1b.log @@ -0,0 +1,15 @@ +08:46:42 short-row test from global round 480 step 130950: 300 steps x 64 rows, lr 8e-05, arms [('ctrl', 0.0), ('s10', 0.1), ('s25', 0.25)] +08:55:10 data ready +09:06:33 p=0.00 step 100 loss 3.7699 (673s) +09:16:08 p=0.00 step 200 loss 3.3052 (1248s) +09:21:21 p=0.00 step 300 loss 2.9552 (1561s) +09:21:32 ctrl: long 2.9008 short 3.9863 f_mc 0.5495 f_nll 6.0579 f_em 0.0724 +09:26:58 p=0.10 step 100 loss 3.8773 (316s) +09:32:07 p=0.10 step 200 loss 3.3056 (624s) +09:37:16 p=0.10 step 300 loss 3.0879 (934s) +09:37:18 s10: long 2.8972 short 3.9656 f_mc 0.5571 f_nll 6.0090 f_em 0.0781 +09:42:31 p=0.25 step 100 loss 4.0335 (303s) +09:47:34 p=0.25 step 200 loss 3.4373 (606s) +09:52:37 p=0.25 step 300 loss 3.0885 (908s) +09:52:39 s25: long 2.9020 short 3.9541 f_mc 0.5552 f_nll 6.0246 f_em 0.0781 +09:52:40 SHORT1B DONE diff --git a/pilots/speed_pilot.json b/pilots/speed_pilot.json new file mode 100644 index 0000000000000000000000000000000000000000..9203bbcde7bed11569e9088e44a72283f9f1505f --- /dev/null +++ b/pilots/speed_pilot.json @@ -0,0 +1,32 @@ +[ + { + "ce@0.3": 2.0451419353485107, + "ce@0.6": 2.9197587966918945, + "pair": 0.22929714683368127, + "tok": 0.4538263995891115, + "ce@0.9": 4.784701347351074, + "ce": 3.2498673597971597, + "tag": "muon.005", + "secs": 474.74618161700073 + }, + { + "ce@0.3": 2.822192668914795, + "ce@0.6": 3.6363308429718018, + "pair": 0.12839248434237996, + "tok": 0.3546995377503852, + "ce@0.9": 5.054590225219727, + "ce": 3.837704579035441, + "tag": "muon.0025", + "secs": 511.9214362620005 + }, + { + "ce@0.3": 1.9032678604125977, + "ce@0.6": 2.835272789001465, + "pair": 0.23434237995824633, + "tok": 0.4601951720595788, + "ce@0.9": 4.782243251800537, + "ce": 3.1735946337381997, + "tag": "elbo+muon.01", + "secs": 510.435403114001 + } +] \ No newline at end of file diff --git a/pilots/speed_pilot.log b/pilots/speed_pilot.log new file mode 100644 index 0000000000000000000000000000000000000000..082e59411a056231cc8090eba9caaebb9414e2c6 --- /dev/null +++ b/pilots/speed_pilot.log @@ -0,0 +1,70 @@ +14:43:36 pilot: 4000 steps, bs 64, lr 3e-04, span 0.50, arms base,elbo,muon.02,muon.05 +14:45:35 base 1000 train=5.9347 (117s) +14:47:26 base 2000 train=4.7405 (228s) +14:49:15 base 3000 train=4.1446 (337s) +14:50:24 base 4000 train=4.0345 (406s) +14:50:24 base ce 3.7481 | 0.3:2.7062 0.6:3.5129 0.9:5.0252 | pair 0.1505 tok 0.3764 | 406s +14:52:19 elbo 1000 train=5.9021 (110s) +14:54:14 elbo 2000 train=4.3993 (225s) +14:56:12 elbo 3000 train=3.8315 (344s) +14:58:09 elbo 4000 train=3.7139 (460s) +14:58:09 elbo ce 3.6824 | 0.3:2.5832 0.6:3.4377 0.9:5.0264 | pair 0.1606 tok 0.3823 | 460s +15:00:37 muon.02 1000 train=5.1154 (142s) +15:02:51 muon.02 2000 train=3.9587 (277s) +15:04:51 muon.02 3000 train=3.7960 (396s) +15:06:22 muon.02 4000 train=3.6844 (487s) +15:06:22 muon.02 ce 3.3524 | 0.3:2.1389 0.6:3.0555 0.9:4.8627 | pair 0.2013 tok 0.4269 | 487s +15:08:49 muon.05 1000 train=5.9116 (142s) +15:11:05 muon.05 2000 train=5.6896 (277s) +15:13:21 muon.05 3000 train=5.6586 (413s) +15:15:34 muon.05 4000 train=5.6134 (547s) +15:15:34 muon.05 ce 5.6264 | 0.3:5.4918 0.6:5.6091 0.9:5.7782 | pair 0.0021 tok 0.0750 | 547s +15:15:39 --- versus base --- +15:15:39 base dCE +0.0000 dpair +0.0000 dtok +0.0000 time x1.00 +15:15:39 elbo dCE -0.0657 dpair +0.0101 dtok +0.0060 time x1.13 +15:15:39 muon.02 dCE -0.3957 dpair +0.0508 dtok +0.0505 time x1.20 +15:15:39 muon.05 dCE +1.8783 dpair -0.1484 dtok -0.3014 time x1.35 +15:15:40 published +15:15:40 SPEED PILOT DONE +15:19:59 pilot: 4000 steps, bs 64, lr 3e-04, span 0.50, arms base,muon.01,elbo+muon +15:21:19 base 1000 train=5.9347 (79s) +15:22:56 base 2000 train=4.7405 (176s) +15:24:48 base 3000 train=4.1446 (288s) +15:26:41 base 4000 train=4.0345 (400s) +15:26:41 base ce 3.7481 | 0.3:2.7062 0.6:3.5129 0.9:5.0252 | pair 0.1505 tok 0.3764 | 401s +15:29:07 muon.01 1000 train=5.2554 (141s) +15:31:21 muon.01 2000 train=3.8364 (275s) +15:33:34 muon.01 3000 train=3.6132 (408s) +15:35:38 muon.01 4000 train=3.4897 (532s) +15:35:38 muon.01 ce 3.1766 | 0.3:1.9299 0.6:2.8305 0.9:4.7693 | pair 0.2410 tok 0.4633 | 532s +15:38:06 elbo+muon 1000 train=4.9167 (143s) +15:40:23 elbo+muon 2000 train=3.6558 (281s) +15:42:42 elbo+muon 3000 train=3.4897 (419s) +15:44:59 elbo+muon 4000 train=3.3686 (556s) +15:44:59 elbo+muon ce 3.3307 | 0.3:2.0687 0.6:3.0293 0.9:4.8940 | pair 0.2032 tok 0.4341 | 556s +15:45:04 --- versus base --- +15:45:04 base dCE +0.0000 dpair +0.0000 dtok +0.0000 time x1.00 +15:45:04 muon.01 dCE -0.5715 dpair +0.0905 dtok +0.0869 time x1.33 +15:45:04 elbo+muon dCE -0.4174 dpair +0.0527 dtok +0.0577 time x1.39 +15:45:05 published +15:45:05 SPEED PILOT DONE +15:57:45 pilot: 4000 steps, bs 64, lr 3e-04, span 0.50, arms muon.005,muon.0025,elbo+muon.01 +15:59:23 muon.005 1000 train=5.6731 (97s) +16:01:18 muon.005 2000 train=4.0277 (211s) +16:03:30 muon.005 3000 train=3.6820 (343s) +16:05:41 muon.005 4000 train=3.5531 (475s) +16:05:41 muon.005 ce 3.2499 | 0.3:2.0451 0.6:2.9198 0.9:4.7847 | pair 0.2293 tok 0.4538 | 475s +16:08:04 muon.0025 1000 train=6.0100 (137s) +16:10:13 muon.0025 2000 train=5.0753 (266s) +16:12:23 muon.0025 3000 train=4.2618 (397s) +16:14:19 muon.0025 4000 train=4.1084 (512s) +16:14:19 muon.0025 ce 3.8377 | 0.3:2.8222 0.6:3.6363 0.9:5.0546 | pair 0.1284 tok 0.3547 | 512s +16:16:09 elbo+muon.01 1000 train=5.0888 (106s) +16:18:24 elbo+muon.01 2000 train=3.5406 (240s) +16:20:39 elbo+muon.01 3000 train=3.3056 (376s) +16:22:54 elbo+muon.01 4000 train=3.1724 (510s) +16:22:54 elbo+muon.01 ce 3.1736 | 0.3:1.9033 0.6:2.8353 0.9:4.7822 | pair 0.2343 tok 0.4602 | 510s +16:22:59 --- versus base --- +16:22:59 muon.005 dCE +0.0000 dpair +0.0000 dtok +0.0000 time x1.00 +16:22:59 muon.0025 dCE +0.5878 dpair -0.1009 dtok -0.0991 time x1.08 +16:22:59 elbo+muon.01 dCE -0.0763 dpair +0.0050 dtok +0.0064 time x1.08 diff --git a/pilots/strat_pilot.json b/pilots/strat_pilot.json new file mode 100644 index 0000000000000000000000000000000000000000..567c932924dc38792a0f700ff13ddcab52c0bea8 --- /dev/null +++ b/pilots/strat_pilot.json @@ -0,0 +1,42 @@ +[ + { + "ce@0.3": 2.541595935821533, + "ce@0.6": 3.409759521484375, + "pair": 0.16301322199025747, + "tok": 0.3892141756548536, + "ce@0.9": 5.002624988555908, + "ce": 3.651326815287272, + "tag": "base", + "secs": 417.0274459819998 + }, + { + "ce@0.3": 2.5500025749206543, + "ce@0.6": 3.4197211265563965, + "pair": 0.15414057063326375, + "tok": 0.3827426810477658, + "ce@0.9": 5.0042619705200195, + "ce": 3.6579952239990234, + "tag": "strat", + "secs": 428.79050159300004 + }, + { + "ce@0.3": 2.5832340717315674, + "ce@0.6": 3.443695068359375, + "pair": 0.15605427974947808, + "tok": 0.38685156651258346, + "ce@0.9": 5.03913688659668, + "ce": 3.6886886755625405, + "tag": "comp", + "secs": 415.56753008500027 + }, + { + "ce@0.3": 2.543626308441162, + "ce@0.6": 3.4074501991271973, + "pair": 0.16701461377870563, + "tok": 0.38849512069851055, + "ce@0.9": 5.019223213195801, + "ce": 3.6567665735880532, + "tag": "strat+comp", + "secs": 431.1185980959999 + } +] \ No newline at end of file diff --git a/pilots/strat_pilot.log b/pilots/strat_pilot.log new file mode 100644 index 0000000000000000000000000000000000000000..3579bb6cd2bcc8acb6d7249cd508672a267922ac --- /dev/null +++ b/pilots/strat_pilot.log @@ -0,0 +1,10 @@ +19:27:33 strat pilot: 4000 steps, bs 64, lr 3e-04, arms base,strat,comp,strat+comp +19:34:31 base ce 3.6513 | 0.3:2.5416 0.6:3.4098 0.9:5.0026 | pair 0.1630 tok 0.3892 | 417s +19:41:45 strat ce 3.6580 | 0.3:2.5500 0.6:3.4197 0.9:5.0043 | pair 0.1541 tok 0.3827 | 429s +19:48:46 comp ce 3.6887 | 0.3:2.5832 0.6:3.4437 0.9:5.0391 | pair 0.1561 tok 0.3869 | 416s +19:56:03 strat+comp ce 3.6568 | 0.3:2.5436 0.6:3.4075 0.9:5.0192 | pair 0.1670 tok 0.3885 | 431s +19:56:08 --- versus base (comp arms see half as many distinct rows, at equal compute) --- +19:56:08 base dCE +0.0000 dpair +0.0000 time x1.00 +19:56:08 strat dCE +0.0067 dpair -0.0089 time x1.03 +19:56:08 comp dCE +0.0374 dpair -0.0070 time x1.00 +19:56:08 strat+comp dCE +0.0054 dpair +0.0040 time x1.03 diff --git a/pretrain_25800.pt b/pretrain_25800.pt new file mode 100644 index 0000000000000000000000000000000000000000..9dd0582a7d0596ecd4025554c6ec6c9d76ccf1cc --- /dev/null +++ b/pretrain_25800.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:759cd30b2cfd273b4018501acf6a30f3314eef470d4b35a4280de61324709e12 +size 2152005481