Add model card and LICENSE; remove training logs and internal step counter

LICENSE

@@ -0,0 +1,63 @@
+My Name is Riemann
+Copyright (c) 2026 OpenPeer AI, Riemann Computing Inc., and Andrew Magdy Kamal
+
+This project is released under two licenses:
+
+1. The Open Innovation License, Version 2 (OPNL-2), for the project as a
+   whole, including the model weights, data formats, configuration files,
+   and documentation. Reproduced verbatim below, as the license requires.
+   Canonical source: https://github.com/OPNL/OPNL/tree/main/Version%202
+
+2. The MIT License, for the software implementation (the contents of
+   riemann/, scripts/, and configs/). Full text below.
+
+The OpenPeerLLM base model (https://huggingface.co/OpenPeerAI/OpenPeerLLM,
+DOI 10.57967/hf/6469) is itself released under OPNL, MIT and CC BY 4.0.
+Training data retains the licenses of its respective sources as listed in
+README.md. Third-party reference code under vendor/ retains its original
+licenses and is not covered by this file.
+
+======================================================================
+
+# The Open Innovation License
+*Version 2, 28th December 2020*\
+*Copyright © 2020 Stark Drones Corporation*\
+*Copyright © 2020 Andrew Magdy Kamal*
+
+## Preamble
+
+The *Stark Drones Corporation* believes in building or releasing technology for the betterment of humanity. Technology should not be meant with the intention of harming a human being. We believe in a prima facie moral duty through personal moral obligation to understand that technology should be within the concept of moral good or outcomes that are morally right and/or ethical. We believe in promoting the advancement of humanity and civilization as a whole. We believe in a sense of adventurement, edification, and the expansion of the human mind. 
+
+**Released under the Open Innovation License**
+
+*Copyright © 2026 OpenPeer AI, Riemann Computing Inc., and Andrew Magdy Kamal*
+
+This project is licensed under the *Open Innovation License*. This means any code, file, diagrams, data format, or other innovation containing this license within it can be copied, modified, redistributed, published, or even used for non and/or commercial purposes within the context of this license. 
+
+##### Any code, file, diagrams, data format, or other innovation containing this license is understood to be fully "AS IS", no claims are made in regards to safety, security, warranty, usability, or other form of merchantability and market-readiness. In no events are copyright holders, authors, or publishers are to be held liable for any claims, damage or results from usage of what have been licensed under this license.
+
+The context of this license includes: Keeping this original license text and file verbatim, as well as the copyright notice included in any redistribution of said project. Project is defined as what is using this license. For purposes of context, the copyright notice after the preamble is meant to be modified for whomsoever publishes or releases "any code, file, diagrams, data format, or other innovation", so that they can include their information.
+
+======================================================================
+
+MIT License
+
+Copyright (c) 2026 OpenPeer AI, Riemann Computing Inc., and Andrew Magdy Kamal
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,828 @@
+---
+language:
+  - en
+  - ar
+license: other
+license_name: opnl-2-and-mit
+license_link: LICENSE
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - amd
+  - ntk
+  - code
+  - fsdp
+  - rocm
+  - coding
+  - mi325x
+  - decoder
+  - code gen
+  - directml
+  - openpeer
+  - ryzen-ai
+  - reasoning
+  - rewriting
+  - multimodal
+  - paraphrase
+  - prerelease
+  - ntk-trainer
+  - openpeerllm
+  - safetensors
+  - transformers
+  - growth-ladder
+  - style-transfer
+  - text-rewriting
+  - riemann-computing
+  - progressive-growth
+base_model: OpenPeerAI/OpenPeerLLM
+datasets:
+  - MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x
+  - bakrianoo/jabarti-llm-dataset
+  - HelenPeng/ExeWorldBench
+  - code-video-model/CodeVideoBench
+  - Tiiny/PowerCoding
+model-index:
+  - name: My Name is Riemann Prerelease
+    results:
+      - task:
+          type: text-generation
+          name: Held-out next-token prediction
+        dataset:
+          type: bakrianoo/jabarti-llm-dataset
+          name: My Name is Riemann NTK validation slice, base weights
+          config: stage6_12.85b-base
+          split: validation
+        metrics:
+          - type: nll
+            value: 1.3228
+            name: Validation NLL (1,000 held-out jabarti finetune and Fable first-turn pairs), frozen stage 6 weights
+          - type: accuracy
+            value: 67.73
+            name: Validation next-token accuracy (%), frozen stage 6 weights
+      - task:
+          type: text-generation
+          name: Held-out next-token prediction
+        dataset:
+          type: bakrianoo/jabarti-llm-dataset
+          name: My Name is Riemann NTK validation slice, with NTK gate controller
+          config: stage6_12.85b-ntk
+          split: validation
+        metrics:
+          - type: nll
+            value: 1.3104
+            name: Validation NLL (1,000 held-out jabarti finetune and Fable first-turn pairs), stage 6 weights plus 6,732-gate controller
+          - type: accuracy
+            value: 67.84
+            name: Validation next-token accuracy (%), stage 6 weights plus controller
+      - task:
+          type: text-generation
+          name: Pretraining
+        dataset:
+          type: MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x
+          name: My Name is Riemann pretraining corpus (five datasets, 2.30B GPT-2 tokens)
+          config: stage6_12.85b
+          split: train
+        metrics:
+          - type: loss
+            value: 1.5888
+            name: Final training cross-entropy at step 2,064 (2.16B tokens, bf16, 1,048,576 tokens per step)
+---
+
+# My Name is Riemann Prerelease
+
+My Name is Riemann is a 13.1 billion parameter decoder-only
+language model grown rung by rung from the published weights of
+[OpenPeerAI/OpenPeerLLM](https://huggingface.co/OpenPeerAI/OpenPeerLLM)
+and fine-tuned with the
+[OpenPeer-AI-NTK-Trainer](https://huggingface.co/OpenPeerAI/OpenPeer-AI-NTK-Trainer)
+gate controller instead of LoRA. It is the first public release of the My
+Name is Riemann line and is published as a prerelease: complete and
+loadable, with the final 13.1B stage trained on 2.16 billion tokens from a
+2.30 billion token corpus, and substantially more training planned for
+later releases.
+
+## Model Details
+
+### Model Description
+
+My Name is Riemann starts from OpenPeerLLM's checkpoint (163M parameters,
+loaded tensor for tensor into a faithful reimplementation of its
+architecture) and climbs a ten rung growth ladder. At each rung the
+previous checkpoint is widened and deepened into the next shape, so that
+every stage inherits what the one before it learned, and training
+continues on a mixed corpus of agentic coding transcripts, Arabic and
+English encyclopedia text, executable world-model tasks, code-to-video
+pairs and raw source code. The Prerelease is stage 6 of that ladder.
+
+Fine-tuning uses the OpenPeer-AI-NTK-Trainer: a few thousand signed
+log-gates are fitted on the frozen model's layer outputs with an objective
+derived from neural tangent kernel theory. The base weights are never
+modified. The fitted controller is a small separate file that ships beside
+the checkpoint.
+
+The model is named after Riemann Computing.
+
+- **Developed by:** OpenPeer AI, Riemann Computing Inc., and Andrew Magdy Kamal
+- **Model type:** Decoder-only causal language model (OpenPeerLLM architecture, `RiemannForCausalLM`)
+- **Language(s) (NLP):** English, Arabic, and source code
+- **License:** Open Innovation License, Version 2 (OPNL-2) and MIT
+- **Finetuned from model:** [OpenPeerAI/OpenPeerLLM](https://huggingface.co/OpenPeerAI/OpenPeerLLM)
+- **Release status:** Prerelease (stage 6 of the growth ladder)
+- **Parameters:** 13,110,702,080 (13.1B), fp32, 52.4 GB in eleven safetensors shards
+- **Training tokens:** 2.16B at stage 6, 6.5B cumulative across the ladder
+- **Trained:** 19 to 20 October 2026
+
+### Model Sources
+
+- **Repository:** This Hugging Face repository (model weights, NTK controller, remote code, training pipeline)
+- **Base model:** [OpenPeerAI/OpenPeerLLM](https://huggingface.co/OpenPeerAI/OpenPeerLLM), DOI [10.57967/hf/6469](https://doi.org/10.57967/hf/6469)
+- **Fine-tuning method:** [OpenPeerAI/OpenPeer-AI-NTK-Trainer](https://huggingface.co/OpenPeerAI/OpenPeer-AI-NTK-Trainer) and [ntkmirror](https://github.com/leochlon/ntkmirror)
+
+## Uses
+
+### Direct Use
+
+Text generation with an emphasis on code and step-by-step reasoning, in
+English and Arabic. The training mix is dominated by agentic coding
+transcripts that include tool calls and tool outputs, so the model is best
+suited to code completion, code explanation, multi-step problem solving
+with explicit intermediate reasoning, and bilingual encyclopedic question
+answering.
+
+### Downstream Use
+
+The checkpoint is a suitable base for continued pretraining on a larger
+corpus, for fitting additional NTK gate controllers on task-specific
+prompt and completion data, or for conventional fine-tuning. Because every
+rung of the growth ladder is a standard `save_pretrained()` folder, the
+intermediate stages can also be used directly where a smaller model is
+preferred.
+
+### Out-of-Scope Use
+
+The Prerelease is not an instruction-tuned assistant and has not been
+aligned or safety-tuned. It should not be deployed in settings that
+require factual reliability, in high-stakes decision making, or to
+generate content in languages other than English and Arabic. The
+multimodal pathway is a demonstration of the wiring, not a capable video
+model.
+
+## Bias, Risks, and Limitations
+
+- **Prerelease token budget.** Stage 6 has seen 2.16 billion tokens,
+  about 94 percent of one pass over the 2.30 billion token corpus. That is
+  far below a compute-optimal budget for a 13B model and is the main thing
+  later releases will change. Expect capability well below similarly sized
+  models trained on hundreds of billions of tokens.
+- **Uneven corpus coverage.** The corpus is read in file order, so stages 2
+  to 5 (1.08 billion tokens each) saw mainly the first half of the file,
+  which is the Arabic and English Wikipedia block. The stage 6 run was
+  configured as two passes of an estimated 1.08 billion tokens; because
+  the corpus is in fact 2.30 billion tokens, that run instead covered the
+  Fable transcripts, PowerCoding and ExeWorldBench once and the Wikipedia
+  block roughly twice. Training loss on the second visit to the Wikipedia
+  block dropped as low as 0.12, which reflects memorisation of repeated
+  text rather than generalisation; the held-out validation figures in the
+  Evaluation section are the meaningful quality measure. Later releases
+  shuffle document order per pass.
+- **Narrow data mix.** Five datasets, dominated by synthetic agentic coding
+  traces (Fable 5.1), Arabic and English Wikipedia (jabarti), and synthetic
+  Python (PowerCoding). The model inherits the biases, errors and gaps of
+  those sources and of the models that generated the synthetic portions.
+- **Inherited behaviour.** The lowest rung is OpenPeerLLM's own checkpoint,
+  so any properties of that model carry forward.
+- **Context length.** 2,048 tokens at stage 6 (1,024 through stage 3).
+- **No alignment.** The model has had no RLHF, DPO or safety tuning.
+
+### Recommendations
+
+Users should treat outputs as drafts requiring review, apply their own
+content filtering before any user-facing deployment, and prefer further
+training for production use. The NTK controller shipped with the
+checkpoint is fitted on question answering pairs and improves held-out NLL
+on that distribution; it does not add safety behaviour.
+
+## How to Get Started with the Model
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+repo = "<repo-id>"
+tokenizer = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(
+    repo, trust_remote_code=True, torch_dtype=torch.bfloat16
+)
+
+prompt = "user: Write a function that checks whether a number is prime.\nassistant:"
+inputs = tokenizer(prompt, return_tensors="pt")
+output = model.generate(**inputs, max_new_tokens=200, do_sample=True, top_p=0.9)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+`trust_remote_code=True` is required because the repository ships its own
+`configuration_riemann.py` and `modeling_riemann.py`. Weights are stored in
+fp32 (about 52 GB); loading in bf16 halves the memory footprint.
+
+To apply the NTK gate controller:
+
+```python
+from ntkmirror import ForwardFineTuner
+
+tuner = ForwardFineTuner(model, tokenizer)
+tuner.load("ntk_controller/stage6_12.85b.pt", allow_model_mismatch=True)
+tuner.controller.attach()
+```
+
+## Training Details
+
+### Training Data
+
+Five datasets are combined into a single pretraining corpus of 7,218,101
+documents (2.30 billion GPT-2 tokens, 318 tokens per document on
+average), plus a separate file of 73,824 prompt and completion pairs used
+to fit the NTK controller.
+
+| Dataset | Portion used | Role |
+|---|---|---|
+| [MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x](https://huggingface.co/datasets/MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x) | All 10,000 transcripts | Pretraining; first user turn and final assistant turn form an NTK pair |
+| [bakrianoo/jabarti-llm-dataset](https://huggingface.co/datasets/bakrianoo/jabarti-llm-dataset) | `pretrain` split in full (7,053,893 documents) and `finetune` split in full (about 64,000 pairs) | Pretraining; finetune pairs also feed the NTK controller |
+| [HelenPeng/ExeWorldBench](https://huggingface.co/datasets/HelenPeng/ExeWorldBench) | Full test split, 779 tasks | Pretraining |
+| [code-video-model/CodeVideoBench](https://huggingface.co/datasets/code-video-model/CodeVideoBench) | All 22 cases | Prompts and code in pretraining; videos feed the multimodal pathway |
+| [Tiiny/PowerCoding](https://huggingface.co/datasets/Tiiny/PowerCoding) | About 90,000 Python examples from several raw shards | Pretraining |
+
+Fable transcripts are serialised turn by turn with the role as a prefix;
+tool calls are wrapped in `<tool_call name=...>` tags and tool outputs are
+prefixed with the tool name. Text read from these datasets is treated
+strictly as data; nothing in any record is executed or interpreted as an
+instruction by the training code.
+
+### Training Procedure
+
+Training proceeds along a growth ladder (see Technical Specifications for
+the full table). A stage is trained, its checkpoint is grown into the next
+shape, and training continues. Width growth copies existing rows and
+columns into new positions at one tenth scale so the grown network starts
+near the function it already computes; depth growth inserts new blocks
+with zeroed output projections so they begin as identity passthroughs.
+
+The climb to the Prerelease is split across two environments.
+
+**Stage 1 on AMD consumer hardware.** Stage 0 is materialised from
+OpenPeerLLM's published weights and grown to stage 1 (331M parameters).
+Stage 1 trains for 1,500 optimizer steps (about 9 million tokens) on a
+Radeon RX 7600M XT with a Ryzen AI 9 465 CPU running one in four
+micro-batches on a mirrored copy of the model. An NTK controller is fitted
+on the result.
+
+**Stages 2 through 6 on an 8x AMD MI325X node.** At each rung the
+checkpoint is grown, pretrained for 1,032 optimizer steps of 1,048,576
+tokens (1.08 billion tokens) with sharded data parallelism in bf16, and
+given an NTK controller warm-started from the previous rung's controller.
+Stage 6 ran for 2,064 steps (2.16 billion tokens). The whole climb took
+20.4 node-hours including growth, controller fits and the final upload.
+
+| Rung | Parameters | Tokens | Pretrain wall clock | Rung total incl. grow and fit | Training loss, first to last logged step |
+|---|---|---|---|---|---|
+| stage2_538m | 0.60B | 1.08B | 52 min | 55 min | 6.78 to 2.22 |
+| stage3_1.19b | 1.27B | 1.08B | 49 min | 54 min | 2.17 to 1.88 |
+| stage4_2.65b | 2.78B | 1.08B | 101 min | 110 min | 2.83 to 1.72 |
+| stage5_6.66b | 6.86B | 1.08B | 197 min | 216 min | 1.88 to 1.63 |
+| stage6_12.85b | 13.1B | 2.16B | 746 min | 786 min | 2.03 to 1.59 |
+
+Stage 2 ran at roughly 16 percent of the MI325X's bf16 peak, which is
+typical for a sub-1B model where per-step overheads dominate. Stages 3 to
+6 used fused scaled-dot-product attention, gradient-and-optimizer sharding
+instead of full parameter sharding, and no activation recomputation,
+since the node's 256 GB per GPU makes that memory saving unnecessary;
+stage 3 ran 50 percent faster than it would have on the stage 2
+configuration. The full per-step logs for every rung are published in
+`training_logs/` alongside the checkpoint.
+
+The NTK fits are what keep the budget small. Fitting a few thousand gates
+on a frozen model is forward-dominated, takes minutes per rung, and carries
+no optimizer state, where a supervised fine-tuning epoch at 13B would take
+hours with the full training memory footprint. Carrying each rung's gates
+forward means only the grown dimensions need new gates.
+
+#### Preprocessing
+
+Documents are written one per line with internal newlines encoded as
+U+2029. At training time each document is tokenised with the GPT-2
+byte-level BPE tokenizer (50,257 tokens), terminated with the end-of-text
+token, and packed into fixed-length windows. Under distributed training
+each rank reads a disjoint subset of lines.
+
+#### Training Hyperparameters
+
+| | Stage 1 (AMD Radeon, DirectML) | Stages 2 to 6 (AMD MI325X, ROCm) |
+|---|---|---|
+| Training regime | fp32 | bf16 mixed precision, fp32 master weights |
+| Optimizer | Adafactor (relative step, warmup init) | AdamW, betas 0.9 / 0.95, weight decay 0.1 |
+| Learning rate | Adafactor relative | 3e-4 peak, 200 step linear warmup, cosine decay to 3e-5 |
+| Gradient clipping | none | 1.0 |
+| Sequence length | 384 | 1,024 (stages 2, 3), 2,048 (stages 4 to 6) |
+| Tokens per optimizer step | 6,144 | 1,048,576 |
+| Gradient checkpointing | yes | stage 2 only; off from stage 3 onward |
+| Attention kernel | explicit matmul and softmax | fused scaled-dot-product attention |
+| Parallelism | single GPU plus CPU mirror | FSDP across 8 GPUs (full shard at stage 2, gradient and optimizer sharding from stage 3) |
+
+#### Speeds, Sizes, Times
+
+| Stage | Parameters | Checkpoint size (fp32) | Hardware | Throughput | Wall clock |
+|---|---|---|---|---|---|
+| stage1_279m | 331M | 1.3 GB | Radeon RX 7600M XT + Ryzen AI 9 465 | 350 tokens/s (17.5 s/step) | about 7 h for 1,500 steps |
+| stage2_538m | 602M | 2.4 GB | 8x MI325X | 345,600 tokens/s | 52 min for 1,032 steps |
+| stage3_1.19b | 1.27B | 5.1 GB | 8x MI325X | 370,300 tokens/s | 49 min for 1,032 steps |
+| stage4_2.65b | 2.78B | 11.1 GB | 8x MI325X | 178,500 tokens/s | 1 h 41 min for 1,032 steps |
+| stage5_6.66b | 6.86B | 27.5 GB | 8x MI325X | 91,800 tokens/s | 3 h 17 min for 1,032 steps |
+| stage6_12.85b | 13.1B | 52.4 GB | 8x MI325X | 48,400 tokens/s | 12 h 26 min for 2,064 steps |
+
+Stage 6 is 52.4 GB on disk in fp32, saved as eleven safetensors shards.
+Total node time from first boot to final upload was 21.0 hours.
+
+## Evaluation
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+A held-out slice of 1,000 examples from the NTK prompt and completion
+file (drawn from the jabarti finetune split and Fable first-turn pairs),
+never used for controller fitting, is scored before and after each rung's
+controller is fitted. The remaining 19,000 examples are used for fitting.
+Some of this text also appears in the pretraining corpus, so these
+figures measure in-distribution next-token prediction rather than
+generalisation to unseen domains.
+
+#### Factors
+
+Results are reported per rung of the growth ladder, with and without the
+NTK controller attached.
+
+#### Metrics
+
+Negative log-likelihood (NLL) and next-token accuracy on the held-out
+slice. These measure how well the frozen model predicts the fine-tuning
+distribution and how much the controller improves it; they are not a
+general capability benchmark.
+
+### Results
+
+| Rung | Parameters | Gates | NLL, base | NLL, with controller | Token accuracy, base | Token accuracy, with controller |
+|---|---|---|---|---|---|---|
+| stage1_279m | 331M | 4,096 | 7.044 | 7.021 | 11.97% | 12.30% |
+| stage2_538m | 602M | 4,096 | 1.968 | 1.954 | 54.55% | 54.73% |
+| stage3_1.19b | 1.27B | 4,631 | 1.629 | 1.618 | 62.45% | 62.59% |
+| stage4_2.65b | 2.78B | 4,102 | 1.512 | 1.505 | 65.08% | 65.17% |
+| stage5_6.66b | 6.86B | 4,102 | 1.451 | 1.447 | 66.41% | 66.47% |
+| **stage6_12.85b (Prerelease)** | **13.1B** | **6,732** | **1.323** | **1.310** | **67.73%** | **67.84%** |
+
+The stage 1 figures were measured on a 200-example slice during the local
+phase (4,000 fitting examples); stages 2 to 6 use the 1,000-example slice.
+The per-rung JSON summaries in `ntk_controller/` record the same figures
+along with fit time, learning rate and gate counts.
+
+#### Summary
+
+Held-out NLL falls monotonically along the ladder, from 7.04 at stage 1
+to 1.32 at stage 6, and next-token accuracy rises from 12 percent to 68
+percent, with each rung inheriting and extending the previous one. The
+NTK controller gives a consistent small improvement on top of the frozen
+weights at every rung (0.004 to 0.023 nats) at a cost of under 150 KB per
+controller. Broader benchmark results are not yet available for this
+prerelease; users should run their own evaluation before relying on it.
+
+## Environmental Impact
+
+Carbon emissions can be estimated using the
+[Machine Learning Impact calculator](https://mlco2.github.io/impact#compute)
+presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** Stage 1: AMD Radeon RX 7600M XT (eGPU) and AMD Ryzen AI 9 465. Stages 2 to 6: 8x AMD Instinct MI325X 256 GB.
+- **Hours used:** Stage 1: about 7 hours on one laptop with an external GPU. Stages 2 to 6: 21.0 node-hours (168 GPU-hours), of which 19.1 hours were pretraining and the rest growth, controller fitting, setup and upload.
+- **Cloud Provider:** DigitalOcean (stages 2 to 6)
+- **Compute Region:** NYC2 (New York)
+- **Carbon Emitted:** The MI325X ran at its 1,000 W board power cap throughout, so GPU power alone was about 168 kWh (8 GPUs x 21 h x 1 kW); with host and cooling overhead at a typical PUE of 1.3, about 220 kWh total. The node was in New York, where the grid's total output emission rate is 466.6 lb CO2e per MWh, or 0.212 kg per kWh, according to the EPA's eGRID2023 summary tables. At that rate the cloud phase produced roughly 47 kg of CO2e, about the same as a 120 mile drive in an average petrol car. Stage 1 on the laptop and eGPU added about 1.5 kWh.
+
+  Source: U.S. Environmental Protection Agency. (2025). *eGRID2023 Summary Tables* (rev. 2, March 2025), Table 3: State Output Emission Rates. [epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf](https://www.epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf)
+
+## Technical Specifications
+
+### Model Architecture and Objective
+
+OpenPeerLLM architecture, reimplemented as `RiemannForCausalLM` with module
+names matched to the published checkpoint so its 197 tensors load with a
+plain state-dict copy. Pre-norm transformer blocks with separate query,
+key, value and output projections, GELU MLP, untied input and output
+embeddings, learned absolute position embeddings. Two additions over the
+reference implementation: a causal attention mask (required for
+autoregressive training) and `inputs_embeds` support for the multimodal
+prefix. The objective is standard next-token cross-entropy.
+
+The growth ladder:
+
+| Stage | Parameters | Hidden size | Layers | Heads | FFN size | Context |
+|---|---|---|---|---|---|---|
+| stage0_124m | 163M | 768 | 12 | 12 | 3,072 | 1,024 |
+| stage1_279m | 331M | 1,024 | 18 | 16 | 4,096 | 1,024 |
+| stage2_538m | 602M | 1,280 | 24 | 20 | 5,120 | 1,024 |
+| stage3_1.19b | 1.27B | 1,600 | 36 | 25 | 6,400 | 1,024 |
+| stage4_2.65b | 2.78B | 2,560 | 32 | 32 | 10,240 | 2,048 |
+| stage5_6.66b | 6.86B | 4,096 | 32 | 32 | 16,384 | 2,048 |
+| **stage6_12.85b (Prerelease)** | **13.1B** | **5,120** | **40** | **40** | **20,480** | **2,048** |
+| stage7_29.97b | 30.3B | 7,168 | 48 | 56 | 28,672 | 2,048 |
+| stage8_51.97b | 52.4B | 8,192 | 64 | 64 | 32,768 | 2,048 |
+| stage9_64.86b | 65.3B | 8,192 | 80 | 64 | 32,768 | 2,048 |
+
+Parameter counts include both embedding matrices. Stage names use the
+tied-embedding count, which is the convention most published sizes follow.
+Stages 7 to 9 are defined so the same checkpoints and growth code keep
+working for later releases.
+
+**NTK gate controller.** A sparse set of signed log-gates, each bound to
+one decoder layer and one residual-stream channel, that multiply the layer
+output by `exp(s)` with `s` bounded to a small interval. Gates are selected
+by scoring activations on the fine-tuning data, then fitted with AdamW
+against next-token loss while the base model stays frozen.
+
+**Multimodal pathway.** A small convolutional frame encoder samples eight
+frames per CodeVideoBench video, mean-pools their embeddings, and projects
+the result into the model's hidden size as one prefix token. It runs on a
+secondary GPU during training.
+
+**Hybrid CPU and GPU training (stage 1).** The primary GPU runs most
+micro-batches. A CPU mirror of the model runs one in four micro-batches on
+a background thread concurrently with the GPU; its gradients are summed
+into the main model's before each optimizer step and the mirror is
+resynchronised afterwards. DirectML does not split one model across two
+AMD GPUs, so the secondary GPU is given the frame encoder as a separate
+job.
+
+### Compute Infrastructure
+
+#### Hardware
+
+Stage 1:
+
+| Component | Detail |
+|---|---|
+| System | ASUS Zenbook S16 (UM5606GA) |
+| CPU | AMD Ryzen AI 9 465, 10 cores / 20 threads, with integrated NPU |
+| Primary GPU | AMD Radeon RX 7600M XT, 8 GB GDDR6, external eGPU |
+| Secondary GPU | AMD Radeon 880M integrated graphics |
+| System memory | 32 GB LPDDR5X |
+| Storage | 2 TB NVMe |
+
+Stages 2 to 6:
+
+| Component | Detail |
+|---|---|
+| Node | DigitalOcean `gpu-mi325x8-2048gb`: 8x AMD Instinct MI325X, 256 GB HBM3E each (2 TB total GPU memory), Infinity Fabric interconnect |
+| Host | 160 vCPUs, 1.26 TB system memory, 2 TB NVMe |
+| Precision | bf16 compute, fp32 master weights |
+| Parallelism | PyTorch FSDP: full sharding at stage 2, gradient and optimizer state sharding from stage 3 |
+| Checkpoints | fp32 safetensors, 5 GB shards |
+
+Memory guidance for keeping a stage resident on one device (fp32 weights;
+bf16 halves these): stages 0 to 3 fit on an 8 GB GPU with gradient
+checkpointing at batch 1; stage 4 (about 11 GB of weights) needs a 24 GB
+class GPU; stage 5 (about 27 GB) a 48 GB GPU; stage 6 (about 52 GB) trains
+comfortably on 8x 256 GB with parameters resident on every GPU and only
+gradients and optimizer state sharded.
+
+#### Software
+
+- Stage 1: Windows 10 Pro 22H2, PyTorch 2.4.1, torch-directml 0.2.5, transformers 4.56, AMD Adrenalin 32.0.31041
+- Stages 2 to 6: Ubuntu 24.04, ROCm 7.1 host driver, PyTorch 2.9.1 (ROCm 6.4 build), RCCL, transformers 4.57, safetensors, ntkmirror 0.4
+- Tokenizer: GPT-2 byte-level BPE via `transformers`
+
+## Citation
+
+**BibTeX:**
+
+```bibtex
+@misc{my-name-is-riemann-prerelease,
+  author    = {{OpenPeer AI} and {Riemann Computing Inc.} and Kamal, Andrew Magdy},
+  title     = {My Name is Riemann Prerelease},
+  year      = {2026},
+  publisher = {Hugging Face},
+  journal   = {Hugging Face Model Hub},
+  note      = {Stage 6 of the My Name is Riemann growth ladder, 13.1B parameters}
+}
+```
+
+**APA:**
+
+OpenPeer AI, Riemann Computing Inc., & Kamal, A. M. (2026). *My Name is
+Riemann Prerelease* [Language model]. Hugging Face Model Hub.
+
+### Sources cited
+
+Base model and trainer:
+
+- Nassief, A. M. K. (2025). *OpenPeerLLM: A Decentralized Language Model.*
+  Stark Publishing Group, Hugging Face Model Hub.
+  DOI [10.57967/hf/6469](https://doi.org/10.57967/hf/6469). Released
+  under OPNL, MIT and CC BY 4.0.
+
+  ```bibtex
+  @misc{openpeer-llm,
+    author    = {Andrew Magdy Kamal Nassief},
+    title     = {OpenPeerLLM: A Decentralized Language Model},
+    year      = {2025},
+    publisher = {Stark Publishing Group},
+    journal   = {Hugging Face Model Hub},
+    doi       = {10.57967/hf/6469}
+  }
+  ```
+
+- OpenPeer AI. *OpenPeer-AI-NTK-Trainer.*
+  [huggingface.co/OpenPeerAI/OpenPeer-AI-NTK-Trainer](https://huggingface.co/OpenPeerAI/OpenPeer-AI-NTK-Trainer).
+  MIT. Depends on [ntkmirror](https://github.com/leochlon/ntkmirror).
+
+Datasets:
+
+- MoreThought. *Fable-5.1-Max-Reasoning-Filtered-10000x.*
+  [Hub](https://huggingface.co/datasets/MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x).
+  Apache 2.0. No DOI or paper issued.
+- bakrianoo. *jabarti-llm-dataset.*
+  [Hub](https://huggingface.co/datasets/bakrianoo/jabarti-llm-dataset).
+  CC BY-SA 4.0. No DOI or paper issued. Derived in part from
+  [CohereLabs/wikipedia-2023-11-embed-multilingual-v3](https://huggingface.co/datasets/CohereLabs/wikipedia-2023-11-embed-multilingual-v3).
+- Peng, X., Fang, Z., Wang, R., Lv, Y., Li, H., Zhang, G., Ma, X., Lu, Y.,
+  Chen, L., Che, Z., & Zhao, F. (2026). *ExeWorldBench: Benchmarking
+  Code-Based Executable 3D World Modeling.* arXiv:2610.15166.
+  [arxiv.org/abs/2610.15166](https://arxiv.org/abs/2610.15166). Dataset
+  [Hub](https://huggingface.co/datasets/HelenPeng/ExeWorldBench), CC BY 4.0.
+
+  ```bibtex
+  @article{peng2026exeworldbench,
+    author  = {Peng, Xinge and Fang, Zhen and Wang, Ruibing and Lv, Yongxuan and Li, Haodong and Zhang, Guohui and Ma, Xiaoxiao and Lu, Yiting and Chen, Lin and Che, Zehui and Zhao, Feng},
+    title   = {ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling},
+    journal = {arXiv preprint arXiv:2610.15166},
+    year    = {2026}
+  }
+  ```
+
+- code-video-model. *CodeVideoBench.*
+  [Hub](https://huggingface.co/datasets/code-video-model/CodeVideoBench).
+  Cases and baseline controls CC BY 4.0, project code MIT. No DOI or paper
+  issued.
+- Tiiny. *PowerCoding.*
+  [Hub](https://huggingface.co/datasets/Tiiny/PowerCoding). Apache 2.0.
+  No DOI or paper issued for the dataset. Its card states the data was
+  synthesised with Qwen 2.5 models using the methods of:
+  - Hao, X., Zhu, R., Zhang, G., Shen, K., & Li, C. (2025).
+    *Reformulation for Pretraining Data Augmentation.* arXiv:2502.04235.
+    [arxiv.org/abs/2502.04235](https://arxiv.org/abs/2502.04235).
+  - Ge, T., Chan, X., Wang, X., Yu, D., Mi, H., & Yu, D. (2024). *Scaling
+    Synthetic Data Creation with 1,000,000,000 Personas.* arXiv:2406.20094.
+    [arxiv.org/abs/2406.20094](https://arxiv.org/abs/2406.20094).
+  - Zhu, Y., Zhou, K., Mao, K., et al. (2024). *YuLan: An Open-source
+    Large Language Model.* arXiv:2406.19853.
+    [arxiv.org/abs/2406.19853](https://arxiv.org/abs/2406.19853).
+
+Tokenizer:
+
+- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I.
+  (2019). *Language Models are Unsupervised Multitask Learners.* OpenAI.
+
+License:
+
+- Stark Drones Corporation & Kamal, A. M. (2020). *The Open Innovation
+  License, Version 2.*
+  [github.com/OPNL/OPNL/tree/main/Version%202](https://github.com/OPNL/OPNL/tree/main/Version%202).
+
+Environmental impact:
+
+- U.S. Environmental Protection Agency. (2025). *eGRID2023 Summary
+  Tables* (rev. 2, March 2025), Table 3: State Output Emission Rates.
+  [epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf](https://www.epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf).
+- Lacoste, A., Luccioni, A., Schmidt, V., & Dandres, T. (2019).
+  *Quantifying the Carbon Emissions of Machine Learning.* arXiv:1910.09700.
+  [arxiv.org/abs/1910.09700](https://arxiv.org/abs/1910.09700).
+
+## Glossary
+
+- **Growth ladder.** The sequence of ten model shapes, each produced by
+  widening and deepening the previous stage's trained checkpoint.
+- **Rung / stage.** One shape on the ladder, named by its approximate
+  tied-embedding parameter count (for example `stage6_12.85b`).
+- **NTK gate controller.** The small set of per-layer, per-channel
+  multiplicative gates fitted by the OpenPeer-AI-NTK-Trainer on a frozen
+  model; the project's replacement for LoRA.
+- **Warm start.** Initialising a rung's controller from the previous
+  rung's fitted gates before scoring new gates for the grown dimensions.
+- **Prerelease.** The first public build of the line, at stage 6, with a
+  deliberately small token budget relative to its size.
+
+## More Information
+
+### Release history
+
+| Release | Ladder stage | Parameters | Training tokens | Status |
+|---|---|---|---|---|
+| My Name is Riemann Prerelease | stage6_12.85b | 13.1B | 2.16B at stage 6; 6.5B cumulative across stages 1 to 6 | This release |
+
+Later releases continue training the stage 6 checkpoint on a larger
+corpus and may climb to stage 7 and beyond. Each will be published under
+its own name with the Prerelease kept available.
+
+### License
+
+Released under the
+[Open Innovation License, Version 2](https://github.com/OPNL/OPNL/tree/main/Version%202)
+(OPNL-2) for the project as a whole, including model weights,
+configuration and documentation, and under the MIT License for the
+software implementation. OPNL-2 permits copying, modification,
+redistribution and commercial use on the condition that the license text
+and copyright notice are kept verbatim in any redistribution. Both texts
+are in `LICENSE`.
+
+### Reproducing the training
+
+The repository contains the complete pipeline. Set `RIEMANN_ROOT` to a
+directory with enough free space; data, checkpoints, controllers and the
+download cache all live under it.
+
+Data:
+
+```bash
+python -m riemann.data
+```
+
+Stage 1 on a single AMD GPU:
+
+```bash
+python scripts/pretrain.py --stage stage0_124m --max-steps 0
+python scripts/grow_and_continue.py --from-stage stage0_124m --to-stage stage1_279m
+python scripts/supervise.py --stage stage1_279m --max-steps 1500 -- --batch-size 1 --grad-accum 16 --save-every 50
+python scripts/fit_ntk.py --stage stage1_279m
+```
+
+The supervisor relaunches `pretrain.py --resume` from the last saved
+checkpoint if the process exits early and waits for the primary GPU to
+reappear if it has dropped off the bus, which external GPUs occasionally
+do under sustained load. Arguments after `--` are forwarded to
+`pretrain.py`.
+
+Stages 2 to 6 on an 8-GPU node (ROCm or CUDA), after copying `data/`,
+`checkpoints/stage1_279m/` and `ntk_controllers/stage1_279m.pt` under
+`RIEMANN_ROOT` on the node and installing PyTorch for the node's GPU
+backend plus `requirements-cuda.txt` (minus `torch`):
+
+```bash
+python scripts/grow_and_continue.py --from-stage stage1_279m --to-stage stage2_538m
+torchrun --nproc_per_node 8 scripts/pretrain_cuda.py --stage stage2_538m --epochs 1
+python scripts/fit_ntk.py --stage stage2_538m --init-from stage1_279m
+```
+
+Repeat for `stage2_538m -> stage3_1.19b`, `stage3_1.19b -> stage4_2.65b`,
+`stage4_2.65b -> stage5_6.66b` and `stage5_6.66b -> stage6_12.85b`, using
+`--epochs 2` on the final rung. On GPUs with 128 GB or more, add
+`--sharding grad_op --no-activation-checkpointing` for a substantial
+speedup. `cloud/bootstrap.sh` runs this whole sequence unattended,
+including a short FSDP smoke test before the first rung.
+
+### Command-line flags
+
+`pretrain.py` (single device, DirectML or CUDA or CPU):
+
+- `--batch-size`, `--grad-accum`, `--seq-len` override the stage defaults.
+- `--cpu-micro-step-every N` sends one in every N micro-batches to the CPU
+  mirror; `--no-cpu-accel` disables the CPU lane; `--cpu-threads` caps the
+  thread count.
+- `--multimodal` enables the video pathway; `--multimodal-every N` controls
+  how often a video example is interleaved.
+- `--save-every`, `--log-every`, `--max-steps`, `--resume`.
+- `--no-pretrained-init` builds stage 0 from random weights instead of
+  OpenPeerLLM's published checkpoint.
+
+`pretrain_cuda.py` (multi-GPU, launched with `torchrun`):
+
+- `--epochs N` passes over the corpus; `--max-steps` as an alternative cap.
+- `--batch-size`, `--grad-accum`, `--seq-len` per GPU.
+- `--lr`, `--warmup-steps`, `--weight-decay`, `--grad-clip`.
+- `--save-every`, `--log-every`, `--max-shard-size`.
+
+`fit_ntk.py`:
+
+- `--init-from STAGE` warm-starts from a previous rung's controller.
+- `--gates`, `--steps`, `--lr`, `--max-examples`, `--val-fraction`,
+  `--early-stop-patience` control the fit.
+- `--cpu` runs the fit on the CPU so a GPU can keep pretraining.
+
+`riemann.data`:
+
+- `--only name [name ...]` runs a subset of loaders.
+- `--append` adds to the existing corpus files instead of replacing them.
+- `--jabarti-skip N` resumes the Arabic pretrain stream from row N.
+
+### Repository notes
+
+The source files in `riemann/` and `scripts/` are kept free of comments and
+docstrings by convention; everything that would normally be a comment is
+recorded here instead.
+
+**`riemann/configuration_riemann.py`, `riemann/modeling_riemann.py`.** The
+`RiemannConfig` and `RiemannForCausalLM` classes described under Technical
+Specifications. `RiemannForCausalLM` mixes in `GenerationMixin` so
+`generate()` works on current transformers releases.
+
+**`riemann/openpeer_weights.py`.** Downloads OpenPeerLLM's
+`model.safetensors` and copies it into a freshly built stage 0 model,
+reporting any missing or unexpected keys.
+
+**`riemann/growth_ladder.py`, `configs/growth_ladder.json`.** The
+`GrowthStage` dataclass and the ten-rung ladder. The JSON file is generated
+from the Python definitions and contains no free-text fields.
+`approx_param_count()` includes both embedding matrices;
+`to_riemann_config()` maps a stage onto a `RiemannConfig`.
+
+**`riemann/growth.py`.** `grow_width` expands every linear, embedding and
+layer-norm tensor by copying randomly chosen existing rows or columns into
+the new positions at one tenth scale. `grow_depth` appends new transformer
+blocks with zeroed `attention.out` and `mlp.dense_4h_to_h` weights.
+`grow_to` applies width then depth. `load_checkpoint_and_grow` reads a
+saved stage and returns the grown model.
+
+**`riemann/device.py`.** `main_device()` returns the first DirectML
+adapter, otherwise the first CUDA device, otherwise CPU.
+`secondary_device()` returns the second adapter of the same backend or
+falls back to the first. `configure_cpu_threads()` sets PyTorch's intra-op
+and inter-op pools to all logical cores.
+
+**`riemann/paths.py`.** Every on-disk location derives from `RIEMANN_ROOT`.
+
+**`riemann/dataset.py`.** `PackedTextDataset` streams the corpus,
+tokenises, appends end-of-text, and packs into fixed windows. It takes a
+shard index and count so each distributed rank reads a disjoint subset of
+lines, composed with PyTorch's per-worker split. It also takes an epoch
+number: epoch 0 reads the file in order (as the Prerelease was trained),
+and every later epoch rotates the shard assignment and shuffles documents
+through a seeded reservoir buffer so a second pass sees a different
+ordering and no rank repeats its own slice. This was added after the
+Prerelease run and applies to later releases.
+
+**`scripts/count_corpus_tokens.py`.** Tokenises the whole corpus once and
+writes `corpus_token_count.json` with the exact token total and
+per-document average, which `pretrain_cuda.py` uses to size an epoch.
+
+**`riemann/data.py`.** One loader per dataset. Fable is read from its
+single JSONL shard; the Arabic pretrain split is streamed and resumable
+from any row; PowerCoding is read by HTTP byte range so only the needed
+prefix of each shard is downloaded. Output goes through
+`ResilientWriter`, which reopens the destination in append mode and
+retries with backoff if the volume disconnects mid-write. Hub downloads
+use the same retry policy.
+
+**`riemann/multimodal.py`.** `VideoPrefixAdapter` holds the frame encoder
+and the projection into the model's hidden size;
+`build_multimodal_training_examples()` decodes CodeVideoBench's videos and
+pairs them with their prompts.
+
+**`scripts/pretrain.py`.** Single-device training. Loads or builds the
+stage, optionally constructs the CPU mirror, and runs the packed-corpus
+loop. Checkpoints are saved after a round trip to CPU because DirectML
+tensors cannot be serialised directly; `training_state.json` records the
+stage and global step so `--resume` continues the counter.
+
+**`scripts/pretrain_cuda.py`.** Multi-GPU training under `torchrun`. Wraps
+each transformer block in FSDP with full sharding, bf16 compute and fp32
+master weights, applies activation checkpointing per block, shards the
+corpus across ranks, and trains with AdamW under linear warmup and cosine
+decay for a fixed number of corpus passes. Rank 0 gathers the full state
+dict and writes sharded fp32 safetensors in the same layout `pretrain.py`
+produces.
+
+**`scripts/supervise.py`.** A restart loop around `pretrain.py` for
+single-device runs: reads the last saved step, waits for the primary GPU,
+launches `pretrain.py --resume`, and repeats until the target step is
+reached.
+
+**`scripts/grow_and_continue.py`.** Grows a finished stage into a later
+one and writes the new checkpoint plus tokenizer.
+
+**`scripts/fit_ntk.py`.** Drives `ntkmirror`'s `ForwardFineTuner` on a
+`RiemannForCausalLM` checkpoint. Holds out a validation slice, scores and
+selects gates, fits with early stopping and best-on-validation selection,
+and saves the controller with a JSON summary. With `--init-from`, the
+previous rung's gates are carried onto the grown model and fresh gates are
+scored only for the new layers and channels.
+
+**`scripts/push_to_hub.py`.** Copies the two remote-code files into the
+checkpoint folder, writes an `auto_map` into `config.json`, and uploads
+the checkpoint (resumable for large folders), the controller and its
+summary, this card and the license.
+
+**`vendor/`.** Reference copies of the upstream OpenPeerLLM source and the
+NTK trainer. Third-party files, not modified.
+
+## Authors
+
+OpenPeer AI, Riemann Computing Inc., and Andrew Magdy Kamal
+
+## Contact
+
+Open a discussion on this model's Hugging Face repository.

training_logs/ntk_stage2_538m.log

@@ -1,33 +0,0 @@
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-2026-10-20 00:08:06,042 INFO Loading stage2_538m from /workspace/RiemannModel/checkpoints/stage2_538m onto cuda:0
-2026-10-20 00:08:07,240 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
-Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-20 00:08:10,110 INFO Warm start from stage1_279m.pt: carried 4096 of 4096 gates, added 0 for grown dims, 4096 total (18 layers, hidden 1024 -> 24 layers, hidden 1280)
-2026-10-20 00:08:18,150 INFO Validation before fitting: nll 1.9682, token acc 0.5455
-step    1/240: loss=1.602540 task=1.602540
-validation step   20: nll=1.966694 token_acc=0.5458
-validation step   40: nll=1.965223 token_acc=0.5460
-step   48/240: loss=1.277483 task=1.277483
-validation step   60: nll=1.963804 token_acc=0.5463
-validation step   80: nll=1.962513 token_acc=0.5464
-step   96/240: loss=1.726861 task=1.726861
-validation step  100: nll=1.961262 token_acc=0.5464
-validation step  120: nll=1.959995 token_acc=0.5465
-validation step  140: nll=1.958817 token_acc=0.5467
-step  144/240: loss=1.582556 task=1.582556
-validation step  160: nll=1.957703 token_acc=0.5468
-validation step  180: nll=1.956681 token_acc=0.5469
-step  192/240: loss=4.779078 task=4.779078
-validation step  200: nll=1.955745 token_acc=0.5470
-validation step  220: nll=1.954824 token_acc=0.5470
-step  240/240: loss=2.130554 task=2.130554
-validation step  240: nll=1.953964 token_acc=0.5473
-2026-10-20 00:10:33,592 INFO Validation after fitting: nll 1.9540 (base 1.9682, delta -0.0142), token acc 0.5473
-2026-10-20 00:10:33,595 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage2_538m.pt (summary at /workspace/RiemannModel/ntk_controllers/stage2_538m.json)

training_logs/ntk_stage3_1.19b.log

@@ -1,34 +0,0 @@
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-2026-10-20 01:10:54,411 INFO Loading stage3_1.19b from /workspace/RiemannModel/checkpoints/stage3_1.19b onto cuda:0
-
-2026-10-20 01:10:56,281 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
-Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-20 01:11:01,200 INFO Warm start from stage2_538m.pt: carried 4096 of 4096 gates, added 535 for grown dims, 4631 total (24 layers, hidden 1280 -> 36 layers, hidden 1600)
-2026-10-20 01:11:17,200 INFO Validation before fitting: nll 1.6293, token acc 0.6245
-step    1/240: loss=1.200066 task=1.200066
-validation step   20: nll=1.628074 token_acc=0.6247
-validation step   40: nll=1.626826 token_acc=0.6250
-step   48/240: loss=0.990680 task=0.990680
-validation step   60: nll=1.625639 token_acc=0.6250
-validation step   80: nll=1.624534 token_acc=0.6250
-step   96/240: loss=1.420327 task=1.420327
-validation step  100: nll=1.623530 token_acc=0.6249
-validation step  120: nll=1.622574 token_acc=0.6250
-validation step  140: nll=1.621629 token_acc=0.6252
-step  144/240: loss=1.298729 task=1.298729
-validation step  160: nll=1.620797 token_acc=0.6252
-validation step  180: nll=1.620012 token_acc=0.6254
-step  192/240: loss=4.044214 task=4.044214
-validation step  200: nll=1.619268 token_acc=0.6256
-validation step  220: nll=1.618548 token_acc=0.6257
-step  240/240: loss=1.844396 task=1.844396
-validation step  240: nll=1.617883 token_acc=0.6259
-2026-10-20 01:15:47,961 INFO Validation after fitting: nll 1.6179 (base 1.6293, delta -0.0114), token acc 0.6259
-2026-10-20 01:15:47,963 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage3_1.19b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage3_1.19b.json)

training_logs/ntk_stage4_2.65b.log

@@ -1,34 +0,0 @@
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-2026-10-20 02:57:56,796 INFO Loading stage4_2.65b from /workspace/RiemannModel/checkpoints/stage4_2.65b onto cuda:0
-
-2026-10-20 02:57:59,845 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
-Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-20 02:58:08,439 INFO Warm start from stage3_1.19b.pt: carried 4102 of 4631 gates, added 0 for grown dims, 4102 total (36 layers, hidden 1600 -> 32 layers, hidden 2560)
-2026-10-20 02:58:37,177 INFO Validation before fitting: nll 1.5118, token acc 0.6508
-step    1/240: loss=1.074017 task=1.074017
-validation step   20: nll=1.511132 token_acc=0.6509
-validation step   40: nll=1.510413 token_acc=0.6512
-step   48/240: loss=0.905892 task=0.905892
-validation step   60: nll=1.509766 token_acc=0.6511
-validation step   80: nll=1.509152 token_acc=0.6512
-step   96/240: loss=1.314439 task=1.314439
-validation step  100: nll=1.508580 token_acc=0.6513
-validation step  120: nll=1.508032 token_acc=0.6513
-validation step  140: nll=1.507495 token_acc=0.6514
-step  144/240: loss=1.224094 task=1.224094
-validation step  160: nll=1.506989 token_acc=0.6515
-validation step  180: nll=1.506523 token_acc=0.6516
-step  192/240: loss=3.798501 task=3.798501
-validation step  200: nll=1.506091 token_acc=0.6515
-validation step  220: nll=1.505665 token_acc=0.6516
-step  240/240: loss=1.704812 task=1.704812
-validation step  240: nll=1.505260 token_acc=0.6517
-2026-10-20 03:06:45,030 INFO Validation after fitting: nll 1.5053 (base 1.5118, delta -0.0065), token acc 0.6517
-2026-10-20 03:06:45,032 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage4_2.65b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage4_2.65b.json)

training_logs/ntk_stage5_6.66b.log

@@ -1,34 +0,0 @@
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-2026-10-20 06:25:03,172 INFO Loading stage5_6.66b from /workspace/RiemannModel/checkpoints/stage5_6.66b onto cuda:0
-
-2026-10-20 06:25:09,498 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
-Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-20 06:25:26,489 INFO Warm start from stage4_2.65b.pt: carried 4102 of 4102 gates, added 0 for grown dims, 4102 total (32 layers, hidden 2560 -> 32 layers, hidden 4096)
-2026-10-20 06:26:28,141 INFO Validation before fitting: nll 1.4514, token acc 0.6641
-step    1/240: loss=1.024633 task=1.024633
-validation step   20: nll=1.450900 token_acc=0.6641
-validation step   40: nll=1.450434 token_acc=0.6642
-step   48/240: loss=0.860201 task=0.860201
-validation step   60: nll=1.450008 token_acc=0.6643
-validation step   80: nll=1.449611 token_acc=0.6643
-step   96/240: loss=1.264806 task=1.264806
-validation step  100: nll=1.449235 token_acc=0.6643
-validation step  120: nll=1.448885 token_acc=0.6644
-validation step  140: nll=1.448534 token_acc=0.6645
-step  144/240: loss=1.174451 task=1.174451
-validation step  160: nll=1.448192 token_acc=0.6645
-validation step  180: nll=1.447886 token_acc=0.6645
-step  192/240: loss=3.678861 task=3.678861
-validation step  200: nll=1.447578 token_acc=0.6646
-validation step  220: nll=1.447284 token_acc=0.6646
-step  240/240: loss=1.606066 task=1.606066
-validation step  240: nll=1.447003 token_acc=0.6647
-2026-10-20 06:43:42,579 INFO Validation after fitting: nll 1.4470 (base 1.4514, delta -0.0044), token acc 0.6647
-2026-10-20 06:43:42,581 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage5_6.66b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage5_6.66b.json)

training_logs/ntk_stage6_12.85b.log

@@ -1,34 +0,0 @@
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-2026-10-20 19:11:49,338 INFO Loading stage6_12.85b from /workspace/RiemannModel/checkpoints/stage6_12.85b onto cuda:0
-
-2026-10-20 19:12:02,159 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
-Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-20 19:12:36,467 INFO Warm start from stage5_6.66b.pt: carried 4102 of 4102 gates, added 2630 for grown dims, 6732 total (32 layers, hidden 4096 -> 40 layers, hidden 5120)
-2026-10-20 19:14:45,832 INFO Validation before fitting: nll 1.3228, token acc 0.6773
-step    1/240: loss=0.967124 task=0.967124
-validation step   20: nll=1.321372 token_acc=0.6774
-validation step   40: nll=1.320021 token_acc=0.6775
-step   48/240: loss=0.830313 task=0.830313
-validation step   60: nll=1.318690 token_acc=0.6776
-validation step   80: nll=1.317392 token_acc=0.6777
-step   96/240: loss=1.159311 task=1.159311
-validation step  100: nll=1.316291 token_acc=0.6777
-validation step  120: nll=1.315336 token_acc=0.6779
-validation step  140: nll=1.314391 token_acc=0.6780
-step  144/240: loss=1.085732 task=1.085732
-validation step  160: nll=1.313549 token_acc=0.6781
-validation step  180: nll=1.312607 token_acc=0.6782
-step  192/240: loss=2.922054 task=2.922054
-validation step  200: nll=1.311809 token_acc=0.6783
-validation step  220: nll=1.311043 token_acc=0.6784
-step  240/240: loss=1.457985 task=1.457985
-validation step  240: nll=1.310352 token_acc=0.6784
-2026-10-20 19:50:41,408 INFO Validation after fitting: nll 1.3104 (base 1.3228, delta -0.0124), token acc 0.6784
-2026-10-20 19:50:41,515 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage6_12.85b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage6_12.85b.json)

training_logs/pretrain_stage2_538m.log

@@ -1,232 +0,0 @@
-W1019 23:15:18.503000 12692 torch/distributed/run.py:803] 
-W1019 23:15:18.503000 12692 torch/distributed/run.py:803] *****************************************
-W1019 23:15:18.503000 12692 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
-W1019 23:15:18.503000 12692 torch/distributed/run.py:803] *****************************************
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-2026-10-19 23:15:32,353 INFO [rank 0] Loading stage2_538m from /workspace/RiemannModel/checkpoints/stage2_538m
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-riemann-climb:12761:12761 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:12761:12761 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-RCCL version : 2.22.3-HEAD:7d8d67c
-HIP version  : 6.4.43484-123eb5128
-ROCm version : 6.4.2.0-120-e7d83f5
-Hostname     : riemann-climb
-Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
-
-riemann-climb:12763:12763 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:12764:12764 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:12768:12768 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:12767:12767 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:12765:12765 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:12766:12766 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:12762:12762 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-2026-10-19 23:15:39,177 INFO [rank 0] Training stage2_538m: world=8 batch=16 accum=8 seq=1024 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.08B params
-Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-19 23:16:44,488 INFO [rank 0] step 20/1032 | loss 6.7805 | lr 3.00e-05 | 321301 tok/s | 0.0h elapsed | ~0.9h remaining
-2026-10-19 23:17:44,253 INFO [rank 0] step 40/1032 | loss 5.9708 | lr 6.00e-05 | 335454 tok/s | 0.0h elapsed | ~0.9h remaining
-2026-10-19 23:18:44,026 INFO [rank 0] step 60/1032 | loss 4.6836 | lr 9.00e-05 | 340434 tok/s | 0.1h elapsed | ~0.8h remaining
-2026-10-19 23:19:43,740 INFO [rank 0] step 80/1032 | loss 4.0124 | lr 1.20e-04 | 343061 tok/s | 0.1h elapsed | ~0.8h remaining
-2026-10-19 23:20:43,558 INFO [rank 0] step 100/1032 | loss 3.8263 | lr 1.50e-04 | 344540 tok/s | 0.1h elapsed | ~0.8h remaining
-2026-10-19 23:21:43,493 INFO [rank 0] step 120/1032 | loss 3.7315 | lr 1.80e-04 | 345423 tok/s | 0.1h elapsed | ~0.8h remaining
-2026-10-19 23:22:43,313 INFO [rank 0] step 140/1032 | loss 3.6715 | lr 2.10e-04 | 346152 tok/s | 0.1h elapsed | ~0.8h remaining
-2026-10-19 23:23:43,055 INFO [rank 0] step 160/1032 | loss 3.6275 | lr 2.40e-04 | 346754 tok/s | 0.1h elapsed | ~0.7h remaining
-2026-10-19 23:24:43,248 INFO [rank 0] step 180/1032 | loss 3.5879 | lr 2.70e-04 | 346936 tok/s | 0.2h elapsed | ~0.7h remaining
-2026-10-19 23:25:42,998 INFO [rank 0] step 200/1032 | loss 3.5558 | lr 3.00e-04 | 347337 tok/s | 0.2h elapsed | ~0.7h remaining
-2026-10-19 23:26:42,700 INFO [rank 0] step 220/1032 | loss 3.5210 | lr 3.00e-04 | 347691 tok/s | 0.2h elapsed | ~0.7h remaining
-2026-10-19 23:27:42,492 INFO [rank 0] step 240/1032 | loss 3.4818 | lr 2.99e-04 | 347944 tok/s | 0.2h elapsed | ~0.7h remaining
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-2026-10-19 23:28:16,343 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage2_538m
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
-  warnings.warn(  # warn only once
-2026-10-19 23:28:46,363 INFO [rank 0] step 260/1032 | loss 3.4492 | lr 2.97e-04 | 346352 tok/s | 0.2h elapsed | ~0.6h remaining
-2026-10-19 23:29:46,357 INFO [rank 0] step 280/1032 | loss 3.3954 | lr 2.94e-04 | 346580 tok/s | 0.2h elapsed | ~0.6h remaining
-2026-10-19 23:30:46,389 INFO [rank 0] step 300/1032 | loss 3.3589 | lr 2.91e-04 | 346762 tok/s | 0.3h elapsed | ~0.6h remaining
-2026-10-19 23:31:46,682 INFO [rank 0] step 320/1032 | loss 3.2849 | lr 2.87e-04 | 346829 tok/s | 0.3h elapsed | ~0.6h remaining
-2026-10-19 23:32:47,127 INFO [rank 0] step 340/1032 | loss 3.2320 | lr 2.82e-04 | 346836 tok/s | 0.3h elapsed | ~0.6h remaining
-2026-10-19 23:33:47,192 INFO [rank 0] step 360/1032 | loss 3.1603 | lr 2.76e-04 | 346964 tok/s | 0.3h elapsed | ~0.6h remaining
-2026-10-19 23:34:47,659 INFO [rank 0] step 380/1032 | loss 3.0949 | lr 2.70e-04 | 346956 tok/s | 0.3h elapsed | ~0.5h remaining
-2026-10-19 23:35:48,297 INFO [rank 0] step 400/1032 | loss 3.0317 | lr 2.64e-04 | 346901 tok/s | 0.3h elapsed | ~0.5h remaining
-2026-10-19 23:36:48,671 INFO [rank 0] step 420/1032 | loss 2.9633 | lr 2.56e-04 | 346922 tok/s | 0.4h elapsed | ~0.5h remaining
-2026-10-19 23:37:49,237 INFO [rank 0] step 440/1032 | loss 2.8976 | lr 2.49e-04 | 346892 tok/s | 0.4h elapsed | ~0.5h remaining
-2026-10-19 23:38:50,024 INFO [rank 0] step 460/1032 | loss 2.8261 | lr 2.40e-04 | 346810 tok/s | 0.4h elapsed | ~0.5h remaining
-2026-10-19 23:39:50,600 INFO [rank 0] step 480/1032 | loss 2.7663 | lr 2.32e-04 | 346784 tok/s | 0.4h elapsed | ~0.5h remaining
-2026-10-19 23:40:51,673 INFO [rank 0] step 500/1032 | loss 2.7029 | lr 2.23e-04 | 346647 tok/s | 0.4h elapsed | ~0.4h remaining
-2026-10-19 23:40:55,314 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage2_538m
-2026-10-19 23:41:55,852 INFO [rank 0] step 520/1032 | loss 2.6581 | lr 2.13e-04 | 345838 tok/s | 0.4h elapsed | ~0.4h remaining
-2026-10-19 23:42:56,449 INFO [rank 0] step 540/1032 | loss 2.6058 | lr 2.04e-04 | 345847 tok/s | 0.5h elapsed | ~0.4h remaining
-2026-10-19 23:43:57,212 INFO [rank 0] step 560/1032 | loss 2.5614 | lr 1.94e-04 | 345821 tok/s | 0.5h elapsed | ~0.4h remaining
-2026-10-19 23:44:57,948 INFO [rank 0] step 580/1032 | loss 2.5165 | lr 1.84e-04 | 345803 tok/s | 0.5h elapsed | ~0.4h remaining
-2026-10-19 23:45:58,530 INFO [rank 0] step 600/1032 | loss 2.4934 | lr 1.74e-04 | 345815 tok/s | 0.5h elapsed | ~0.4h remaining
-2026-10-19 23:46:59,125 INFO [rank 0] step 620/1032 | loss 2.4446 | lr 1.63e-04 | 345824 tok/s | 0.5h elapsed | ~0.3h remaining
-2026-10-19 23:47:59,698 INFO [rank 0] step 640/1032 | loss 2.4215 | lr 1.53e-04 | 345836 tok/s | 0.5h elapsed | ~0.3h remaining
-2026-10-19 23:49:00,149 INFO [rank 0] step 660/1032 | loss 2.4078 | lr 1.43e-04 | 345869 tok/s | 0.6h elapsed | ~0.3h remaining
-2026-10-19 23:50:00,674 INFO [rank 0] step 680/1032 | loss 2.3760 | lr 1.33e-04 | 345887 tok/s | 0.6h elapsed | ~0.3h remaining
-2026-10-19 23:51:00,926 INFO [rank 0] step 700/1032 | loss 2.3554 | lr 1.23e-04 | 345949 tok/s | 0.6h elapsed | ~0.3h remaining
-2026-10-19 23:52:01,371 INFO [rank 0] step 720/1032 | loss 2.3478 | lr 1.14e-04 | 345977 tok/s | 0.6h elapsed | ~0.3h remaining
-2026-10-19 23:53:01,732 INFO [rank 0] step 740/1032 | loss 2.3374 | lr 1.05e-04 | 346016 tok/s | 0.6h elapsed | ~0.2h remaining
-2026-10-19 23:53:35,348 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage2_538m
-2026-10-19 23:54:05,438 INFO [rank 0] step 760/1032 | loss 2.3081 | lr 9.56e-05 | 345552 tok/s | 0.6h elapsed | ~0.2h remaining
-2026-10-19 23:55:05,741 INFO [rank 0] step 780/1032 | loss 2.3049 | lr 8.71e-05 | 345608 tok/s | 0.7h elapsed | ~0.2h remaining
-2026-10-19 23:56:06,151 INFO [rank 0] step 800/1032 | loss 2.2835 | lr 7.90e-05 | 345647 tok/s | 0.7h elapsed | ~0.2h remaining
-2026-10-19 23:57:06,671 INFO [rank 0] step 820/1032 | loss 2.2798 | lr 7.14e-05 | 345668 tok/s | 0.7h elapsed | ~0.2h remaining
-2026-10-19 23:58:07,142 INFO [rank 0] step 840/1032 | loss 2.2702 | lr 6.43e-05 | 345695 tok/s | 0.7h elapsed | ~0.2h remaining
-2026-10-19 23:59:07,781 INFO [rank 0] step 860/1032 | loss 2.2573 | lr 5.78e-05 | 345698 tok/s | 0.7h elapsed | ~0.1h remaining
-2026-10-20 00:00:08,262 INFO [rank 0] step 880/1032 | loss 2.2535 | lr 5.19e-05 | 345722 tok/s | 0.7h elapsed | ~0.1h remaining
-2026-10-20 00:01:08,844 INFO [rank 0] step 900/1032 | loss 2.2496 | lr 4.67e-05 | 345732 tok/s | 0.8h elapsed | ~0.1h remaining
-2026-10-20 00:02:09,402 INFO [rank 0] step 920/1032 | loss 2.2503 | lr 4.21e-05 | 345744 tok/s | 0.8h elapsed | ~0.1h remaining
-2026-10-20 00:03:09,745 INFO [rank 0] step 940/1032 | loss 2.2272 | lr 3.82e-05 | 345782 tok/s | 0.8h elapsed | ~0.1h remaining
-2026-10-20 00:04:09,963 INFO [rank 0] step 960/1032 | loss 2.2299 | lr 3.51e-05 | 345833 tok/s | 0.8h elapsed | ~0.1h remaining
-2026-10-20 00:05:10,276 INFO [rank 0] step 980/1032 | loss 2.2276 | lr 3.27e-05 | 345872 tok/s | 0.8h elapsed | ~0.0h remaining
-2026-10-20 00:06:10,633 INFO [rank 0] step 1000/1032 | loss 2.2226 | lr 3.10e-05 | 345903 tok/s | 0.8h elapsed | ~0.0h remaining
-2026-10-20 00:06:14,141 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage2_538m
-2026-10-20 00:07:14,407 INFO [rank 0] step 1020/1032 | loss 2.2236 | lr 3.02e-05 | 345552 tok/s | 0.9h elapsed | ~0.0h remaining
-2026-10-20 00:07:54,004 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage2_538m
-2026-10-20 00:07:54,061 INFO [rank 0] Done. stage2_538m at step 1032, 1.08B tokens seen, 0.87h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage2_538m

training_logs/pretrain_stage3_1.19b.log

@@ -1,240 +0,0 @@
-W1020 00:21:31.953000 19307 torch/distributed/run.py:803] 
-W1020 00:21:31.953000 19307 torch/distributed/run.py:803] *****************************************
-W1020 00:21:31.953000 19307 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
-W1020 00:21:31.953000 19307 torch/distributed/run.py:803] *****************************************
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-
-
-
-2026-10-20 00:21:46,052 INFO [rank 0] Loading stage3_1.19b from /workspace/RiemannModel/checkpoints/stage3_1.19b
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-
-
-
-
-riemann-climb:19376:19376 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:19376:19376 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-RCCL version : 2.22.3-HEAD:7d8d67c
-HIP version  : 6.4.43484-123eb5128
-ROCm version : 6.4.2.0-120-e7d83f5
-Hostname     : riemann-climb
-Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
-
-riemann-climb:19377:19377 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:19381:19381 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:19380:19380 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:19378:19378 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:19379:19379 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:19383:19383 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:19382:19382 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-2026-10-20 00:21:53,536 INFO [rank 0] Training stage3_1.19b: world=8 batch=32 accum=4 seq=1024 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.16B params
-Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-20 00:22:54,219 INFO [rank 0] step 20/1032 | loss 2.1712 | lr 3.00e-05 | 345896 tok/s | 0.0h elapsed | ~0.9h remaining
-2026-10-20 00:23:50,164 INFO [rank 0] step 40/1032 | loss 2.2865 | lr 6.00e-05 | 359789 tok/s | 0.0h elapsed | ~0.8h remaining
-2026-10-20 00:24:46,046 INFO [rank 0] step 60/1032 | loss 2.2602 | lr 9.00e-05 | 364815 tok/s | 0.0h elapsed | ~0.8h remaining
-2026-10-20 00:25:42,050 INFO [rank 0] step 80/1032 | loss 2.2436 | lr 1.20e-04 | 367182 tok/s | 0.1h elapsed | ~0.8h remaining
-2026-10-20 00:26:37,954 INFO [rank 0] step 100/1032 | loss 2.2440 | lr 1.50e-04 | 368743 tok/s | 0.1h elapsed | ~0.7h remaining
-2026-10-20 00:27:33,872 INFO [rank 0] step 120/1032 | loss 2.2451 | lr 1.80e-04 | 369776 tok/s | 0.1h elapsed | ~0.7h remaining
-2026-10-20 00:28:29,821 INFO [rank 0] step 140/1032 | loss 2.2413 | lr 2.10e-04 | 370494 tok/s | 0.1h elapsed | ~0.7h remaining
-2026-10-20 00:29:25,780 INFO [rank 0] step 160/1032 | loss 2.2577 | lr 2.40e-04 | 371021 tok/s | 0.1h elapsed | ~0.7h remaining
-2026-10-20 00:30:21,737 INFO [rank 0] step 180/1032 | loss 2.2464 | lr 2.70e-04 | 371434 tok/s | 0.1h elapsed | ~0.7h remaining
-2026-10-20 00:31:17,702 INFO [rank 0] step 200/1032 | loss 2.2439 | lr 3.00e-04 | 371761 tok/s | 0.2h elapsed | ~0.7h remaining
-2026-10-20 00:32:13,687 INFO [rank 0] step 220/1032 | loss 2.2371 | lr 3.00e-04 | 372017 tok/s | 0.2h elapsed | ~0.6h remaining
-2026-10-20 00:33:09,685 INFO [rank 0] step 240/1032 | loss 2.1983 | lr 2.99e-04 | 372224 tok/s | 0.2h elapsed | ~0.6h remaining
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-2026-10-20 00:33:44,804 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
-  warnings.warn(  # warn only once
-2026-10-20 00:34:12,719 INFO [rank 0] step 260/1032 | loss 2.2012 | lr 2.97e-04 | 368851 tok/s | 0.2h elapsed | ~0.6h remaining
-2026-10-20 00:35:08,605 INFO [rank 0] step 280/1032 | loss 2.1747 | lr 2.94e-04 | 369302 tok/s | 0.2h elapsed | ~0.6h remaining
-2026-10-20 00:36:04,551 INFO [rank 0] step 300/1032 | loss 2.1743 | lr 2.91e-04 | 369667 tok/s | 0.2h elapsed | ~0.6h remaining
-2026-10-20 00:37:00,544 INFO [rank 0] step 320/1032 | loss 2.1484 | lr 2.87e-04 | 369968 tok/s | 0.3h elapsed | ~0.6h remaining
-2026-10-20 00:37:56,514 INFO [rank 0] step 340/1032 | loss 2.1395 | lr 2.82e-04 | 370243 tok/s | 0.3h elapsed | ~0.5h remaining
-2026-10-20 00:38:52,517 INFO [rank 0] step 360/1032 | loss 2.1176 | lr 2.76e-04 | 370475 tok/s | 0.3h elapsed | ~0.5h remaining
-2026-10-20 00:39:48,506 INFO [rank 0] step 380/1032 | loss 2.1059 | lr 2.70e-04 | 370688 tok/s | 0.3h elapsed | ~0.5h remaining
-2026-10-20 00:40:44,552 INFO [rank 0] step 400/1032 | loss 2.0935 | lr 2.64e-04 | 370860 tok/s | 0.3h elapsed | ~0.5h remaining
-2026-10-20 00:41:40,561 INFO [rank 0] step 420/1032 | loss 2.0781 | lr 2.56e-04 | 371030 tok/s | 0.3h elapsed | ~0.5h remaining
-2026-10-20 00:42:36,557 INFO [rank 0] step 440/1032 | loss 2.0756 | lr 2.49e-04 | 371187 tok/s | 0.3h elapsed | ~0.5h remaining
-2026-10-20 00:43:32,537 INFO [rank 0] step 460/1032 | loss 2.0530 | lr 2.40e-04 | 371335 tok/s | 0.4h elapsed | ~0.4h remaining
-2026-10-20 00:44:28,613 INFO [rank 0] step 480/1032 | loss 2.0511 | lr 2.32e-04 | 371445 tok/s | 0.4h elapsed | ~0.4h remaining
-2026-10-20 00:45:24,602 INFO [rank 0] step 500/1032 | loss 2.0276 | lr 2.23e-04 | 371568 tok/s | 0.4h elapsed | ~0.4h remaining
-2026-10-20 00:45:31,669 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
-2026-10-20 00:46:27,580 INFO [rank 0] step 520/1032 | loss 2.0268 | lr 2.13e-04 | 369921 tok/s | 0.4h elapsed | ~0.4h remaining
-2026-10-20 00:47:23,533 INFO [rank 0] step 540/1032 | loss 2.0125 | lr 2.04e-04 | 370099 tok/s | 0.4h elapsed | ~0.4h remaining
-2026-10-20 00:48:19,559 INFO [rank 0] step 560/1032 | loss 1.9975 | lr 1.94e-04 | 370248 tok/s | 0.4h elapsed | ~0.4h remaining
-2026-10-20 00:49:15,626 INFO [rank 0] step 580/1032 | loss 1.9867 | lr 1.84e-04 | 370378 tok/s | 0.5h elapsed | ~0.4h remaining
-2026-10-20 00:50:11,620 INFO [rank 0] step 600/1032 | loss 1.9871 | lr 1.74e-04 | 370515 tok/s | 0.5h elapsed | ~0.3h remaining
-2026-10-20 00:51:07,629 INFO [rank 0] step 620/1032 | loss 1.9603 | lr 1.63e-04 | 370641 tok/s | 0.5h elapsed | ~0.3h remaining
-2026-10-20 00:52:03,641 INFO [rank 0] step 640/1032 | loss 1.9573 | lr 1.53e-04 | 370756 tok/s | 0.5h elapsed | ~0.3h remaining
-2026-10-20 00:52:59,636 INFO [rank 0] step 660/1032 | loss 1.9553 | lr 1.43e-04 | 370870 tok/s | 0.5h elapsed | ~0.3h remaining
-2026-10-20 00:53:55,627 INFO [rank 0] step 680/1032 | loss 1.9386 | lr 1.33e-04 | 370977 tok/s | 0.5h elapsed | ~0.3h remaining
-2026-10-20 00:54:51,698 INFO [rank 0] step 700/1032 | loss 1.9342 | lr 1.23e-04 | 371063 tok/s | 0.5h elapsed | ~0.3h remaining
-2026-10-20 00:55:47,709 INFO [rank 0] step 720/1032 | loss 1.9314 | lr 1.14e-04 | 371155 tok/s | 0.6h elapsed | ~0.2h remaining
-2026-10-20 00:56:43,749 INFO [rank 0] step 740/1032 | loss 1.9295 | lr 1.05e-04 | 371238 tok/s | 0.6h elapsed | ~0.2h remaining
-2026-10-20 00:57:18,729 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
-2026-10-20 00:57:46,686 INFO [rank 0] step 760/1032 | loss 1.9113 | lr 9.56e-05 | 370127 tok/s | 0.6h elapsed | ~0.2h remaining
-2026-10-20 00:58:42,610 INFO [rank 0] step 780/1032 | loss 1.9142 | lr 8.71e-05 | 370249 tok/s | 0.6h elapsed | ~0.2h remaining
-2026-10-20 00:59:38,592 INFO [rank 0] step 800/1032 | loss 1.9000 | lr 7.90e-05 | 370358 tok/s | 0.6h elapsed | ~0.2h remaining
-2026-10-20 01:00:34,585 INFO [rank 0] step 820/1032 | loss 1.8998 | lr 7.14e-05 | 370459 tok/s | 0.6h elapsed | ~0.2h remaining
-2026-10-20 01:01:30,655 INFO [rank 0] step 840/1032 | loss 1.8986 | lr 6.43e-05 | 370543 tok/s | 0.7h elapsed | ~0.2h remaining
-2026-10-20 01:02:26,668 INFO [rank 0] step 860/1032 | loss 1.8892 | lr 5.78e-05 | 370631 tok/s | 0.7h elapsed | ~0.1h remaining
-2026-10-20 01:03:22,690 INFO [rank 0] step 880/1032 | loss 1.8870 | lr 5.19e-05 | 370715 tok/s | 0.7h elapsed | ~0.1h remaining
-2026-10-20 01:04:18,770 INFO [rank 0] step 900/1032 | loss 1.8881 | lr 4.67e-05 | 370786 tok/s | 0.7h elapsed | ~0.1h remaining
-2026-10-20 01:05:14,920 INFO [rank 0] step 920/1032 | loss 1.8887 | lr 4.21e-05 | 370845 tok/s | 0.7h elapsed | ~0.1h remaining
-2026-10-20 01:06:10,941 INFO [rank 0] step 940/1032 | loss 1.8743 | lr 3.82e-05 | 370919 tok/s | 0.7h elapsed | ~0.1h remaining
-2026-10-20 01:07:06,967 INFO [rank 0] step 960/1032 | loss 1.8769 | lr 3.51e-05 | 370989 tok/s | 0.8h elapsed | ~0.1h remaining
-2026-10-20 01:08:03,065 INFO [rank 0] step 980/1032 | loss 1.8764 | lr 3.27e-05 | 371047 tok/s | 0.8h elapsed | ~0.0h remaining
-2026-10-20 01:08:59,090 INFO [rank 0] step 1000/1032 | loss 1.8733 | lr 3.10e-05 | 371112 tok/s | 0.8h elapsed | ~0.0h remaining
-2026-10-20 01:09:05,916 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
-2026-10-20 01:10:01,843 INFO [rank 0] step 1020/1032 | loss 1.8773 | lr 3.02e-05 | 370310 tok/s | 0.8h elapsed | ~0.0h remaining
-2026-10-20 01:10:42,451 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
-2026-10-20 01:10:42,530 INFO [rank 0] Done. stage3_1.19b at step 1032, 1.08B tokens seen, 0.81h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage3_1.19b

training_logs/pretrain_stage4_2.65b.log

@@ -1,242 +0,0 @@
-W1020 01:16:10.814000 22649 torch/distributed/run.py:803] 
-W1020 01:16:10.814000 22649 torch/distributed/run.py:803] *****************************************
-W1020 01:16:10.814000 22649 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
-W1020 01:16:10.814000 22649 torch/distributed/run.py:803] *****************************************
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-2026-10-20 01:16:24,511 INFO [rank 0] Loading stage4_2.65b from /workspace/RiemannModel/checkpoints/stage4_2.65b
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-
-
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-
-riemann-climb:22718:22718 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:22718:22718 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-RCCL version : 2.22.3-HEAD:7d8d67c
-HIP version  : 6.4.43484-123eb5128
-ROCm version : 6.4.2.0-120-e7d83f5
-Hostname     : riemann-climb
-Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
-
-riemann-climb:22724:22724 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:22723:22723 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:22720:22720 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:22725:22725 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:22721:22721 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:22722:22722 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:22719:22719 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-2026-10-20 01:16:33,422 INFO [rank 0] Training stage4_2.65b: world=8 batch=16 accum=4 seq=2048 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.35B params
-Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-20 01:18:33,880 INFO [rank 0] step 20/1032 | loss 2.8309 | lr 3.00e-05 | 174380 tok/s | 0.0h elapsed | ~1.7h remaining
-2026-10-20 01:20:29,958 INFO [rank 0] step 40/1032 | loss 2.3154 | lr 6.00e-05 | 177489 tok/s | 0.1h elapsed | ~1.6h remaining
-2026-10-20 01:22:26,317 INFO [rank 0] step 60/1032 | loss 1.9829 | lr 9.00e-05 | 178398 tok/s | 0.1h elapsed | ~1.6h remaining
-2026-10-20 01:24:22,837 INFO [rank 0] step 80/1032 | loss 1.9206 | lr 1.20e-04 | 178791 tok/s | 0.1h elapsed | ~1.6h remaining
-2026-10-20 01:26:19,383 INFO [rank 0] step 100/1032 | loss 1.9109 | lr 1.50e-04 | 179019 tok/s | 0.2h elapsed | ~1.5h remaining
-2026-10-20 01:28:15,822 INFO [rank 0] step 120/1032 | loss 1.9083 | lr 1.80e-04 | 179200 tok/s | 0.2h elapsed | ~1.5h remaining
-2026-10-20 01:30:12,298 INFO [rank 0] step 140/1032 | loss 1.9123 | lr 2.10e-04 | 179320 tok/s | 0.2h elapsed | ~1.4h remaining
-2026-10-20 01:32:08,787 INFO [rank 0] step 160/1032 | loss 1.9199 | lr 2.40e-04 | 179409 tok/s | 0.3h elapsed | ~1.4h remaining
-2026-10-20 01:34:05,389 INFO [rank 0] step 180/1032 | loss 1.9169 | lr 2.70e-04 | 179459 tok/s | 0.3h elapsed | ~1.4h remaining
-2026-10-20 01:36:01,918 INFO [rank 0] step 200/1032 | loss 1.9244 | lr 3.00e-04 | 179509 tok/s | 0.3h elapsed | ~1.4h remaining
-2026-10-20 01:37:58,436 INFO [rank 0] step 220/1032 | loss 1.9171 | lr 3.00e-04 | 179552 tok/s | 0.4h elapsed | ~1.3h remaining
-2026-10-20 01:39:54,962 INFO [rank 0] step 240/1032 | loss 1.8900 | lr 2.99e-04 | 179588 tok/s | 0.4h elapsed | ~1.3h remaining
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-2026-10-20 01:41:03,878 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
-  warnings.warn(  # warn only once
-2026-10-20 01:42:02,031 INFO [rank 0] step 260/1032 | loss 1.8978 | lr 2.97e-04 | 178377 tok/s | 0.4h elapsed | ~1.3h remaining
-2026-10-20 01:43:58,446 INFO [rank 0] step 280/1032 | loss 1.8798 | lr 2.94e-04 | 178503 tok/s | 0.5h elapsed | ~1.2h remaining
-2026-10-20 01:45:54,978 INFO [rank 0] step 300/1032 | loss 1.8846 | lr 2.91e-04 | 178600 tok/s | 0.5h elapsed | ~1.2h remaining
-2026-10-20 01:47:51,545 INFO [rank 0] step 320/1032 | loss 1.8651 | lr 2.87e-04 | 178681 tok/s | 0.5h elapsed | ~1.2h remaining
-2026-10-20 01:49:48,143 INFO [rank 0] step 340/1032 | loss 1.8600 | lr 2.82e-04 | 178750 tok/s | 0.6h elapsed | ~1.1h remaining
-2026-10-20 01:51:44,699 INFO [rank 0] step 360/1032 | loss 1.8486 | lr 2.76e-04 | 178815 tok/s | 0.6h elapsed | ~1.1h remaining
-2026-10-20 01:53:41,243 INFO [rank 0] step 380/1032 | loss 1.8418 | lr 2.70e-04 | 178874 tok/s | 0.6h elapsed | ~1.1h remaining
-2026-10-20 01:55:37,805 INFO [rank 0] step 400/1032 | loss 1.8348 | lr 2.64e-04 | 178926 tok/s | 0.7h elapsed | ~1.0h remaining
-2026-10-20 01:57:34,332 INFO [rank 0] step 420/1032 | loss 1.8271 | lr 2.56e-04 | 178975 tok/s | 0.7h elapsed | ~1.0h remaining
-2026-10-20 01:59:30,887 INFO [rank 0] step 440/1032 | loss 1.8252 | lr 2.49e-04 | 179019 tok/s | 0.7h elapsed | ~1.0h remaining
-2026-10-20 02:01:27,461 INFO [rank 0] step 460/1032 | loss 1.8150 | lr 2.40e-04 | 179057 tok/s | 0.7h elapsed | ~0.9h remaining
-2026-10-20 02:03:24,053 INFO [rank 0] step 480/1032 | loss 1.8129 | lr 2.32e-04 | 179090 tok/s | 0.8h elapsed | ~0.9h remaining
-2026-10-20 02:05:20,620 INFO [rank 0] step 500/1032 | loss 1.7988 | lr 2.23e-04 | 179123 tok/s | 0.8h elapsed | ~0.9h remaining
-2026-10-20 02:05:31,366 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
-2026-10-20 02:07:27,936 INFO [rank 0] step 520/1032 | loss 1.8017 | lr 2.13e-04 | 178523 tok/s | 0.8h elapsed | ~0.8h remaining
-2026-10-20 02:09:24,523 INFO [rank 0] step 540/1032 | loss 1.7935 | lr 2.04e-04 | 178572 tok/s | 0.9h elapsed | ~0.8h remaining
-2026-10-20 02:11:21,131 INFO [rank 0] step 560/1032 | loss 1.7843 | lr 1.94e-04 | 178618 tok/s | 0.9h elapsed | ~0.8h remaining
-2026-10-20 02:13:17,699 INFO [rank 0] step 580/1032 | loss 1.7782 | lr 1.84e-04 | 178662 tok/s | 0.9h elapsed | ~0.7h remaining
-2026-10-20 02:15:14,264 INFO [rank 0] step 600/1032 | loss 1.7819 | lr 1.74e-04 | 178703 tok/s | 1.0h elapsed | ~0.7h remaining
-2026-10-20 02:17:10,849 INFO [rank 0] step 620/1032 | loss 1.7621 | lr 1.63e-04 | 178741 tok/s | 1.0h elapsed | ~0.7h remaining
-2026-10-20 02:19:07,411 INFO [rank 0] step 640/1032 | loss 1.7619 | lr 1.53e-04 | 178778 tok/s | 1.0h elapsed | ~0.6h remaining
-2026-10-20 02:21:04,061 INFO [rank 0] step 660/1032 | loss 1.7629 | lr 1.43e-04 | 178808 tok/s | 1.1h elapsed | ~0.6h remaining
-2026-10-20 02:23:00,624 INFO [rank 0] step 680/1032 | loss 1.7502 | lr 1.33e-04 | 178840 tok/s | 1.1h elapsed | ~0.6h remaining
-2026-10-20 02:24:57,247 INFO [rank 0] step 700/1032 | loss 1.7490 | lr 1.23e-04 | 178868 tok/s | 1.1h elapsed | ~0.5h remaining
-2026-10-20 02:26:53,839 INFO [rank 0] step 720/1032 | loss 1.7482 | lr 1.14e-04 | 178896 tok/s | 1.2h elapsed | ~0.5h remaining
-2026-10-20 02:28:50,483 INFO [rank 0] step 740/1032 | loss 1.7484 | lr 1.05e-04 | 178920 tok/s | 1.2h elapsed | ~0.5h remaining
-2026-10-20 02:29:59,102 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
-2026-10-20 02:30:57,443 INFO [rank 0] step 760/1032 | loss 1.7339 | lr 9.56e-05 | 178529 tok/s | 1.2h elapsed | ~0.4h remaining
-2026-10-20 02:32:54,110 INFO [rank 0] step 780/1032 | loss 1.7377 | lr 8.71e-05 | 178560 tok/s | 1.3h elapsed | ~0.4h remaining
-2026-10-20 02:34:50,732 INFO [rank 0] step 800/1032 | loss 1.7268 | lr 7.90e-05 | 178592 tok/s | 1.3h elapsed | ~0.4h remaining
-2026-10-20 02:36:47,475 INFO [rank 0] step 820/1032 | loss 1.7281 | lr 7.14e-05 | 178619 tok/s | 1.3h elapsed | ~0.3h remaining
-2026-10-20 02:38:44,173 INFO [rank 0] step 840/1032 | loss 1.7287 | lr 6.43e-05 | 178643 tok/s | 1.4h elapsed | ~0.3h remaining
-
-riemann-climb:22723:22803 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/misc/socket.cc:438 NCCL WARN socketFinalizeAccept: wrong magic 1ee00010316 != b646e86fa12d961e
-2026-10-20 02:40:40,816 INFO [rank 0] step 860/1032 | loss 1.7210 | lr 5.78e-05 | 178670 tok/s | 1.4h elapsed | ~0.3h remaining
-2026-10-20 02:42:37,462 INFO [rank 0] step 880/1032 | loss 1.7200 | lr 5.19e-05 | 178695 tok/s | 1.4h elapsed | ~0.2h remaining
-2026-10-20 02:44:34,096 INFO [rank 0] step 900/1032 | loss 1.7222 | lr 4.67e-05 | 178720 tok/s | 1.5h elapsed | ~0.2h remaining
-2026-10-20 02:46:30,759 INFO [rank 0] step 920/1032 | loss 1.7234 | lr 4.21e-05 | 178742 tok/s | 1.5h elapsed | ~0.2h remaining
-2026-10-20 02:48:27,404 INFO [rank 0] step 940/1032 | loss 1.7106 | lr 3.82e-05 | 178764 tok/s | 1.5h elapsed | ~0.1h remaining
-2026-10-20 02:50:24,060 INFO [rank 0] step 960/1032 | loss 1.7138 | lr 3.51e-05 | 178785 tok/s | 1.6h elapsed | ~0.1h remaining
-2026-10-20 02:52:20,691 INFO [rank 0] step 980/1032 | loss 1.7143 | lr 3.27e-05 | 178806 tok/s | 1.6h elapsed | ~0.1h remaining
-2026-10-20 02:54:17,305 INFO [rank 0] step 1000/1032 | loss 1.7122 | lr 3.10e-05 | 178826 tok/s | 1.6h elapsed | ~0.1h remaining
-2026-10-20 02:54:27,615 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
-2026-10-20 02:56:24,341 INFO [rank 0] step 1020/1032 | loss 1.7164 | lr 3.02e-05 | 178535 tok/s | 1.7h elapsed | ~0.0h remaining
-2026-10-20 02:57:44,569 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
-2026-10-20 02:57:44,694 INFO [rank 0] Done. stage4_2.65b at step 1032, 1.08B tokens seen, 1.69h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage4_2.65b

training_logs/pretrain_stage5_6.66b.log

@@ -1,240 +0,0 @@
-W1020 03:07:33.887000 28035 torch/distributed/run.py:803] 
-W1020 03:07:33.887000 28035 torch/distributed/run.py:803] *****************************************
-W1020 03:07:33.887000 28035 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
-W1020 03:07:33.887000 28035 torch/distributed/run.py:803] *****************************************
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-2026-10-20 03:07:47,807 INFO [rank 0] Loading stage5_6.66b from /workspace/RiemannModel/checkpoints/stage5_6.66b
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-
-
-
-riemann-climb:28104:28104 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:28104:28104 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-RCCL version : 2.22.3-HEAD:7d8d67c
-HIP version  : 6.4.43484-123eb5128
-ROCm version : 6.4.2.0-120-e7d83f5
-Hostname     : riemann-climb
-Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
-
-riemann-climb:28111:28111 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:28110:28110 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:28107:28107 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:28108:28108 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:28105:28105 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:28106:28106 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:28109:28109 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-2026-10-20 03:08:00,347 INFO [rank 0] Training stage5_6.66b: world=8 batch=8 accum=8 seq=2048 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.86B params
-Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-20 03:11:50,429 INFO [rank 0] step 20/1032 | loss 1.8763 | lr 3.00e-05 | 91263 tok/s | 0.1h elapsed | ~3.2h remaining
-2026-10-20 03:15:36,949 INFO [rank 0] step 40/1032 | loss 1.7875 | lr 6.00e-05 | 91918 tok/s | 0.1h elapsed | ~3.1h remaining
-2026-10-20 03:19:23,573 INFO [rank 0] step 60/1032 | loss 1.7555 | lr 9.00e-05 | 92124 tok/s | 0.2h elapsed | ~3.1h remaining
-2026-10-20 03:23:10,316 INFO [rank 0] step 80/1032 | loss 1.7371 | lr 1.20e-04 | 92214 tok/s | 0.3h elapsed | ~3.0h remaining
-2026-10-20 03:26:57,200 INFO [rank 0] step 100/1032 | loss 1.7389 | lr 1.50e-04 | 92259 tok/s | 0.3h elapsed | ~2.9h remaining
-2026-10-20 03:30:44,108 INFO [rank 0] step 120/1032 | loss 1.7405 | lr 1.80e-04 | 92285 tok/s | 0.4h elapsed | ~2.9h remaining
-2026-10-20 03:34:31,069 INFO [rank 0] step 140/1032 | loss 1.7441 | lr 2.10e-04 | 92302 tok/s | 0.4h elapsed | ~2.8h remaining
-2026-10-20 03:38:18,036 INFO [rank 0] step 160/1032 | loss 1.7498 | lr 2.40e-04 | 92313 tok/s | 0.5h elapsed | ~2.8h remaining
-2026-10-20 03:42:04,922 INFO [rank 0] step 180/1032 | loss 1.8745 | lr 2.70e-04 | 92327 tok/s | 0.6h elapsed | ~2.7h remaining
-2026-10-20 03:45:51,783 INFO [rank 0] step 200/1032 | loss 1.8192 | lr 3.00e-04 | 92338 tok/s | 0.6h elapsed | ~2.6h remaining
-2026-10-20 03:49:38,612 INFO [rank 0] step 220/1032 | loss 1.7556 | lr 3.00e-04 | 92349 tok/s | 0.7h elapsed | ~2.6h remaining
-2026-10-20 03:53:25,573 INFO [rank 0] step 240/1032 | loss 1.7410 | lr 2.99e-04 | 92354 tok/s | 0.8h elapsed | ~2.5h remaining
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-2026-10-20 03:55:38,918 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
-  warnings.warn(  # warn only once
-2026-10-20 03:57:32,525 INFO [rank 0] step 260/1032 | loss 1.7487 | lr 2.97e-04 | 91736 tok/s | 0.8h elapsed | ~2.5h remaining
-2026-10-20 04:01:19,418 INFO [rank 0] step 280/1032 | loss 1.7394 | lr 2.94e-04 | 91785 tok/s | 0.9h elapsed | ~2.4h remaining
-2026-10-20 04:05:06,298 INFO [rank 0] step 300/1032 | loss 1.7467 | lr 2.91e-04 | 91828 tok/s | 1.0h elapsed | ~2.3h remaining
-2026-10-20 04:08:53,206 INFO [rank 0] step 320/1032 | loss 1.7297 | lr 2.87e-04 | 91866 tok/s | 1.0h elapsed | ~2.3h remaining
-2026-10-20 04:12:40,233 INFO [rank 0] step 340/1032 | loss 1.7289 | lr 2.82e-04 | 91895 tok/s | 1.1h elapsed | ~2.2h remaining
-2026-10-20 04:16:27,133 INFO [rank 0] step 360/1032 | loss 1.7165 | lr 2.76e-04 | 91925 tok/s | 1.1h elapsed | ~2.1h remaining
-2026-10-20 04:20:14,126 INFO [rank 0] step 380/1032 | loss 1.7133 | lr 2.70e-04 | 91949 tok/s | 1.2h elapsed | ~2.1h remaining
-2026-10-20 04:24:01,114 INFO [rank 0] step 400/1032 | loss 1.7121 | lr 2.64e-04 | 91971 tok/s | 1.3h elapsed | ~2.0h remaining
-2026-10-20 04:27:48,401 INFO [rank 0] step 420/1032 | loss 1.7046 | lr 2.56e-04 | 91985 tok/s | 1.3h elapsed | ~1.9h remaining
-2026-10-20 04:31:35,479 INFO [rank 0] step 440/1032 | loss 1.7053 | lr 2.49e-04 | 92001 tok/s | 1.4h elapsed | ~1.9h remaining
-2026-10-20 04:35:22,576 INFO [rank 0] step 460/1032 | loss 1.6966 | lr 2.40e-04 | 92016 tok/s | 1.5h elapsed | ~1.8h remaining
-2026-10-20 04:39:09,469 INFO [rank 0] step 480/1032 | loss 1.6970 | lr 2.32e-04 | 92033 tok/s | 1.5h elapsed | ~1.7h remaining
-2026-10-20 04:42:56,451 INFO [rank 0] step 500/1032 | loss 1.6855 | lr 2.23e-04 | 92048 tok/s | 1.6h elapsed | ~1.7h remaining
-2026-10-20 04:43:16,690 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
-2026-10-20 04:47:03,662 INFO [rank 0] step 520/1032 | loss 1.6906 | lr 2.13e-04 | 91747 tok/s | 1.7h elapsed | ~1.6h remaining
-2026-10-20 04:50:50,338 INFO [rank 0] step 540/1032 | loss 1.6834 | lr 2.04e-04 | 91775 tok/s | 1.7h elapsed | ~1.6h remaining
-2026-10-20 04:54:36,948 INFO [rank 0] step 560/1032 | loss 1.6769 | lr 1.94e-04 | 91802 tok/s | 1.8h elapsed | ~1.5h remaining
-2026-10-20 04:58:23,530 INFO [rank 0] step 580/1032 | loss 1.6720 | lr 1.84e-04 | 91828 tok/s | 1.8h elapsed | ~1.4h remaining
-2026-10-20 05:02:10,121 INFO [rank 0] step 600/1032 | loss 1.6772 | lr 1.74e-04 | 91852 tok/s | 1.9h elapsed | ~1.4h remaining
-2026-10-20 05:05:56,716 INFO [rank 0] step 620/1032 | loss 1.6600 | lr 1.63e-04 | 91875 tok/s | 2.0h elapsed | ~1.3h remaining
-2026-10-20 05:09:43,279 INFO [rank 0] step 640/1032 | loss 1.6608 | lr 1.53e-04 | 91896 tok/s | 2.0h elapsed | ~1.2h remaining
-2026-10-20 05:13:29,845 INFO [rank 0] step 660/1032 | loss 1.6639 | lr 1.43e-04 | 91917 tok/s | 2.1h elapsed | ~1.2h remaining
-2026-10-20 05:17:16,420 INFO [rank 0] step 680/1032 | loss 1.6535 | lr 1.33e-04 | 91935 tok/s | 2.2h elapsed | ~1.1h remaining
-2026-10-20 05:21:02,987 INFO [rank 0] step 700/1032 | loss 1.6506 | lr 1.23e-04 | 91953 tok/s | 2.2h elapsed | ~1.1h remaining
-2026-10-20 05:24:49,497 INFO [rank 0] step 720/1032 | loss 1.6547 | lr 1.14e-04 | 91971 tok/s | 2.3h elapsed | ~1.0h remaining
-2026-10-20 05:28:36,096 INFO [rank 0] step 740/1032 | loss 1.6544 | lr 1.05e-04 | 91986 tok/s | 2.3h elapsed | ~0.9h remaining
-2026-10-20 05:30:49,206 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
-2026-10-20 05:32:42,907 INFO [rank 0] step 760/1032 | loss 1.6420 | lr 9.56e-05 | 91787 tok/s | 2.4h elapsed | ~0.9h remaining
-2026-10-20 05:36:29,941 INFO [rank 0] step 780/1032 | loss 1.6449 | lr 8.71e-05 | 91802 tok/s | 2.5h elapsed | ~0.8h remaining
-2026-10-20 05:40:16,948 INFO [rank 0] step 800/1032 | loss 1.6364 | lr 7.90e-05 | 91816 tok/s | 2.5h elapsed | ~0.7h remaining
-2026-10-20 05:44:04,017 INFO [rank 0] step 820/1032 | loss 1.6391 | lr 7.14e-05 | 91829 tok/s | 2.6h elapsed | ~0.7h remaining
-2026-10-20 05:47:51,033 INFO [rank 0] step 840/1032 | loss 1.6383 | lr 6.43e-05 | 91842 tok/s | 2.7h elapsed | ~0.6h remaining
-2026-10-20 05:51:38,074 INFO [rank 0] step 860/1032 | loss 1.6322 | lr 5.78e-05 | 91854 tok/s | 2.7h elapsed | ~0.5h remaining
-2026-10-20 05:55:25,642 INFO [rank 0] step 880/1032 | loss 1.6335 | lr 5.19e-05 | 91861 tok/s | 2.8h elapsed | ~0.5h remaining
-2026-10-20 05:59:12,629 INFO [rank 0] step 900/1032 | loss 1.6343 | lr 4.67e-05 | 91873 tok/s | 2.9h elapsed | ~0.4h remaining
-2026-10-20 06:02:59,587 INFO [rank 0] step 920/1032 | loss 1.6370 | lr 4.21e-05 | 91884 tok/s | 2.9h elapsed | ~0.4h remaining
-2026-10-20 06:06:46,622 INFO [rank 0] step 940/1032 | loss 1.6234 | lr 3.82e-05 | 91895 tok/s | 3.0h elapsed | ~0.3h remaining
-2026-10-20 06:10:33,534 INFO [rank 0] step 960/1032 | loss 1.6279 | lr 3.51e-05 | 91906 tok/s | 3.0h elapsed | ~0.2h remaining
-2026-10-20 06:14:20,507 INFO [rank 0] step 980/1032 | loss 1.6289 | lr 3.27e-05 | 91916 tok/s | 3.1h elapsed | ~0.2h remaining
-2026-10-20 06:18:07,471 INFO [rank 0] step 1000/1032 | loss 1.6276 | lr 3.10e-05 | 91925 tok/s | 3.2h elapsed | ~0.1h remaining
-2026-10-20 06:18:27,596 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
-2026-10-20 06:22:14,994 INFO [rank 0] step 1020/1032 | loss 1.6308 | lr 3.02e-05 | 91772 tok/s | 3.2h elapsed | ~0.0h remaining
-2026-10-20 06:24:50,821 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
-2026-10-20 06:24:51,260 INFO [rank 0] Done. stage5_6.66b at step 1032, 1.08B tokens seen, 3.28h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage5_6.66b

training_logs/pretrain_stage6_12.85b.log

@@ -1,296 +0,0 @@
-W1020 06:45:09.583000 484096 torch/distributed/run.py:803] 
-W1020 06:45:09.583000 484096 torch/distributed/run.py:803] *****************************************
-W1020 06:45:09.583000 484096 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
-W1020 06:45:09.583000 484096 torch/distributed/run.py:803] *****************************************
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-`torch_dtype` is deprecated! Use `dtype` instead!
-2026-10-20 06:45:23,811 INFO [rank 0] Loading stage6_12.85b from /workspace/RiemannModel/checkpoints/stage6_12.85b
-`torch_dtype` is deprecated! Use `dtype` instead!
-
-
-
-
-
-
-
-
-riemann-climb:484165:484165 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:484165:484165 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-RCCL version : 2.22.3-HEAD:7d8d67c
-HIP version  : 6.4.43484-123eb5128
-ROCm version : 6.4.2.0-120-e7d83f5
-Hostname     : riemann-climb
-Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
-
-riemann-climb:484170:484170 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:484172:484172 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:484166:484166 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:484167:484167 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:484168:484168 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:484169:484169 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-
-riemann-climb:484171:484171 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
-2026-10-20 06:45:42,993 INFO [rank 0] Training stage6_12.85b: world=8 batch=4 accum=16 seq=2048 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 2064 steps (2.00 epochs), 1.64B params
-Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
-Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
-2026-10-20 06:52:56,995 INFO [rank 0] step 20/2064 | loss 2.0337 | lr 3.00e-05 | 48399 tok/s | 0.1h elapsed | ~12.3h remaining
-2026-10-20 07:00:07,512 INFO [rank 0] step 40/2064 | loss 1.8370 | lr 6.00e-05 | 48556 tok/s | 0.2h elapsed | ~12.1h remaining
-2026-10-20 07:07:18,582 INFO [rank 0] step 60/2064 | loss 1.7434 | lr 9.00e-05 | 48588 tok/s | 0.4h elapsed | ~12.0h remaining
-2026-10-20 07:14:29,312 INFO [rank 0] step 80/2064 | loss 1.7276 | lr 1.20e-04 | 48613 tok/s | 0.5h elapsed | ~11.9h remaining
-2026-10-20 07:21:40,016 INFO [rank 0] step 100/2064 | loss 1.7116 | lr 1.50e-04 | 48629 tok/s | 0.6h elapsed | ~11.8h remaining
-2026-10-20 07:28:50,781 INFO [rank 0] step 120/2064 | loss 1.7295 | lr 1.80e-04 | 48638 tok/s | 0.7h elapsed | ~11.6h remaining
-2026-10-20 07:36:02,039 INFO [rank 0] step 140/2064 | loss 1.7355 | lr 2.10e-04 | 48636 tok/s | 0.8h elapsed | ~11.5h remaining
-2026-10-20 07:43:12,982 INFO [rank 0] step 160/2064 | loss 1.6994 | lr 2.40e-04 | 48640 tok/s | 1.0h elapsed | ~11.4h remaining
-2026-10-20 07:50:23,812 INFO [rank 0] step 180/2064 | loss 1.7228 | lr 2.70e-04 | 48644 tok/s | 1.1h elapsed | ~11.3h remaining
-2026-10-20 07:57:34,417 INFO [rank 0] step 200/2064 | loss 1.8852 | lr 3.00e-04 | 48650 tok/s | 1.2h elapsed | ~11.2h remaining
-2026-10-20 08:04:44,979 INFO [rank 0] step 220/2064 | loss 1.9192 | lr 3.00e-04 | 48655 tok/s | 1.3h elapsed | ~11.0h remaining
-2026-10-20 08:11:55,576 INFO [rank 0] step 240/2064 | loss 1.7140 | lr 3.00e-04 | 48659 tok/s | 1.4h elapsed | ~10.9h remaining
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
-  warnings.warn(
-2026-10-20 08:16:06,855 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
-/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
-  warnings.warn(  # warn only once
-2026-10-20 08:19:42,600 INFO [rank 0] step 260/2064 | loss 1.6971 | lr 2.99e-04 | 48348 tok/s | 1.6h elapsed | ~10.9h remaining
-2026-10-20 08:26:52,491 INFO [rank 0] step 280/2064 | loss 1.6805 | lr 2.99e-04 | 48379 tok/s | 1.7h elapsed | ~10.7h remaining
-2026-10-20 08:34:02,503 INFO [rank 0] step 300/2064 | loss 1.6987 | lr 2.98e-04 | 48405 tok/s | 1.8h elapsed | ~10.6h remaining
-2026-10-20 08:41:12,457 INFO [rank 0] step 320/2064 | loss 1.6803 | lr 2.97e-04 | 48428 tok/s | 1.9h elapsed | ~10.5h remaining
-2026-10-20 08:48:22,433 INFO [rank 0] step 340/2064 | loss 1.6833 | lr 2.96e-04 | 48448 tok/s | 2.0h elapsed | ~10.4h remaining
-2026-10-20 08:55:32,405 INFO [rank 0] step 360/2064 | loss 1.6622 | lr 2.95e-04 | 48466 tok/s | 2.2h elapsed | ~10.2h remaining
-2026-10-20 09:02:42,359 INFO [rank 0] step 380/2064 | loss 1.6607 | lr 2.94e-04 | 48482 tok/s | 2.3h elapsed | ~10.1h remaining
-2026-10-20 09:09:52,314 INFO [rank 0] step 400/2064 | loss 1.6562 | lr 2.92e-04 | 48497 tok/s | 2.4h elapsed | ~10.0h remaining
-2026-10-20 09:17:02,309 INFO [rank 0] step 420/2064 | loss 1.6503 | lr 2.91e-04 | 48510 tok/s | 2.5h elapsed | ~9.9h remaining
-2026-10-20 09:24:12,230 INFO [rank 0] step 440/2064 | loss 1.6571 | lr 2.89e-04 | 48522 tok/s | 2.6h elapsed | ~9.7h remaining
-2026-10-20 09:31:22,153 INFO [rank 0] step 460/2064 | loss 1.6430 | lr 2.87e-04 | 48533 tok/s | 2.8h elapsed | ~9.6h remaining
-2026-10-20 09:38:32,079 INFO [rank 0] step 480/2064 | loss 1.6455 | lr 2.85e-04 | 48543 tok/s | 2.9h elapsed | ~9.5h remaining
-2026-10-20 09:45:41,989 INFO [rank 0] step 500/2064 | loss 1.6392 | lr 2.83e-04 | 48553 tok/s | 3.0h elapsed | ~9.4h remaining
-2026-10-20 09:46:18,041 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
-2026-10-20 09:53:28,754 INFO [rank 0] step 520/2064 | loss 1.6442 | lr 2.81e-04 | 48403 tok/s | 3.1h elapsed | ~9.3h remaining
-2026-10-20 10:00:38,640 INFO [rank 0] step 540/2064 | loss 1.6359 | lr 2.79e-04 | 48417 tok/s | 3.2h elapsed | ~9.2h remaining
-2026-10-20 10:07:48,531 INFO [rank 0] step 560/2064 | loss 1.6306 | lr 2.76e-04 | 48430 tok/s | 3.4h elapsed | ~9.0h remaining
-2026-10-20 10:14:58,386 INFO [rank 0] step 580/2064 | loss 1.6298 | lr 2.73e-04 | 48442 tok/s | 3.5h elapsed | ~8.9h remaining
-2026-10-20 10:22:08,202 INFO [rank 0] step 600/2064 | loss 1.6348 | lr 2.71e-04 | 48454 tok/s | 3.6h elapsed | ~8.8h remaining
-2026-10-20 10:29:18,066 INFO [rank 0] step 620/2064 | loss 1.6209 | lr 2.68e-04 | 48464 tok/s | 3.7h elapsed | ~8.7h remaining
-2026-10-20 10:36:27,903 INFO [rank 0] step 640/2064 | loss 1.6229 | lr 2.65e-04 | 48474 tok/s | 3.8h elapsed | ~8.6h remaining
-2026-10-20 10:43:37,790 INFO [rank 0] step 660/2064 | loss 1.6260 | lr 2.62e-04 | 48484 tok/s | 4.0h elapsed | ~8.4h remaining
-2026-10-20 10:50:47,659 INFO [rank 0] step 680/2064 | loss 1.6188 | lr 2.58e-04 | 48493 tok/s | 4.1h elapsed | ~8.3h remaining
-2026-10-20 10:57:57,566 INFO [rank 0] step 700/2064 | loss 1.6183 | lr 2.55e-04 | 48501 tok/s | 4.2h elapsed | ~8.2h remaining
-2026-10-20 11:05:07,432 INFO [rank 0] step 720/2064 | loss 1.6237 | lr 2.52e-04 | 48509 tok/s | 4.3h elapsed | ~8.1h remaining
-2026-10-20 11:12:17,403 INFO [rank 0] step 740/2064 | loss 1.6255 | lr 2.48e-04 | 48516 tok/s | 4.4h elapsed | ~7.9h remaining
-2026-10-20 11:16:27,999 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
-2026-10-20 11:20:03,710 INFO [rank 0] step 760/2064 | loss 1.6134 | lr 2.44e-04 | 48415 tok/s | 4.6h elapsed | ~7.8h remaining
-2026-10-20 11:27:13,649 INFO [rank 0] step 780/2064 | loss 1.6184 | lr 2.41e-04 | 48425 tok/s | 4.7h elapsed | ~7.7h remaining
-2026-10-20 11:34:23,560 INFO [rank 0] step 800/2064 | loss 1.6114 | lr 2.37e-04 | 48433 tok/s | 4.8h elapsed | ~7.6h remaining
-2026-10-20 11:41:33,419 INFO [rank 0] step 820/2064 | loss 1.6148 | lr 2.33e-04 | 48442 tok/s | 4.9h elapsed | ~7.5h remaining
-2026-10-20 11:48:43,360 INFO [rank 0] step 840/2064 | loss 1.6149 | lr 2.29e-04 | 48450 tok/s | 5.0h elapsed | ~7.4h remaining
-2026-10-20 11:55:53,243 INFO [rank 0] step 860/2064 | loss 1.6090 | lr 2.25e-04 | 48458 tok/s | 5.2h elapsed | ~7.2h remaining
-2026-10-20 12:03:03,071 INFO [rank 0] step 880/2064 | loss 1.6103 | lr 2.21e-04 | 48465 tok/s | 5.3h elapsed | ~7.1h remaining
-2026-10-20 12:10:12,911 INFO [rank 0] step 900/2064 | loss 1.6113 | lr 2.17e-04 | 48472 tok/s | 5.4h elapsed | ~7.0h remaining
-2026-10-20 12:17:22,778 INFO [rank 0] step 920/2064 | loss 1.6132 | lr 2.12e-04 | 48479 tok/s | 5.5h elapsed | ~6.9h remaining
-2026-10-20 12:24:32,513 INFO [rank 0] step 940/2064 | loss 1.6001 | lr 2.08e-04 | 48486 tok/s | 5.6h elapsed | ~6.8h remaining
-2026-10-20 12:31:42,315 INFO [rank 0] step 960/2064 | loss 1.6044 | lr 2.04e-04 | 48492 tok/s | 5.8h elapsed | ~6.6h remaining
-2026-10-20 12:38:52,214 INFO [rank 0] step 980/2064 | loss 1.6050 | lr 1.99e-04 | 48498 tok/s | 5.9h elapsed | ~6.5h remaining
-2026-10-20 12:46:02,061 INFO [rank 0] step 1000/2064 | loss 1.6006 | lr 1.95e-04 | 48504 tok/s | 6.0h elapsed | ~6.4h remaining
-2026-10-20 12:46:39,533 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
-2026-10-20 12:53:50,088 INFO [rank 0] step 1020/2064 | loss 1.6042 | lr 1.91e-04 | 48426 tok/s | 6.1h elapsed | ~6.3h remaining
-2026-10-20 13:00:59,373 INFO [rank 0] step 1040/2064 | loss 2.9027 | lr 1.86e-04 | 48434 tok/s | 6.3h elapsed | ~6.2h remaining
-2026-10-20 13:08:09,075 INFO [rank 0] step 1060/2064 | loss 2.0565 | lr 1.82e-04 | 48441 tok/s | 6.4h elapsed | ~6.0h remaining
-2026-10-20 13:15:19,026 INFO [rank 0] step 1080/2064 | loss 1.7069 | lr 1.77e-04 | 48447 tok/s | 6.5h elapsed | ~5.9h remaining
-2026-10-20 13:22:28,935 INFO [rank 0] step 1100/2064 | loss 1.5320 | lr 1.73e-04 | 48453 tok/s | 6.6h elapsed | ~5.8h remaining
-2026-10-20 13:29:38,958 INFO [rank 0] step 1120/2064 | loss 1.6455 | lr 1.68e-04 | 48458 tok/s | 6.7h elapsed | ~5.7h remaining
-2026-10-20 13:36:48,911 INFO [rank 0] step 1140/2064 | loss 1.9496 | lr 1.63e-04 | 48464 tok/s | 6.9h elapsed | ~5.6h remaining
-2026-10-20 13:43:59,087 INFO [rank 0] step 1160/2064 | loss 1.7489 | lr 1.59e-04 | 48469 tok/s | 7.0h elapsed | ~5.4h remaining
-2026-10-20 13:51:09,376 INFO [rank 0] step 1180/2064 | loss 1.0277 | lr 1.54e-04 | 48474 tok/s | 7.1h elapsed | ~5.3h remaining
-2026-10-20 13:58:19,757 INFO [rank 0] step 1200/2064 | loss 0.5920 | lr 1.50e-04 | 48478 tok/s | 7.2h elapsed | ~5.2h remaining
-2026-10-20 14:05:30,104 INFO [rank 0] step 1220/2064 | loss 0.3200 | lr 1.45e-04 | 48482 tok/s | 7.3h elapsed | ~5.1h remaining
-2026-10-20 14:12:40,598 INFO [rank 0] step 1240/2064 | loss 0.1636 | lr 1.41e-04 | 48486 tok/s | 7.4h elapsed | ~5.0h remaining
-2026-10-20 14:16:51,776 INFO [rank 0] Saved checkpoint at step 1250 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
-2026-10-20 14:20:27,885 INFO [rank 0] step 1260/2064 | loss 0.1215 | lr 1.36e-04 | 48424 tok/s | 7.6h elapsed | ~4.8h remaining
-2026-10-20 14:27:39,282 INFO [rank 0] step 1280/2064 | loss 0.1280 | lr 1.32e-04 | 48427 tok/s | 7.7h elapsed | ~4.7h remaining
-2026-10-20 14:34:50,839 INFO [rank 0] step 1300/2064 | loss 0.3189 | lr 1.28e-04 | 48429 tok/s | 7.8h elapsed | ~4.6h remaining
-2026-10-20 14:42:02,393 INFO [rank 0] step 1320/2064 | loss 0.7960 | lr 1.23e-04 | 48432 tok/s | 7.9h elapsed | ~4.5h remaining
-2026-10-20 14:49:13,890 INFO [rank 0] step 1340/2064 | loss 0.6287 | lr 1.19e-04 | 48434 tok/s | 8.1h elapsed | ~4.4h remaining
-2026-10-20 14:56:25,453 INFO [rank 0] step 1360/2064 | loss 0.4928 | lr 1.15e-04 | 48437 tok/s | 8.2h elapsed | ~4.2h remaining
-2026-10-20 15:03:37,045 INFO [rank 0] step 1380/2064 | loss 0.3535 | lr 1.10e-04 | 48439 tok/s | 8.3h elapsed | ~4.1h remaining
-2026-10-20 15:10:48,605 INFO [rank 0] step 1400/2064 | loss 0.2982 | lr 1.06e-04 | 48441 tok/s | 8.4h elapsed | ~4.0h remaining
-2026-10-20 15:18:00,146 INFO [rank 0] step 1420/2064 | loss 0.1965 | lr 1.02e-04 | 48443 tok/s | 8.5h elapsed | ~3.9h remaining
-2026-10-20 15:25:11,651 INFO [rank 0] step 1440/2064 | loss 0.1961 | lr 9.82e-05 | 48445 tok/s | 8.7h elapsed | ~3.8h remaining
-2026-10-20 15:32:23,124 INFO [rank 0] step 1460/2064 | loss 0.1520 | lr 9.43e-05 | 48448 tok/s | 8.8h elapsed | ~3.6h remaining
-2026-10-20 15:39:34,733 INFO [rank 0] step 1480/2064 | loss 0.1754 | lr 9.05e-05 | 48450 tok/s | 8.9h elapsed | ~3.5h remaining
-2026-10-20 15:46:46,436 INFO [rank 0] step 1500/2064 | loss 0.1258 | lr 8.67e-05 | 48451 tok/s | 9.0h elapsed | ~3.4h remaining
-2026-10-20 15:47:22,216 INFO [rank 0] Saved checkpoint at step 1500 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
-2026-10-20 15:54:33,724 INFO [rank 0] step 1520/2064 | loss 0.1315 | lr 8.31e-05 | 48401 tok/s | 9.1h elapsed | ~3.3h remaining
-2026-10-20 16:01:44,312 INFO [rank 0] step 1540/2064 | loss 0.1572 | lr 7.95e-05 | 48404 tok/s | 9.3h elapsed | ~3.2h remaining
-2026-10-20 16:08:54,953 INFO [rank 0] step 1560/2064 | loss 0.1346 | lr 7.60e-05 | 48408 tok/s | 9.4h elapsed | ~3.0h remaining
-2026-10-20 16:16:05,501 INFO [rank 0] step 1580/2064 | loss 0.1335 | lr 7.26e-05 | 48412 tok/s | 9.5h elapsed | ~2.9h remaining
-2026-10-20 16:23:16,059 INFO [rank 0] step 1600/2064 | loss 0.1271 | lr 6.94e-05 | 48416 tok/s | 9.6h elapsed | ~2.8h remaining
-2026-10-20 16:30:26,563 INFO [rank 0] step 1620/2064 | loss 0.1318 | lr 6.62e-05 | 48419 tok/s | 9.7h elapsed | ~2.7h remaining
-2026-10-20 16:37:37,117 INFO [rank 0] step 1640/2064 | loss 0.1584 | lr 6.32e-05 | 48423 tok/s | 9.9h elapsed | ~2.6h remaining
-2026-10-20 16:44:47,716 INFO [rank 0] step 1660/2064 | loss 0.1393 | lr 6.02e-05 | 48426 tok/s | 10.0h elapsed | ~2.4h remaining
-2026-10-20 16:51:58,296 INFO [rank 0] step 1680/2064 | loss 0.1321 | lr 5.74e-05 | 48430 tok/s | 10.1h elapsed | ~2.3h remaining
-2026-10-20 16:59:08,733 INFO [rank 0] step 1700/2064 | loss 0.1417 | lr 5.47e-05 | 48433 tok/s | 10.2h elapsed | ~2.2h remaining
-2026-10-20 17:06:19,228 INFO [rank 0] step 1720/2064 | loss 0.1593 | lr 5.22e-05 | 48436 tok/s | 10.3h elapsed | ~2.1h remaining
-2026-10-20 17:13:29,600 INFO [rank 0] step 1740/2064 | loss 0.3091 | lr 4.98e-05 | 48440 tok/s | 10.5h elapsed | ~1.9h remaining
-2026-10-20 17:17:40,204 INFO [rank 0] Saved checkpoint at step 1750 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
-2026-10-20 17:21:16,399 INFO [rank 0] step 1760/2064 | loss 1.4265 | lr 4.74e-05 | 48397 tok/s | 10.6h elapsed | ~1.8h remaining
-2026-10-20 17:28:27,952 INFO [rank 0] step 1780/2064 | loss 1.6975 | lr 4.53e-05 | 48399 tok/s | 10.7h elapsed | ~1.7h remaining
-2026-10-20 17:35:39,736 INFO [rank 0] step 1800/2064 | loss 1.6714 | lr 4.32e-05 | 48401 tok/s | 10.8h elapsed | ~1.6h remaining
-2026-10-20 17:42:50,874 INFO [rank 0] step 1820/2064 | loss 1.6400 | lr 4.13e-05 | 48403 tok/s | 11.0h elapsed | ~1.5h remaining
-2026-10-20 17:50:01,531 INFO [rank 0] step 1840/2064 | loss 1.6357 | lr 3.96e-05 | 48407 tok/s | 11.1h elapsed | ~1.3h remaining
-2026-10-20 17:57:12,130 INFO [rank 0] step 1860/2064 | loss 1.6244 | lr 3.80e-05 | 48410 tok/s | 11.2h elapsed | ~1.2h remaining
-2026-10-20 18:04:22,788 INFO [rank 0] step 1880/2064 | loss 1.6206 | lr 3.65e-05 | 48413 tok/s | 11.3h elapsed | ~1.1h remaining
-2026-10-20 18:11:33,436 INFO [rank 0] step 1900/2064 | loss 1.6115 | lr 3.52e-05 | 48416 tok/s | 11.4h elapsed | ~1.0h remaining
-2026-10-20 18:18:44,099 INFO [rank 0] step 1920/2064 | loss 1.6138 | lr 3.40e-05 | 48419 tok/s | 11.6h elapsed | ~0.9h remaining
-2026-10-20 18:25:54,843 INFO [rank 0] step 1940/2064 | loss 1.6075 | lr 3.30e-05 | 48421 tok/s | 11.7h elapsed | ~0.7h remaining
-2026-10-20 18:33:05,293 INFO [rank 0] step 1960/2064 | loss 1.6063 | lr 3.21e-05 | 48424 tok/s | 11.8h elapsed | ~0.6h remaining
-2026-10-20 18:40:15,500 INFO [rank 0] step 1980/2064 | loss 1.6019 | lr 3.14e-05 | 48428 tok/s | 11.9h elapsed | ~0.5h remaining
-2026-10-20 18:47:25,587 INFO [rank 0] step 2000/2064 | loss 1.5923 | lr 3.08e-05 | 48431 tok/s | 12.0h elapsed | ~0.4h remaining
-2026-10-20 18:48:01,534 INFO [rank 0] Saved checkpoint at step 2000 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
-2026-10-20 18:55:12,969 INFO [rank 0] step 2020/2064 | loss 1.5895 | lr 3.04e-05 | 48393 tok/s | 12.2h elapsed | ~0.3h remaining
-2026-10-20 19:02:23,852 INFO [rank 0] step 2040/2064 | loss 1.5956 | lr 3.01e-05 | 48396 tok/s | 12.3h elapsed | ~0.1h remaining
-2026-10-20 19:09:34,788 INFO [rank 0] step 2060/2064 | loss 1.5888 | lr 3.00e-05 | 48398 tok/s | 12.4h elapsed | ~0.0h remaining
-2026-10-20 19:11:36,487 INFO [rank 0] Saved checkpoint at step 2064 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
-2026-10-20 19:11:37,289 INFO [rank 0] Done. stage6_12.85b at step 2064, 2.16B tokens seen, 12.43h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage6_12.85b

training_state.json

@@ -1,4 +0,0 @@
-{
-  "stage": "stage6_12.85b",
-  "global_step": 2064
-}