OpenPeerAI/My-Name-is-Riemann-Prerelease

My Name is Riemann Prerelease

My Name is Riemann is a 13.1 billion parameter decoder-only language model grown rung by rung from the published weights of OpenPeerAI/OpenPeerLLM and fine-tuned with the OpenPeer-AI-NTK-Trainer gate controller instead of LoRA. It is the first public release of the My Name is Riemann line and is published as a prerelease: complete and loadable, with the final 13.1B stage trained on 2.16 billion tokens from a 2.30 billion token corpus, and substantially more training planned for later releases.

Model Details

Model Description

My Name is Riemann starts from OpenPeerLLM's checkpoint (163M parameters, loaded tensor for tensor into a faithful reimplementation of its architecture) and climbs a ten rung growth ladder. At each rung the previous checkpoint is widened and deepened into the next shape, so that every stage inherits what the one before it learned, and training continues on a mixed corpus of agentic coding transcripts, Arabic and English encyclopedia text, executable world-model tasks, code-to-video pairs and raw source code. The Prerelease is stage 6 of that ladder.

Fine-tuning uses the OpenPeer-AI-NTK-Trainer: a few thousand signed log-gates are fitted on the frozen model's layer outputs with an objective derived from neural tangent kernel theory. The base weights are never modified. The fitted controller is a small separate file that ships beside the checkpoint.

The model is named after Riemann Computing.

• Developed by: OpenPeer AI, Riemann Computing Inc., and Andrew Magdy Kamal
• Model type: Decoder-only causal language model (OpenPeerLLM architecture, RiemannForCausalLM)
• Language(s) (NLP): English, Arabic, and source code
• License: Open Innovation License, Version 2 (OPNL-2) and MIT
• Finetuned from model: OpenPeerAI/OpenPeerLLM
• Release status: Prerelease (stage 6 of the growth ladder)
• Parameters: 13,110,702,080 (13.1B), fp32, 52.4 GB in eleven safetensors shards
• Training tokens: 2.16B at stage 6, 6.5B cumulative across the ladder
• Trained: 19 to 20 October 2026

Model Sources

• Repository: This Hugging Face repository (model weights, NTK controller, remote code, training pipeline)
• Base model: OpenPeerAI/OpenPeerLLM, DOI 10.57967/hf/6469
• Fine-tuning method: OpenPeerAI/OpenPeer-AI-NTK-Trainer and ntkmirror
• Demo: My Name is Riemann Prerelease - Basic Demo (article completion, token-level surprisal, fluency test). See also the OpenPeerLLM Demo, which puts the 13.1B Prerelease next to its 163M starting rung, and the NTK Trainer Demo, which fits a gate controller on this model live.

Uses

Direct Use

Text generation with an emphasis on code and step-by-step reasoning, in English and Arabic. The training mix is dominated by agentic coding transcripts that include tool calls and tool outputs, so the model is best suited to code completion, code explanation, multi-step problem solving with explicit intermediate reasoning, and bilingual encyclopedic question answering.

Downstream Use

The checkpoint is a suitable base for continued pretraining on a larger corpus, for fitting additional NTK gate controllers on task-specific prompt and completion data, or for conventional fine-tuning. Because every rung of the growth ladder is a standard save_pretrained() folder, the intermediate stages can also be used directly where a smaller model is preferred.

Out-of-Scope Use

The Prerelease is not an instruction-tuned assistant and has not been aligned or safety-tuned. It should not be deployed in settings that require factual reliability, in high-stakes decision making, or to generate content in languages other than English and Arabic. The multimodal pathway is a demonstration of the wiring, not a capable video model.

Bias, Risks, and Limitations

• Prerelease token budget. Stage 6 has seen 2.16 billion tokens, about 94 percent of one pass over the 2.30 billion token corpus. That is far below a compute-optimal budget for a 13B model and is the main thing later releases will change. Expect capability well below similarly sized models trained on hundreds of billions of tokens.
• Uneven corpus coverage. The corpus is read in file order, so stages 2 to 5 (1.08 billion tokens each) saw mainly the first half of the file, which is the Arabic and English Wikipedia block. The stage 6 run was configured as two passes of an estimated 1.08 billion tokens; because the corpus is in fact 2.30 billion tokens, that run instead covered the Fable transcripts, PowerCoding and ExeWorldBench once and the Wikipedia block roughly twice. Training loss on the second visit to the Wikipedia block dropped as low as 0.12, which reflects memorisation of repeated text rather than generalisation; the held-out validation figures in the Evaluation section are the meaningful quality measure. Later releases shuffle document order per pass.
• Narrow data mix. Five datasets, dominated by synthetic agentic coding traces (Fable 5.1), Arabic and English Wikipedia (jabarti), and synthetic Python (PowerCoding). The model inherits the biases, errors and gaps of those sources and of the models that generated the synthetic portions.
• Inherited behaviour. The lowest rung is OpenPeerLLM's own checkpoint, so any properties of that model carry forward.
• Context length. 2,048 tokens at stage 6 (1,024 through stage 3).
• No alignment. The model has had no RLHF, DPO or safety tuning.

Recommendations

Users should treat outputs as drafts requiring review, apply their own content filtering before any user-facing deployment, and prefer further training for production use. The NTK controller shipped with the checkpoint is fitted on question answering pairs and improves held-out NLL on that distribution; it does not add safety behaviour.

How to Get Started with the Model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "<repo-id>"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, torch_dtype=torch.bfloat16
)

prompt = "user: Write a function that checks whether a number is prime.\nassistant:"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=200, do_sample=True, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

trust_remote_code=True is required because the repository ships its own configuration_riemann.py and modeling_riemann.py. Weights are stored in fp32 (about 52 GB); loading in bf16 halves the memory footprint.

To apply the NTK gate controller:

from ntkmirror import ForwardFineTuner

tuner = ForwardFineTuner(model, tokenizer)
tuner.load("ntk_controller/stage6_12.85b.pt", allow_model_mismatch=True)
tuner.controller.attach()

Training Details

Training Data

Five datasets are combined into a single pretraining corpus of 7,218,101 documents (2.30 billion GPT-2 tokens, 318 tokens per document on average), plus a separate file of 73,824 prompt and completion pairs used to fit the NTK controller.

Dataset	Portion used	Role
MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x	All 10,000 transcripts	Pretraining; first user turn and final assistant turn form an NTK pair
bakrianoo/jabarti-llm-dataset	pretrain split in full (7,053,893 documents) and finetune split in full (about 64,000 pairs)	Pretraining; finetune pairs also feed the NTK controller
HelenPeng/ExeWorldBench	Full test split, 779 tasks	Pretraining
code-video-model/CodeVideoBench	All 22 cases	Prompts and code in pretraining; videos feed the multimodal pathway
Tiiny/PowerCoding	About 90,000 Python examples from several raw shards	Pretraining

Fable transcripts are serialised turn by turn with the role as a prefix; tool calls are wrapped in <tool_call name=...> tags and tool outputs are prefixed with the tool name. Text read from these datasets is treated strictly as data; nothing in any record is executed or interpreted as an instruction by the training code.

Training Procedure

Training proceeds along a growth ladder (see Technical Specifications for the full table). A stage is trained, its checkpoint is grown into the next shape, and training continues. Width growth copies existing rows and columns into new positions at one tenth scale so the grown network starts near the function it already computes; depth growth inserts new blocks with zeroed output projections so they begin as identity passthroughs.

The climb to the Prerelease is split across two environments.

Stage 1 on AMD consumer hardware. Stage 0 is materialised from OpenPeerLLM's published weights and grown to stage 1 (331M parameters). Stage 1 trains for 1,500 optimizer steps (about 9 million tokens) on a Radeon RX 7600M XT with a Ryzen AI 9 465 CPU running one in four micro-batches on a mirrored copy of the model. An NTK controller is fitted on the result.

Stages 2 through 6 on an 8x AMD MI325X node. At each rung the checkpoint is grown, pretrained for 1,032 optimizer steps of 1,048,576 tokens (1.08 billion tokens) with sharded data parallelism in bf16, and given an NTK controller warm-started from the previous rung's controller. Stage 6 ran for 2,064 steps (2.16 billion tokens). The whole climb took 20.4 node-hours including growth, controller fits and the final upload.

Rung	Parameters	Tokens	Pretrain wall clock	Rung total incl. grow and fit	Training loss, first to last logged step
stage2_538m	0.60B	1.08B	52 min	55 min	6.78 to 2.22
stage3_1.19b	1.27B	1.08B	49 min	54 min	2.17 to 1.88
stage4_2.65b	2.78B	1.08B	101 min	110 min	2.83 to 1.72
stage5_6.66b	6.86B	1.08B	197 min	216 min	1.88 to 1.63
stage6_12.85b	13.1B	2.16B	746 min	786 min	2.03 to 1.59

Stage 2 ran at roughly 16 percent of the MI325X's bf16 peak, which is typical for a sub-1B model where per-step overheads dominate. Stages 3 to 6 used fused scaled-dot-product attention, gradient-and-optimizer sharding instead of full parameter sharding, and no activation recomputation, since the node's 256 GB per GPU makes that memory saving unnecessary; stage 3 ran 50 percent faster than it would have on the stage 2 configuration. The per-rung controller fit summaries in ntk_controller/ record the validation figures, fit times and gate counts for each rung.

The NTK fits are what keep the budget small. Fitting a few thousand gates on a frozen model is forward-dominated, takes minutes per rung, and carries no optimizer state, where a supervised fine-tuning epoch at 13B would take hours with the full training memory footprint. Carrying each rung's gates forward means only the grown dimensions need new gates.

Preprocessing

Documents are written one per line with internal newlines encoded as U+2029. At training time each document is tokenised with the GPT-2 byte-level BPE tokenizer (50,257 tokens), terminated with the end-of-text token, and packed into fixed-length windows. Under distributed training each rank reads a disjoint subset of lines.

Training Hyperparameters

	Stage 1 (AMD Radeon, DirectML)	Stages 2 to 6 (AMD MI325X, ROCm)
Training regime	fp32	bf16 mixed precision, fp32 master weights
Optimizer	Adafactor (relative step, warmup init)	AdamW, betas 0.9 / 0.95, weight decay 0.1
Learning rate	Adafactor relative	3e-4 peak, 200 step linear warmup, cosine decay to 3e-5
Gradient clipping	none	1.0
Sequence length	384	1,024 (stages 2, 3), 2,048 (stages 4 to 6)
Tokens per optimizer step	6,144	1,048,576
Gradient checkpointing	yes	stage 2 only; off from stage 3 onward
Attention kernel	explicit matmul and softmax	fused scaled-dot-product attention
Parallelism	single GPU plus CPU mirror	FSDP across 8 GPUs (full shard at stage 2, gradient and optimizer sharding from stage 3)

Speeds, Sizes, Times

Stage	Parameters	Checkpoint size (fp32)	Hardware	Throughput	Wall clock
stage1_279m	331M	1.3 GB	Radeon RX 7600M XT + Ryzen AI 9 465	350 tokens/s (17.5 s/step)	about 7 h for 1,500 steps
stage2_538m	602M	2.4 GB	8x MI325X	345,600 tokens/s	52 min for 1,032 steps
stage3_1.19b	1.27B	5.1 GB	8x MI325X	370,300 tokens/s	49 min for 1,032 steps
stage4_2.65b	2.78B	11.1 GB	8x MI325X	178,500 tokens/s	1 h 41 min for 1,032 steps
stage5_6.66b	6.86B	27.5 GB	8x MI325X	91,800 tokens/s	3 h 17 min for 1,032 steps
stage6_12.85b	13.1B	52.4 GB	8x MI325X	48,400 tokens/s	12 h 26 min for 2,064 steps

Stage 6 is 52.4 GB on disk in fp32, saved as eleven safetensors shards. Total node time from first boot to final upload was 21.0 hours.

Evaluation

Testing Data, Factors & Metrics

Testing Data

A held-out slice of 1,000 examples from the NTK prompt and completion file (drawn from the jabarti finetune split and Fable first-turn pairs), never used for controller fitting, is scored before and after each rung's controller is fitted. The remaining 19,000 examples are used for fitting. Some of this text also appears in the pretraining corpus, so these figures measure in-distribution next-token prediction rather than generalisation to unseen domains.

Factors

Results are reported per rung of the growth ladder, with and without the NTK controller attached.

Metrics

Negative log-likelihood (NLL) and next-token accuracy on the held-out slice. These measure how well the frozen model predicts the fine-tuning distribution and how much the controller improves it; they are not a general capability benchmark.

Results

Rung	Parameters	Gates	NLL, base	NLL, with controller	Token accuracy, base	Token accuracy, with controller
stage1_279m	331M	4,096	7.044	7.021	11.97%	12.30%
stage2_538m	602M	4,096	1.968	1.954	54.55%	54.73%
stage3_1.19b	1.27B	4,631	1.629	1.618	62.45%	62.59%
stage4_2.65b	2.78B	4,102	1.512	1.505	65.08%	65.17%
stage5_6.66b	6.86B	4,102	1.451	1.447	66.41%	66.47%
stage6_12.85b (Prerelease)	13.1B	6,732	1.323	1.310	67.73%	67.84%

The stage 1 figures were measured on a 200-example slice during the local phase (4,000 fitting examples); stages 2 to 6 use the 1,000-example slice. The per-rung JSON summaries in ntk_controller/ record the same figures along with fit time, learning rate and gate counts.

Summary

Held-out NLL falls monotonically along the ladder, from 7.04 at stage 1 to 1.32 at stage 6, and next-token accuracy rises from 12 percent to 68 percent, with each rung inheriting and extending the previous one. The NTK controller gives a consistent small improvement on top of the frozen weights at every rung (0.004 to 0.023 nats) at a cost of under 150 KB per controller. Broader benchmark results are not yet available for this prerelease; users should run their own evaluation before relying on it.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: Stage 1: AMD Radeon RX 7600M XT (eGPU) and AMD Ryzen AI 9 465. Stages 2 to 6: 8x AMD Instinct MI325X 256 GB.

• Hours used: Stage 1: about 7 hours on one laptop with an external GPU. Stages 2 to 6: 21.0 node-hours (168 GPU-hours), of which 19.1 hours were pretraining and the rest growth, controller fitting, setup and upload.

• Cloud Provider: DigitalOcean (stages 2 to 6)

• Compute Region: NYC2 (New York)

• Carbon Emitted: The MI325X ran at its 1,000 W board power cap throughout, so GPU power alone was about 168 kWh (8 GPUs x 21 h x 1 kW); with host and cooling overhead at a typical PUE of 1.3, about 220 kWh total. The node was in New York, where the grid's total output emission rate is 466.6 lb CO2e per MWh, or 0.212 kg per kWh, according to the EPA's eGRID2023 summary tables. At that rate the cloud phase produced roughly 47 kg of CO2e, about the same as a 120 mile drive in an average petrol car. Stage 1 on the laptop and eGPU added about 1.5 kWh.

  Source: U.S. Environmental Protection Agency. (2025). eGRID2023 Summary Tables (rev. 2, March 2025), Table 3: State Output Emission Rates. epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf

Technical Specifications

Model Architecture and Objective

OpenPeerLLM architecture, reimplemented as RiemannForCausalLM with module names matched to the published checkpoint so its 197 tensors load with a plain state-dict copy. Pre-norm transformer blocks with separate query, key, value and output projections, GELU MLP, untied input and output embeddings, learned absolute position embeddings. Two additions over the reference implementation: a causal attention mask (required for autoregressive training) and inputs_embeds support for the multimodal prefix. The objective is standard next-token cross-entropy.

The growth ladder:

Stage	Parameters	Hidden size	Layers	Heads	FFN size	Context
stage0_124m	163M	768	12	12	3,072	1,024
stage1_279m	331M	1,024	18	16	4,096	1,024
stage2_538m	602M	1,280	24	20	5,120	1,024
stage3_1.19b	1.27B	1,600	36	25	6,400	1,024
stage4_2.65b	2.78B	2,560	32	32	10,240	2,048
stage5_6.66b	6.86B	4,096	32	32	16,384	2,048
stage6_12.85b (Prerelease)	13.1B	5,120	40	40	20,480	2,048
stage7_29.97b	30.3B	7,168	48	56	28,672	2,048
stage8_51.97b	52.4B	8,192	64	64	32,768	2,048
stage9_64.86b	65.3B	8,192	80	64	32,768	2,048

Parameter counts include both embedding matrices. Stage names use the tied-embedding count, which is the convention most published sizes follow. Stages 7 to 9 are defined so the same checkpoints and growth code keep working for later releases.

NTK gate controller. A sparse set of signed log-gates, each bound to one decoder layer and one residual-stream channel, that multiply the layer output by exp(s) with s bounded to a small interval. Gates are selected by scoring activations on the fine-tuning data, then fitted with AdamW against next-token loss while the base model stays frozen.

Multimodal pathway. A small convolutional frame encoder samples eight frames per CodeVideoBench video, mean-pools their embeddings, and projects the result into the model's hidden size as one prefix token. It runs on a secondary GPU during training.

Hybrid CPU and GPU training (stage 1). The primary GPU runs most micro-batches. A CPU mirror of the model runs one in four micro-batches on a background thread concurrently with the GPU; its gradients are summed into the main model's before each optimizer step and the mirror is resynchronised afterwards. DirectML does not split one model across two AMD GPUs, so the secondary GPU is given the frame encoder as a separate job.

Compute Infrastructure

Hardware

Stage 1:

Component	Detail
System	ASUS Zenbook S16 (UM5606GA)
CPU	AMD Ryzen AI 9 465, 10 cores / 20 threads, with integrated NPU
Primary GPU	AMD Radeon RX 7600M XT, 8 GB GDDR6, external eGPU
Secondary GPU	AMD Radeon 880M integrated graphics
System memory	32 GB LPDDR5X
Storage	2 TB NVMe

Stages 2 to 6:

Component	Detail
Node	DigitalOcean gpu-mi325x8-2048gb: 8x AMD Instinct MI325X, 256 GB HBM3E each (2 TB total GPU memory), Infinity Fabric interconnect
Host	160 vCPUs, 1.26 TB system memory, 2 TB NVMe
Precision	bf16 compute, fp32 master weights
Parallelism	PyTorch FSDP: full sharding at stage 2, gradient and optimizer state sharding from stage 3
Checkpoints	fp32 safetensors, 5 GB shards

Memory guidance for keeping a stage resident on one device (fp32 weights; bf16 halves these): stages 0 to 3 fit on an 8 GB GPU with gradient checkpointing at batch 1; stage 4 (about 11 GB of weights) needs a 24 GB class GPU; stage 5 (about 27 GB) a 48 GB GPU; stage 6 (about 52 GB) trains comfortably on 8x 256 GB with parameters resident on every GPU and only gradients and optimizer state sharded.

Software

• Stage 1: Windows 10 Pro 22H2, PyTorch 2.4.1, torch-directml 0.2.5, transformers 4.56, AMD Adrenalin 32.0.31041
• Stages 2 to 6: Ubuntu 24.04, ROCm 7.1 host driver, PyTorch 2.9.1 (ROCm 6.4 build), RCCL, transformers 4.57, safetensors, ntkmirror 0.4
• Tokenizer: GPT-2 byte-level BPE via transformers

Citation

BibTeX:

@misc{my-name-is-riemann-prerelease,
  author    = {{OpenPeer AI} and {Riemann Computing Inc.} and Kamal, Andrew Magdy},
  title     = {My Name is Riemann Prerelease},
  year      = {2026},
  publisher = {Hugging Face},
  journal   = {Hugging Face Model Hub},
  note      = {Stage 6 of the My Name is Riemann growth ladder, 13.1B parameters}
}

APA:

OpenPeer AI, Riemann Computing Inc., & Kamal, A. M. (2026). My Name is Riemann Prerelease [Language model]. Hugging Face Model Hub.

Sources cited

Base model and trainer:

• Nassief, A. M. K. (2025). OpenPeerLLM: A Decentralized Language Model. Stark Publishing Group, Hugging Face Model Hub. DOI 10.57967/hf/6469. Released under OPNL, MIT and CC BY 4.0.

  @misc{openpeer-llm,
    author    = {Andrew Magdy Kamal Nassief},
    title     = {OpenPeerLLM: A Decentralized Language Model},
    year      = {2025},
    publisher = {Stark Publishing Group},
    journal   = {Hugging Face Model Hub},
    doi       = {10.57967/hf/6469}
  }
  

• OpenPeer AI. OpenPeer-AI-NTK-Trainer. huggingface.co/OpenPeerAI/OpenPeer-AI-NTK-Trainer. MIT. Depends on ntkmirror.

Datasets:

• MoreThought. Fable-5.1-Max-Reasoning-Filtered-10000x. Hub. Apache 2.0. No DOI or paper issued.

• bakrianoo. jabarti-llm-dataset. Hub. CC BY-SA 4.0. No DOI or paper issued. Derived in part from CohereLabs/wikipedia-2023-11-embed-multilingual-v3.

• Peng, X., Fang, Z., Wang, R., Lv, Y., Li, H., Zhang, G., Ma, X., Lu, Y., Chen, L., Che, Z., & Zhao, F. (2026). ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling. arXiv:2610.15166. arxiv.org/abs/2610.15166. Dataset Hub, CC BY 4.0.

  @article{peng2026exeworldbench,
    author  = {Peng, Xinge and Fang, Zhen and Wang, Ruibing and Lv, Yongxuan and Li, Haodong and Zhang, Guohui and Ma, Xiaoxiao and Lu, Yiting and Chen, Lin and Che, Zehui and Zhao, Feng},
    title   = {ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling},
    journal = {arXiv preprint arXiv:2610.15166},
    year    = {2026}
  }
  

• code-video-model. CodeVideoBench. Hub. Cases and baseline controls CC BY 4.0, project code MIT. No DOI or paper issued.

• Tiiny. PowerCoding. Hub. Apache 2.0. No DOI or paper issued for the dataset. Its card states the data was synthesised with Qwen 2.5 models using the methods of:

  • Hao, X., Zhu, R., Zhang, G., Shen, K., & Li, C. (2025). Reformulation for Pretraining Data Augmentation. arXiv:2502.04235. arxiv.org/abs/2502.04235.
  • Ge, T., Chan, X., Wang, X., Yu, D., Mi, H., & Yu, D. (2024). Scaling Synthetic Data Creation with 1,000,000,000 Personas. arXiv:2406.20094. arxiv.org/abs/2406.20094.
  • Zhu, Y., Zhou, K., Mao, K., et al. (2024). YuLan: An Open-source Large Language Model. arXiv:2406.19853. arxiv.org/abs/2406.19853.

Tokenizer:

• Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI.

License:

• Stark Drones Corporation & Kamal, A. M. (2020). The Open Innovation License, Version 2. github.com/OPNL/OPNL/tree/main/Version%202.

Environmental impact:

• U.S. Environmental Protection Agency. (2025). eGRID2023 Summary Tables (rev. 2, March 2025), Table 3: State Output Emission Rates. epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf.
• Lacoste, A., Luccioni, A., Schmidt, V., & Dandres, T. (2019). Quantifying the Carbon Emissions of Machine Learning. arXiv:1910.09700. arxiv.org/abs/1910.09700.

Glossary

• Growth ladder. The sequence of ten model shapes, each produced by widening and deepening the previous stage's trained checkpoint.
• Rung / stage. One shape on the ladder, named by its approximate tied-embedding parameter count (for example stage6_12.85b).
• NTK gate controller. The small set of per-layer, per-channel multiplicative gates fitted by the OpenPeer-AI-NTK-Trainer on a frozen model; the project's replacement for LoRA.
• Warm start. Initialising a rung's controller from the previous rung's fitted gates before scoring new gates for the grown dimensions.
• Prerelease. The first public build of the line, at stage 6, with a deliberately small token budget relative to its size.

More Information

Release history

Release	Ladder stage	Parameters	Training tokens	Status
My Name is Riemann Prerelease	stage6_12.85b	13.1B	2.16B at stage 6; 6.5B cumulative across stages 1 to 6	This release

Later releases continue training the stage 6 checkpoint on a larger corpus and may climb to stage 7 and beyond. Each will be published under its own name with the Prerelease kept available.

License

Released under the Open Innovation License, Version 2 (OPNL-2) for the project as a whole, including model weights, configuration and documentation, and under the MIT License for the software implementation. OPNL-2 permits copying, modification, redistribution and commercial use on the condition that the license text and copyright notice are kept verbatim in any redistribution. Both texts are in LICENSE.

Reproducing the training

The complete pipeline is in this repository under training/. Download that folder (for example with hf download OpenPeerAI/My-Name-is-Riemann-Prerelease --include "training/*"), run the commands below from inside it, and set RIEMANN_ROOT to a directory with enough free space; data, checkpoints, controllers and the download cache all live under it.

Data:

python -m riemann.data

Stage 1 on a single AMD GPU:

python scripts/pretrain.py --stage stage0_124m --max-steps 0
python scripts/grow_and_continue.py --from-stage stage0_124m --to-stage stage1_279m
python scripts/supervise.py --stage stage1_279m --max-steps 1500 -- --batch-size 1 --grad-accum 16 --save-every 50
python scripts/fit_ntk.py --stage stage1_279m

The supervisor relaunches pretrain.py --resume from the last saved checkpoint if the process exits early and waits for the primary GPU to reappear if it has dropped off the bus, which external GPUs occasionally do under sustained load. Arguments after -- are forwarded to pretrain.py.

Stages 2 to 6 on an 8-GPU node (ROCm or CUDA), after copying data/, checkpoints/stage1_279m/ and ntk_controllers/stage1_279m.pt under RIEMANN_ROOT on the node and installing PyTorch for the node's GPU backend plus requirements-cuda.txt (minus torch):

python scripts/grow_and_continue.py --from-stage stage1_279m --to-stage stage2_538m
torchrun --nproc_per_node 8 scripts/pretrain_cuda.py --stage stage2_538m --epochs 1
python scripts/fit_ntk.py --stage stage2_538m --init-from stage1_279m

Repeat for stage2_538m -> stage3_1.19b, stage3_1.19b -> stage4_2.65b, stage4_2.65b -> stage5_6.66b and stage5_6.66b -> stage6_12.85b, using --epochs 2 on the final rung. On GPUs with 128 GB or more, add --sharding grad_op --no-activation-checkpointing for a substantial speedup. cloud/bootstrap.sh runs this whole sequence unattended, including a short FSDP smoke test before the first rung.

Command-line flags

pretrain.py (single device, DirectML or CUDA or CPU):

• --batch-size, --grad-accum, --seq-len override the stage defaults.
• --cpu-micro-step-every N sends one in every N micro-batches to the CPU mirror; --no-cpu-accel disables the CPU lane; --cpu-threads caps the thread count.
• --multimodal enables the video pathway; --multimodal-every N controls how often a video example is interleaved.
• --save-every, --log-every, --max-steps, --resume.
• --no-pretrained-init builds stage 0 from random weights instead of OpenPeerLLM's published checkpoint.

pretrain_cuda.py (multi-GPU, launched with torchrun):

• --epochs N passes over the corpus; --max-steps as an alternative cap.
• --batch-size, --grad-accum, --seq-len per GPU.
• --lr, --warmup-steps, --weight-decay, --grad-clip.
• --save-every, --log-every, --max-shard-size.

fit_ntk.py:

• --init-from STAGE warm-starts from a previous rung's controller.
• --gates, --steps, --lr, --max-examples, --val-fraction, --early-stop-patience control the fit.
• --cpu runs the fit on the CPU so a GPU can keep pretraining.

riemann.data:

• --only name [name ...] runs a subset of loaders.
• --append adds to the existing corpus files instead of replacing them.
• --jabarti-skip N resumes the Arabic pretrain stream from row N.

Repository notes

The complete training pipeline is published in this repository under training/: the riemann/ package, the scripts/ entry points, the configs/ ladder definition, the cloud/ node automation, and requirements-cuda.txt. The two files at the repository root, configuration_riemann.py and modeling_riemann.py, are identical copies of the corresponding riemann/ modules placed where trust_remote_code=True expects them. Source files are kept free of comments and docstrings by convention; everything that would normally be a comment is recorded here instead. Paths below are relative to training/.

riemann/configuration_riemann.py, riemann/modeling_riemann.py. The RiemannConfig and RiemannForCausalLM classes described under Technical Specifications. RiemannForCausalLM mixes in GenerationMixin so generate() works on current transformers releases. Attention uses PyTorch's fused scaled-dot-product kernel on CUDA, ROCm and CPU, and falls back to an explicit matmul and softmax on DirectML, where the fused kernel is unavailable.

riemann/openpeer_weights.py. Downloads OpenPeerLLM's model.safetensors and copies it into a freshly built stage 0 model, reporting any missing or unexpected keys.

riemann/growth_ladder.py, configs/growth_ladder.json. The GrowthStage dataclass and the ten-rung ladder. The JSON file is generated from the Python definitions and contains no free-text fields. approx_param_count() includes both embedding matrices; to_riemann_config() maps a stage onto a RiemannConfig.

riemann/growth.py. grow_width expands every linear, embedding and layer-norm tensor by copying randomly chosen existing rows or columns into the new positions at one tenth scale. grow_depth appends new transformer blocks with zeroed attention.out and mlp.dense_4h_to_h weights. grow_to applies width then depth. load_checkpoint_and_grow reads a saved stage and returns the grown model.

riemann/device.py. main_device() returns the first DirectML adapter, otherwise the first CUDA device, otherwise CPU. secondary_device() returns the second adapter of the same backend or falls back to the first. configure_cpu_threads() sets PyTorch's intra-op and inter-op pools to all logical cores.

riemann/paths.py. Every on-disk location derives from RIEMANN_ROOT.

riemann/dataset.py. PackedTextDataset streams the corpus, tokenises, appends end-of-text, and packs into fixed windows. It takes a shard index and count so each distributed rank reads a disjoint subset of lines, composed with PyTorch's per-worker split. It also takes an epoch number: epoch 0 reads the file in order (as the Prerelease was trained), and every later epoch rotates the shard assignment and shuffles documents through a seeded reservoir buffer so a second pass sees a different ordering and no rank repeats its own slice. This was added after the Prerelease run and applies to later releases.

scripts/count_corpus_tokens.py. Tokenises the whole corpus once and writes corpus_token_count.json with the exact token total and per-document average, which pretrain_cuda.py uses to size an epoch.

riemann/data.py. One loader per dataset. Fable is read from its single JSONL shard; the Arabic pretrain split is streamed and resumable from any row; PowerCoding is read by HTTP byte range so only the needed prefix of each shard is downloaded. Output goes through ResilientWriter, which reopens the destination in append mode and retries with backoff if the volume disconnects mid-write. Hub downloads use the same retry policy.

riemann/multimodal.py. VideoPrefixAdapter holds the frame encoder and the projection into the model's hidden size; build_multimodal_training_examples() decodes CodeVideoBench's videos and pairs them with their prompts.

scripts/pretrain.py. Single-device training. Loads or builds the stage, optionally constructs the CPU mirror, and runs the packed-corpus loop. Checkpoints are saved after a round trip to CPU because DirectML tensors cannot be serialised directly; training_state.json records the stage and global step so --resume continues the counter.

scripts/pretrain_cuda.py. Multi-GPU training under torchrun on CUDA or ROCm. Wraps each transformer block in FSDP with bf16 compute and fp32 master weights, shards the corpus across ranks, and trains with AdamW under linear warmup and cosine decay for a fixed number of corpus passes. --sharding full shards parameters, gradients and optimizer state; --sharding grad_op keeps parameters resident on every GPU and shards only gradients and optimizer state, which is faster when memory allows. --no-activation-checkpointing disables per-block recomputation for the same reason. The epoch size comes from corpus_token_count.json when present. Rank 0 gathers the full state dict and writes sharded fp32 safetensors in the same layout pretrain.py produces. --dry-run-cpu runs the same loop under DDP on the gloo backend for testing without GPUs.

scripts/supervise.py. A restart loop around pretrain.py for single-device runs: reads the last saved step, waits for the primary GPU, launches pretrain.py --resume, and repeats until the target step is reached.

scripts/grow_and_continue.py. Grows a finished stage into a later one and writes the new checkpoint plus tokenizer.

scripts/prepare_datasets.py. Thin command-line wrapper around riemann.data.prepare_all with --only, --append and --data-dir.

scripts/fit_ntk.py. Drives ntkmirror's ForwardFineTuner on a RiemannForCausalLM checkpoint. Holds out a validation slice, scores and selects gates, fits with early stopping and best-on-validation selection, and saves the controller with a JSON summary. With --init-from, the previous rung's gates are carried onto the grown model and fresh gates are scored only for the new layers and channels.

cloud/bootstrap.sh. The unattended node script used for stages 2 to 6. Creates a virtual environment, installs the ROCm or CUDA PyTorch build to match the node, pulls the staged data and stage 1 checkpoint from the private Hub dataset named in STAGING_REPO, runs a short 8-rank FSDP smoke test on a tiny model, then for each rung grows, pretrains and fits the controller, skipping any rung whose .rung_done marker exists. Per-rung training flags are read from pretrain_extra_<stage>.txt under RIEMANN_ROOT so settings can be tuned between rungs without editing the script. Leaves the final checkpoint and controllers on the node for archiving.

cloud/launch_node.ps1. Creates the GPU droplet with doctl using the SSH key name and staging repository given on the command line, waits for SSH, ships the code bundle and the Hugging Face token file, and starts bootstrap.sh under nohup so it survives disconnects.

cloud/mirror_node.ps1, cloud/pull_stage.ps1. Archive completed rungs from the node to the local machine. mirror_node.ps1 polls for .rung_done markers and pulls each finished stage; pull_stage.ps1 fetches one stage file by file with per-file SHA-256 verification and retry, which is the more reliable path for checkpoints above 20 GB.

cloud/verify_before_destroy.ps1. Checks that the node reports completion and that every rung and controller is mirrored and verified locally, optionally also comparing shard sizes against a Hub repository when -ModelRepo is given, and prints a single safe-or-not verdict before the droplet is destroyed.

cloud/make_tiny_stage.py. Writes a two-layer, 64-wide checkpoint in the stage 2 slot for the FSDP smoke test.

configs/growth_ladder.json. The serialised ladder, generated from riemann/growth_ladder.py.

requirements-cuda.txt. Python dependencies for the multi-GPU pipeline, excluding torch, which must be installed from the index matching the node's GPU backend.

The upstream OpenPeerLLM source and the NTK trainer were consulted as reference during development but are not redistributed here; see their own repositories under Sources cited.

Authors

OpenPeer AI, Riemann Computing Inc., and Andrew Magdy Kamal

Contact

Open a discussion on this model's Hugging Face repository.