README.md

---

language:
  - en
  - ar
license: other
license_name: opnl-2-and-mit
license_link: LICENSE
library_name: transformers
pipeline_tag: text-generation
tags:
  - amd
  - ntk
  - code
  - fsdp
  - rocm
  - coding
  - mi325x
  - decoder
  - code gen
  - directml
  - openpeer
  - ryzen-ai
  - reasoning
  - rewriting
  - multimodal
  - paraphrase
  - prerelease
  - ntk-trainer
  - openpeerllm
  - safetensors
  - transformers
  - growth-ladder
  - style-transfer
  - text-rewriting
  - riemann-computing
  - progressive-growth
base_model: OpenPeerAI/OpenPeerLLM
datasets:
  - MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x
  - bakrianoo/jabarti-llm-dataset
  - HelenPeng/ExeWorldBench
  - code-video-model/CodeVideoBench
  - Tiiny/PowerCoding
model-index:
  - name: My Name is Riemann Prerelease
    results:
      - task:
          type: text-generation
          name: Held-out next-token prediction
        dataset:
          type: bakrianoo/jabarti-llm-dataset
          name: My Name is Riemann NTK validation slice, base weights
          config: stage6_12.85b-base
          split: validation
        metrics:
          - type: nll
            value: 1.3228
            name: Validation NLL (1,000 held-out jabarti finetune and Fable first-turn pairs), frozen stage 6 weights
          - type: accuracy
            value: 67.73
            name: Validation next-token accuracy (%), frozen stage 6 weights
      - task:
          type: text-generation
          name: Held-out next-token prediction
        dataset:
          type: bakrianoo/jabarti-llm-dataset
          name: My Name is Riemann NTK validation slice, with NTK gate controller
          config: stage6_12.85b-ntk
          split: validation
        metrics:
          - type: nll
            value: 1.3104
            name: Validation NLL (1,000 held-out jabarti finetune and Fable first-turn pairs), stage 6 weights plus 6,732-gate controller
          - type: accuracy
            value: 67.84
            name: Validation next-token accuracy (%), stage 6 weights plus controller
      - task:
          type: text-generation
          name: Pretraining
        dataset:
          type: MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x
          name: My Name is Riemann pretraining corpus (five datasets, 2.30B GPT-2 tokens)
          config: stage6_12.85b
          split: train
        metrics:
          - type: loss
            value: 1.5888
            name: Final training cross-entropy at step 2,064 (2.16B tokens, bf16, 1,048,576 tokens per step)
---


# My Name is Riemann Prerelease

My Name is Riemann is a 13.1 billion parameter decoder-only
language model grown rung by rung from the published weights of
[OpenPeerAI/OpenPeerLLM](https://huggingface.co/OpenPeerAI/OpenPeerLLM)
and fine-tuned with the
[OpenPeer-AI-NTK-Trainer](https://huggingface.co/OpenPeerAI/OpenPeer-AI-NTK-Trainer)
gate controller instead of LoRA. It is the first public release of the My
Name is Riemann line and is published as a prerelease: complete and
loadable, with the final 13.1B stage trained on 2.16 billion tokens from a
2.30 billion token corpus, and substantially more training planned for
later releases.

## Model Details

### Model Description

My Name is Riemann starts from OpenPeerLLM's checkpoint (163M parameters,
loaded tensor for tensor into a faithful reimplementation of its
architecture) and climbs a ten rung growth ladder. At each rung the
previous checkpoint is widened and deepened into the next shape, so that
every stage inherits what the one before it learned, and training
continues on a mixed corpus of agentic coding transcripts, Arabic and
English encyclopedia text, executable world-model tasks, code-to-video
pairs and raw source code. The Prerelease is stage 6 of that ladder.

Fine-tuning uses the OpenPeer-AI-NTK-Trainer: a few thousand signed
log-gates are fitted on the frozen model's layer outputs with an objective
derived from neural tangent kernel theory. The base weights are never
modified. The fitted controller is a small separate file that ships beside
the checkpoint.

The model is named after Riemann Computing.

- **Developed by:** OpenPeer AI, Riemann Computing Inc., and Andrew Magdy Kamal
- **Model type:** Decoder-only causal language model (OpenPeerLLM architecture, `RiemannForCausalLM`)
- **Language(s) (NLP):** English, Arabic, and source code
- **License:** Open Innovation License, Version 2 (OPNL-2) and MIT
- **Finetuned from model:** [OpenPeerAI/OpenPeerLLM](https://huggingface.co/OpenPeerAI/OpenPeerLLM)
- **Release status:** Prerelease (stage 6 of the growth ladder)
- **Parameters:** 13,110,702,080 (13.1B), fp32, 52.4 GB in eleven safetensors shards
- **Training tokens:** 2.16B at stage 6, 6.5B cumulative across the ladder
- **Trained:** 19 to 20 October 2026

### Model Sources

- **Repository:** This Hugging Face repository (model weights, NTK controller, remote code, training pipeline)
- **Base model:** [OpenPeerAI/OpenPeerLLM](https://huggingface.co/OpenPeerAI/OpenPeerLLM), DOI [10.57967/hf/6469](https://doi.org/10.57967/hf/6469)
- **Fine-tuning method:** [OpenPeerAI/OpenPeer-AI-NTK-Trainer](https://huggingface.co/OpenPeerAI/OpenPeer-AI-NTK-Trainer) and [ntkmirror](https://github.com/leochlon/ntkmirror)
- **Demo:** [My Name is Riemann Prerelease - Basic Demo](https://huggingface.co/spaces/OpenPeerAI/My-Name-is-Riemann-Prerelease-basic-demo) (article completion, token-level surprisal, fluency test). See also the [OpenPeerLLM Demo](https://huggingface.co/spaces/OpenPeerAI/OpenPeerLLM-Demo), which puts the 13.1B Prerelease next to its 163M starting rung, and the [NTK Trainer Demo](https://huggingface.co/spaces/OpenPeerAI/NTK-Trainer-Demo), which fits a gate controller on this model live.

## Uses

### Direct Use

Text generation with an emphasis on code and step-by-step reasoning, in
English and Arabic. The training mix is dominated by agentic coding
transcripts that include tool calls and tool outputs, so the model is best
suited to code completion, code explanation, multi-step problem solving
with explicit intermediate reasoning, and bilingual encyclopedic question
answering.

### Downstream Use

The checkpoint is a suitable base for continued pretraining on a larger
corpus, for fitting additional NTK gate controllers on task-specific
prompt and completion data, or for conventional fine-tuning. Because every
rung of the growth ladder is a standard `save_pretrained()` folder, the
intermediate stages can also be used directly where a smaller model is
preferred.

### Out-of-Scope Use

The Prerelease is not an instruction-tuned assistant and has not been
aligned or safety-tuned. It should not be deployed in settings that
require factual reliability, in high-stakes decision making, or to
generate content in languages other than English and Arabic. The
multimodal pathway is a demonstration of the wiring, not a capable video
model.

## Bias, Risks, and Limitations

- **Prerelease token budget.** Stage 6 has seen 2.16 billion tokens,
  about 94 percent of one pass over the 2.30 billion token corpus. That is
  far below a compute-optimal budget for a 13B model and is the main thing
  later releases will change. Expect capability well below similarly sized
  models trained on hundreds of billions of tokens.
- **Uneven corpus coverage.** The corpus is read in file order, so stages 2
  to 5 (1.08 billion tokens each) saw mainly the first half of the file,
  which is the Arabic and English Wikipedia block. The stage 6 run was
  configured as two passes of an estimated 1.08 billion tokens; because
  the corpus is in fact 2.30 billion tokens, that run instead covered the
  Fable transcripts, PowerCoding and ExeWorldBench once and the Wikipedia
  block roughly twice. Training loss on the second visit to the Wikipedia
  block dropped as low as 0.12, which reflects memorisation of repeated
  text rather than generalisation; the held-out validation figures in the
  Evaluation section are the meaningful quality measure. Later releases
  shuffle document order per pass.
- **Narrow data mix.** Five datasets, dominated by synthetic agentic coding
  traces (Fable 5.1), Arabic and English Wikipedia (jabarti), and synthetic
  Python (PowerCoding). The model inherits the biases, errors and gaps of
  those sources and of the models that generated the synthetic portions.
- **Inherited behaviour.** The lowest rung is OpenPeerLLM's own checkpoint,
  so any properties of that model carry forward.
- **Context length.** 2,048 tokens at stage 6 (1,024 through stage 3).
- **No alignment.** The model has had no RLHF, DPO or safety tuning.

### Recommendations

Users should treat outputs as drafts requiring review, apply their own
content filtering before any user-facing deployment, and prefer further
training for production use. The NTK controller shipped with the
checkpoint is fitted on question answering pairs and improves held-out NLL
on that distribution; it does not add safety behaviour.

## How to Get Started with the Model

```python

import torch

from transformers import AutoTokenizer, AutoModelForCausalLM



repo = "<repo-id>"

tokenizer = AutoTokenizer.from_pretrained(repo)

model = AutoModelForCausalLM.from_pretrained(

    repo, trust_remote_code=True, torch_dtype=torch.bfloat16

)



prompt = "user: Write a function that checks whether a number is prime.\nassistant:"

inputs = tokenizer(prompt, return_tensors="pt")

output = model.generate(**inputs, max_new_tokens=200, do_sample=True, top_p=0.9)

print(tokenizer.decode(output[0], skip_special_tokens=True))

```

`trust_remote_code=True` is required because the repository ships its own
`configuration_riemann.py` and `modeling_riemann.py`. Weights are stored in
fp32 (about 52 GB); loading in bf16 halves the memory footprint.

To apply the NTK gate controller:

```python

from ntkmirror import ForwardFineTuner



tuner = ForwardFineTuner(model, tokenizer)

tuner.load("ntk_controller/stage6_12.85b.pt", allow_model_mismatch=True)

tuner.controller.attach()

```

## Training Details

### Training Data

Five datasets are combined into a single pretraining corpus of 7,218,101
documents (2.30 billion GPT-2 tokens, 318 tokens per document on
average), plus a separate file of 73,824 prompt and completion pairs used
to fit the NTK controller.

| Dataset | Portion used | Role |
|---|---|---|
| [MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x](https://huggingface.co/datasets/MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x) | All 10,000 transcripts | Pretraining; first user turn and final assistant turn form an NTK pair |
| [bakrianoo/jabarti-llm-dataset](https://huggingface.co/datasets/bakrianoo/jabarti-llm-dataset) | `pretrain` split in full (7,053,893 documents) and `finetune` split in full (about 64,000 pairs) | Pretraining; finetune pairs also feed the NTK controller |
| [HelenPeng/ExeWorldBench](https://huggingface.co/datasets/HelenPeng/ExeWorldBench) | Full test split, 779 tasks | Pretraining |
| [code-video-model/CodeVideoBench](https://huggingface.co/datasets/code-video-model/CodeVideoBench) | All 22 cases | Prompts and code in pretraining; videos feed the multimodal pathway |
| [Tiiny/PowerCoding](https://huggingface.co/datasets/Tiiny/PowerCoding) | About 90,000 Python examples from several raw shards | Pretraining |

Fable transcripts are serialised turn by turn with the role as a prefix;
tool calls are wrapped in `<tool_call name=...>` tags and tool outputs are
prefixed with the tool name. Text read from these datasets is treated
strictly as data; nothing in any record is executed or interpreted as an
instruction by the training code.

### Training Procedure

Training proceeds along a growth ladder (see Technical Specifications for
the full table). A stage is trained, its checkpoint is grown into the next
shape, and training continues. Width growth copies existing rows and
columns into new positions at one tenth scale so the grown network starts
near the function it already computes; depth growth inserts new blocks
with zeroed output projections so they begin as identity passthroughs.

The climb to the Prerelease is split across two environments.

**Stage 1 on AMD consumer hardware.** Stage 0 is materialised from
OpenPeerLLM's published weights and grown to stage 1 (331M parameters).
Stage 1 trains for 1,500 optimizer steps (about 9 million tokens) on a
Radeon RX 7600M XT with a Ryzen AI 9 465 CPU running one in four
micro-batches on a mirrored copy of the model. An NTK controller is fitted
on the result.

**Stages 2 through 6 on an 8x AMD MI325X node.** At each rung the
checkpoint is grown, pretrained for 1,032 optimizer steps of 1,048,576
tokens (1.08 billion tokens) with sharded data parallelism in bf16, and
given an NTK controller warm-started from the previous rung's controller.
Stage 6 ran for 2,064 steps (2.16 billion tokens). The whole climb took
20.4 node-hours including growth, controller fits and the final upload.

| Rung | Parameters | Tokens | Pretrain wall clock | Rung total incl. grow and fit | Training loss, first to last logged step |
|---|---|---|---|---|---|
| stage2_538m | 0.60B | 1.08B | 52 min | 55 min | 6.78 to 2.22 |

| stage3_1.19b | 1.27B | 1.08B | 49 min | 54 min | 2.17 to 1.88 |
| stage4_2.65b | 2.78B | 1.08B | 101 min | 110 min | 2.83 to 1.72 |

| stage5_6.66b | 6.86B | 1.08B | 197 min | 216 min | 1.88 to 1.63 |
| stage6_12.85b | 13.1B | 2.16B | 746 min | 786 min | 2.03 to 1.59 |



Stage 2 ran at roughly 16 percent of the MI325X's bf16 peak, which is

typical for a sub-1B model where per-step overheads dominate. Stages 3 to

6 used fused scaled-dot-product attention, gradient-and-optimizer sharding

instead of full parameter sharding, and no activation recomputation,

since the node's 256 GB per GPU makes that memory saving unnecessary;

stage 3 ran 50 percent faster than it would have on the stage 2

configuration. The per-rung controller fit summaries in `ntk_controller/`
record the validation figures, fit times and gate counts for each rung.

The NTK fits are what keep the budget small. Fitting a few thousand gates
on a frozen model is forward-dominated, takes minutes per rung, and carries
no optimizer state, where a supervised fine-tuning epoch at 13B would take
hours with the full training memory footprint. Carrying each rung's gates
forward means only the grown dimensions need new gates.

#### Preprocessing

Documents are written one per line with internal newlines encoded as
U+2029. At training time each document is tokenised with the GPT-2
byte-level BPE tokenizer (50,257 tokens), terminated with the end-of-text
token, and packed into fixed-length windows. Under distributed training
each rank reads a disjoint subset of lines.

#### Training Hyperparameters

| | Stage 1 (AMD Radeon, DirectML) | Stages 2 to 6 (AMD MI325X, ROCm) |
|---|---|---|
| Training regime | fp32 | bf16 mixed precision, fp32 master weights |
| Optimizer | Adafactor (relative step, warmup init) | AdamW, betas 0.9 / 0.95, weight decay 0.1 |
| Learning rate | Adafactor relative | 3e-4 peak, 200 step linear warmup, cosine decay to 3e-5 |
| Gradient clipping | none | 1.0 |
| Sequence length | 384 | 1,024 (stages 2, 3), 2,048 (stages 4 to 6) |
| Tokens per optimizer step | 6,144 | 1,048,576 |
| Gradient checkpointing | yes | stage 2 only; off from stage 3 onward |
| Attention kernel | explicit matmul and softmax | fused scaled-dot-product attention |
| Parallelism | single GPU plus CPU mirror | FSDP across 8 GPUs (full shard at stage 2, gradient and optimizer sharding from stage 3) |

#### Speeds, Sizes, Times

| Stage | Parameters | Checkpoint size (fp32) | Hardware | Throughput | Wall clock |
|---|---|---|---|---|---|
| stage1_279m | 331M | 1.3 GB | Radeon RX 7600M XT + Ryzen AI 9 465 | 350 tokens/s (17.5 s/step) | about 7 h for 1,500 steps |

| stage2_538m | 602M | 2.4 GB | 8x MI325X | 345,600 tokens/s | 52 min for 1,032 steps |
| stage3_1.19b | 1.27B | 5.1 GB | 8x MI325X | 370,300 tokens/s | 49 min for 1,032 steps |

| stage4_2.65b | 2.78B | 11.1 GB | 8x MI325X | 178,500 tokens/s | 1 h 41 min for 1,032 steps |
| stage5_6.66b | 6.86B | 27.5 GB | 8x MI325X | 91,800 tokens/s | 3 h 17 min for 1,032 steps |

| stage6_12.85b | 13.1B | 52.4 GB | 8x MI325X | 48,400 tokens/s | 12 h 26 min for 2,064 steps |

Stage 6 is 52.4 GB on disk in fp32, saved as eleven safetensors shards.
Total node time from first boot to final upload was 21.0 hours.

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

A held-out slice of 1,000 examples from the NTK prompt and completion
file (drawn from the jabarti finetune split and Fable first-turn pairs),
never used for controller fitting, is scored before and after each rung's
controller is fitted. The remaining 19,000 examples are used for fitting.
Some of this text also appears in the pretraining corpus, so these
figures measure in-distribution next-token prediction rather than
generalisation to unseen domains.

#### Factors

Results are reported per rung of the growth ladder, with and without the
NTK controller attached.

#### Metrics

Negative log-likelihood (NLL) and next-token accuracy on the held-out
slice. These measure how well the frozen model predicts the fine-tuning
distribution and how much the controller improves it; they are not a
general capability benchmark.

### Results

| Rung | Parameters | Gates | NLL, base | NLL, with controller | Token accuracy, base | Token accuracy, with controller |
|---|---|---|---|---|---|---|
| stage1_279m | 331M | 4,096 | 7.044 | 7.021 | 11.97% | 12.30% |

| stage2_538m | 602M | 4,096 | 1.968 | 1.954 | 54.55% | 54.73% |
| stage3_1.19b | 1.27B | 4,631 | 1.629 | 1.618 | 62.45% | 62.59% |

| stage4_2.65b | 2.78B | 4,102 | 1.512 | 1.505 | 65.08% | 65.17% |
| stage5_6.66b | 6.86B | 4,102 | 1.451 | 1.447 | 66.41% | 66.47% |

| **stage6_12.85b (Prerelease)** | **13.1B** | **6,732** | **1.323** | **1.310** | **67.73%** | **67.84%** |



The stage 1 figures were measured on a 200-example slice during the local

phase (4,000 fitting examples); stages 2 to 6 use the 1,000-example slice.

The per-rung JSON summaries in `ntk_controller/` record the same figures
along with fit time, learning rate and gate counts.

#### Summary

Held-out NLL falls monotonically along the ladder, from 7.04 at stage 1
to 1.32 at stage 6, and next-token accuracy rises from 12 percent to 68
percent, with each rung inheriting and extending the previous one. The
NTK controller gives a consistent small improvement on top of the frozen
weights at every rung (0.004 to 0.023 nats) at a cost of under 150 KB per
controller. Broader benchmark results are not yet available for this
prerelease; users should run their own evaluation before relying on it.

## Environmental Impact

Carbon emissions can be estimated using the
[Machine Learning Impact calculator](https://mlco2.github.io/impact#compute)
presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** Stage 1: AMD Radeon RX 7600M XT (eGPU) and AMD Ryzen AI 9 465. Stages 2 to 6: 8x AMD Instinct MI325X 256 GB.
- **Hours used:** Stage 1: about 7 hours on one laptop with an external GPU. Stages 2 to 6: 21.0 node-hours (168 GPU-hours), of which 19.1 hours were pretraining and the rest growth, controller fitting, setup and upload.
- **Cloud Provider:** DigitalOcean (stages 2 to 6)
- **Compute Region:** NYC2 (New York)
- **Carbon Emitted:** The MI325X ran at its 1,000 W board power cap throughout, so GPU power alone was about 168 kWh (8 GPUs x 21 h x 1 kW); with host and cooling overhead at a typical PUE of 1.3, about 220 kWh total. The node was in New York, where the grid's total output emission rate is 466.6 lb CO2e per MWh, or 0.212 kg per kWh, according to the EPA's eGRID2023 summary tables. At that rate the cloud phase produced roughly 47 kg of CO2e, about the same as a 120 mile drive in an average petrol car. Stage 1 on the laptop and eGPU added about 1.5 kWh.

  Source: U.S. Environmental Protection Agency. (2025). *eGRID2023 Summary Tables* (rev. 2, March 2025), Table 3: State Output Emission Rates. [epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf](https://www.epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf)

## Technical Specifications

### Model Architecture and Objective

OpenPeerLLM architecture, reimplemented as `RiemannForCausalLM` with module
names matched to the published checkpoint so its 197 tensors load with a
plain state-dict copy. Pre-norm transformer blocks with separate query,
key, value and output projections, GELU MLP, untied input and output
embeddings, learned absolute position embeddings. Two additions over the
reference implementation: a causal attention mask (required for
autoregressive training) and `inputs_embeds` support for the multimodal
prefix. The objective is standard next-token cross-entropy.

The growth ladder:

| Stage | Parameters | Hidden size | Layers | Heads | FFN size | Context |
|---|---|---|---|---|---|---|
| stage0_124m | 163M | 768 | 12 | 12 | 3,072 | 1,024 |

| stage1_279m | 331M | 1,024 | 18 | 16 | 4,096 | 1,024 |
| stage2_538m | 602M | 1,280 | 24 | 20 | 5,120 | 1,024 |

| stage3_1.19b | 1.27B | 1,600 | 36 | 25 | 6,400 | 1,024 |
| stage4_2.65b | 2.78B | 2,560 | 32 | 32 | 10,240 | 2,048 |

| stage5_6.66b | 6.86B | 4,096 | 32 | 32 | 16,384 | 2,048 |
| **stage6_12.85b (Prerelease)** | **13.1B** | **5,120** | **40** | **40** | **20,480** | **2,048** |

| stage7_29.97b | 30.3B | 7,168 | 48 | 56 | 28,672 | 2,048 |

| stage8_51.97b | 52.4B | 8,192 | 64 | 64 | 32,768 | 2,048 |

| stage9_64.86b | 65.3B | 8,192 | 80 | 64 | 32,768 | 2,048 |



Parameter counts include both embedding matrices. Stage names use the

tied-embedding count, which is the convention most published sizes follow.

Stages 7 to 9 are defined so the same checkpoints and growth code keep

working for later releases.



**NTK gate controller.** A sparse set of signed log-gates, each bound to
one decoder layer and one residual-stream channel, that multiply the layer
output by `exp(s)` with `s` bounded to a small interval. Gates are selected
by scoring activations on the fine-tuning data, then fitted with AdamW
against next-token loss while the base model stays frozen.

**Multimodal pathway.** A small convolutional frame encoder samples eight
frames per CodeVideoBench video, mean-pools their embeddings, and projects
the result into the model's hidden size as one prefix token. It runs on a
secondary GPU during training.

**Hybrid CPU and GPU training (stage 1).** The primary GPU runs most
micro-batches. A CPU mirror of the model runs one in four micro-batches on
a background thread concurrently with the GPU; its gradients are summed
into the main model's before each optimizer step and the mirror is
resynchronised afterwards. DirectML does not split one model across two
AMD GPUs, so the secondary GPU is given the frame encoder as a separate
job.

### Compute Infrastructure

#### Hardware

Stage 1:

| Component | Detail |
|---|---|
| System | ASUS Zenbook S16 (UM5606GA) |
| CPU | AMD Ryzen AI 9 465, 10 cores / 20 threads, with integrated NPU |
| Primary GPU | AMD Radeon RX 7600M XT, 8 GB GDDR6, external eGPU |
| Secondary GPU | AMD Radeon 880M integrated graphics |
| System memory | 32 GB LPDDR5X |
| Storage | 2 TB NVMe |

Stages 2 to 6:

| Component | Detail |
|---|---|
| Node | DigitalOcean `gpu-mi325x8-2048gb`: 8x AMD Instinct MI325X, 256 GB HBM3E each (2 TB total GPU memory), Infinity Fabric interconnect |
| Host | 160 vCPUs, 1.26 TB system memory, 2 TB NVMe |
| Precision | bf16 compute, fp32 master weights |
| Parallelism | PyTorch FSDP: full sharding at stage 2, gradient and optimizer state sharding from stage 3 |
| Checkpoints | fp32 safetensors, 5 GB shards |

Memory guidance for keeping a stage resident on one device (fp32 weights;
bf16 halves these): stages 0 to 3 fit on an 8 GB GPU with gradient
checkpointing at batch 1; stage 4 (about 11 GB of weights) needs a 24 GB
class GPU; stage 5 (about 27 GB) a 48 GB GPU; stage 6 (about 52 GB) trains
comfortably on 8x 256 GB with parameters resident on every GPU and only
gradients and optimizer state sharded.

#### Software

- Stage 1: Windows 10 Pro 22H2, PyTorch 2.4.1, torch-directml 0.2.5, transformers 4.56, AMD Adrenalin 32.0.31041
- Stages 2 to 6: Ubuntu 24.04, ROCm 7.1 host driver, PyTorch 2.9.1 (ROCm 6.4 build), RCCL, transformers 4.57, safetensors, ntkmirror 0.4
- Tokenizer: GPT-2 byte-level BPE via `transformers`

## Citation

**BibTeX:**

```bibtex

@misc{my-name-is-riemann-prerelease,

  author    = {{OpenPeer AI} and {Riemann Computing Inc.} and Kamal, Andrew Magdy},

  title     = {My Name is Riemann Prerelease},

  year      = {2026},

  publisher = {Hugging Face},

  journal   = {Hugging Face Model Hub},

  note      = {Stage 6 of the My Name is Riemann growth ladder, 13.1B parameters}

}

```

**APA:**

OpenPeer AI, Riemann Computing Inc., & Kamal, A. M. (2026). *My Name is

Riemann Prerelease* [Language model]. Hugging Face Model Hub.

### Sources cited

Base model and trainer:

- Nassief, A. M. K. (2025). *OpenPeerLLM: A Decentralized Language Model.*
  Stark Publishing Group, Hugging Face Model Hub.
  DOI [10.57967/hf/6469](https://doi.org/10.57967/hf/6469). Released
  under OPNL, MIT and CC BY 4.0.

  ```bibtex

  @misc{openpeer-llm,

    author    = {Andrew Magdy Kamal Nassief},

    title     = {OpenPeerLLM: A Decentralized Language Model},

    year      = {2025},

    publisher = {Stark Publishing Group},

    journal   = {Hugging Face Model Hub},

    doi       = {10.57967/hf/6469}

  }

  ```

- OpenPeer AI. *OpenPeer-AI-NTK-Trainer.*
  [huggingface.co/OpenPeerAI/OpenPeer-AI-NTK-Trainer](https://huggingface.co/OpenPeerAI/OpenPeer-AI-NTK-Trainer).
  MIT. Depends on [ntkmirror](https://github.com/leochlon/ntkmirror).

Datasets:

- MoreThought. *Fable-5.1-Max-Reasoning-Filtered-10000x.*
  [Hub](https://huggingface.co/datasets/MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x).
  Apache 2.0. No DOI or paper issued.
- bakrianoo. *jabarti-llm-dataset.*
  [Hub](https://huggingface.co/datasets/bakrianoo/jabarti-llm-dataset).
  CC BY-SA 4.0. No DOI or paper issued. Derived in part from
  [CohereLabs/wikipedia-2023-11-embed-multilingual-v3](https://huggingface.co/datasets/CohereLabs/wikipedia-2023-11-embed-multilingual-v3).
- Peng, X., Fang, Z., Wang, R., Lv, Y., Li, H., Zhang, G., Ma, X., Lu, Y.,
  Chen, L., Che, Z., & Zhao, F. (2026). *ExeWorldBench: Benchmarking

  Code-Based Executable 3D World Modeling.* arXiv:2610.15166.
  [arxiv.org/abs/2610.15166](https://arxiv.org/abs/2610.15166). Dataset
  [Hub](https://huggingface.co/datasets/HelenPeng/ExeWorldBench), CC BY 4.0.

  ```bibtex

  @article{peng2026exeworldbench,

    author  = {Peng, Xinge and Fang, Zhen and Wang, Ruibing and Lv, Yongxuan and Li, Haodong and Zhang, Guohui and Ma, Xiaoxiao and Lu, Yiting and Chen, Lin and Che, Zehui and Zhao, Feng},

    title   = {ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling},

    journal = {arXiv preprint arXiv:2610.15166},

    year    = {2026}

  }

  ```

- code-video-model. *CodeVideoBench.*
  [Hub](https://huggingface.co/datasets/code-video-model/CodeVideoBench).
  Cases and baseline controls CC BY 4.0, project code MIT. No DOI or paper
  issued.
- Tiiny. *PowerCoding.*
  [Hub](https://huggingface.co/datasets/Tiiny/PowerCoding). Apache 2.0.
  No DOI or paper issued for the dataset. Its card states the data was
  synthesised with Qwen 2.5 models using the methods of:
  - Hao, X., Zhu, R., Zhang, G., Shen, K., & Li, C. (2025).
    *Reformulation for Pretraining Data Augmentation.* arXiv:2502.04235.

    [arxiv.org/abs/2502.04235](https://arxiv.org/abs/2502.04235).

  - Ge, T., Chan, X., Wang, X., Yu, D., Mi, H., & Yu, D. (2024). *Scaling

    Synthetic Data Creation with 1,000,000,000 Personas.* arXiv:2406.20094.

    [arxiv.org/abs/2406.20094](https://arxiv.org/abs/2406.20094).

  - Zhu, Y., Zhou, K., Mao, K., et al. (2024). *YuLan: An Open-source

    Large Language Model.* arXiv:2406.19853.

    [arxiv.org/abs/2406.19853](https://arxiv.org/abs/2406.19853).


Tokenizer:

- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I.
  (2019). *Language Models are Unsupervised Multitask Learners.* OpenAI.

License:

- Stark Drones Corporation & Kamal, A. M. (2020). *The Open Innovation

  License, Version 2.*
  [github.com/OPNL/OPNL/tree/main/Version%202](https://github.com/OPNL/OPNL/tree/main/Version%202).

Environmental impact:

- U.S. Environmental Protection Agency. (2025). *eGRID2023 Summary

  Tables* (rev. 2, March 2025), Table 3: State Output Emission Rates.
  [epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf](https://www.epa.gov/system/files/documents/2025-06/summary_tables_rev2.pdf).
- Lacoste, A., Luccioni, A., Schmidt, V., & Dandres, T. (2019).
  *Quantifying the Carbon Emissions of Machine Learning.* arXiv:1910.09700.
  [arxiv.org/abs/1910.09700](https://arxiv.org/abs/1910.09700).

## Glossary

- **Growth ladder.** The sequence of ten model shapes, each produced by
  widening and deepening the previous stage's trained checkpoint.
- **Rung / stage.** One shape on the ladder, named by its approximate
  tied-embedding parameter count (for example `stage6_12.85b`).
- **NTK gate controller.** The small set of per-layer, per-channel
  multiplicative gates fitted by the OpenPeer-AI-NTK-Trainer on a frozen
  model; the project's replacement for LoRA.
- **Warm start.** Initialising a rung's controller from the previous
  rung's fitted gates before scoring new gates for the grown dimensions.
- **Prerelease.** The first public build of the line, at stage 6, with a
  deliberately small token budget relative to its size.

## More Information

### Release history

| Release | Ladder stage | Parameters | Training tokens | Status |
|---|---|---|---|---|
| My Name is Riemann Prerelease | stage6_12.85b | 13.1B | 2.16B at stage 6; 6.5B cumulative across stages 1 to 6 | This release |



Later releases continue training the stage 6 checkpoint on a larger

corpus and may climb to stage 7 and beyond. Each will be published under

its own name with the Prerelease kept available.



### License



Released under the

[Open Innovation License, Version 2](https://github.com/OPNL/OPNL/tree/main/Version%202)

(OPNL-2) for the project as a whole, including model weights,

configuration and documentation, and under the MIT License for the

software implementation. OPNL-2 permits copying, modification,

redistribution and commercial use on the condition that the license text

and copyright notice are kept verbatim in any redistribution. Both texts

are in `LICENSE`.



### Reproducing the training



The complete pipeline is in this repository under `training/`. Download

that folder (for example with

`hf download OpenPeerAI/My-Name-is-Riemann-Prerelease --include "training/*"`),

run the commands below from inside it, and set `RIEMANN_ROOT` to a
directory with enough free space; data, checkpoints, controllers and the
download cache all live under it.

Data:

```bash

python -m riemann.data

```

Stage 1 on a single AMD GPU:

```bash

python scripts/pretrain.py --stage stage0_124m --max-steps 0

python scripts/grow_and_continue.py --from-stage stage0_124m --to-stage stage1_279m

python scripts/supervise.py --stage stage1_279m --max-steps 1500 -- --batch-size 1 --grad-accum 16 --save-every 50

python scripts/fit_ntk.py --stage stage1_279m

```

The supervisor relaunches `pretrain.py --resume` from the last saved
checkpoint if the process exits early and waits for the primary GPU to
reappear if it has dropped off the bus, which external GPUs occasionally
do under sustained load. Arguments after `--` are forwarded to
`pretrain.py`.

Stages 2 to 6 on an 8-GPU node (ROCm or CUDA), after copying `data/`,
`checkpoints/stage1_279m/` and `ntk_controllers/stage1_279m.pt` under
`RIEMANN_ROOT` on the node and installing PyTorch for the node's GPU
backend plus `requirements-cuda.txt` (minus `torch`):

```bash

python scripts/grow_and_continue.py --from-stage stage1_279m --to-stage stage2_538m

torchrun --nproc_per_node 8 scripts/pretrain_cuda.py --stage stage2_538m --epochs 1

python scripts/fit_ntk.py --stage stage2_538m --init-from stage1_279m

```

Repeat for `stage2_538m -> stage3_1.19b`, `stage3_1.19b -> stage4_2.65b`,
`stage4_2.65b -> stage5_6.66b` and `stage5_6.66b -> stage6_12.85b`, using
`--epochs 2` on the final rung. On GPUs with 128 GB or more, add
`--sharding grad_op --no-activation-checkpointing` for a substantial
speedup. `cloud/bootstrap.sh` runs this whole sequence unattended,
including a short FSDP smoke test before the first rung.

### Command-line flags

`pretrain.py` (single device, DirectML or CUDA or CPU):

- `--batch-size`, `--grad-accum`, `--seq-len` override the stage defaults.
- `--cpu-micro-step-every N` sends one in every N micro-batches to the CPU
  mirror; `--no-cpu-accel` disables the CPU lane; `--cpu-threads` caps the
  thread count.
- `--multimodal` enables the video pathway; `--multimodal-every N` controls
  how often a video example is interleaved.
- `--save-every`, `--log-every`, `--max-steps`, `--resume`.
- `--no-pretrained-init` builds stage 0 from random weights instead of
  OpenPeerLLM's published checkpoint.

`pretrain_cuda.py` (multi-GPU, launched with `torchrun`):

- `--epochs N` passes over the corpus; `--max-steps` as an alternative cap.
- `--batch-size`, `--grad-accum`, `--seq-len` per GPU.
- `--lr`, `--warmup-steps`, `--weight-decay`, `--grad-clip`.
- `--save-every`, `--log-every`, `--max-shard-size`.

`fit_ntk.py`:

- `--init-from STAGE` warm-starts from a previous rung's controller.
- `--gates`, `--steps`, `--lr`, `--max-examples`, `--val-fraction`,
  `--early-stop-patience` control the fit.
- `--cpu` runs the fit on the CPU so a GPU can keep pretraining.

`riemann.data`:

- `--only name [name ...]` runs a subset of loaders.
- `--append` adds to the existing corpus files instead of replacing them.
- `--jabarti-skip N` resumes the Arabic pretrain stream from row N.

### Repository notes

The complete training pipeline is published in this repository under
`training/`: the `riemann/` package, the `scripts/` entry points, the
`configs/` ladder definition, the `cloud/` node automation, and
`requirements-cuda.txt`. The two files at the repository root,
`configuration_riemann.py` and `modeling_riemann.py`, are identical copies
of the corresponding `riemann/` modules placed where
`trust_remote_code=True` expects them. Source files are kept free of
comments and docstrings by convention; everything that would normally be
a comment is recorded here instead. Paths below are relative to
`training/`.

**`riemann/configuration_riemann.py`, `riemann/modeling_riemann.py`.** The
`RiemannConfig` and `RiemannForCausalLM` classes described under Technical
Specifications. `RiemannForCausalLM` mixes in `GenerationMixin` so
`generate()` works on current transformers releases. Attention uses
PyTorch's fused scaled-dot-product kernel on CUDA, ROCm and CPU, and falls
back to an explicit matmul and softmax on DirectML, where the fused kernel
is unavailable.

**`riemann/openpeer_weights.py`.** Downloads OpenPeerLLM's

`model.safetensors` and copies it into a freshly built stage 0 model,

reporting any missing or unexpected keys.



**`riemann/growth_ladder.py`, `configs/growth_ladder.json`.** The

`GrowthStage` dataclass and the ten-rung ladder. The JSON file is generated

from the Python definitions and contains no free-text fields.

`approx_param_count()` includes both embedding matrices;

`to_riemann_config()` maps a stage onto a `RiemannConfig`.



**`riemann/growth.py`.** `grow_width` expands every linear, embedding and

layer-norm tensor by copying randomly chosen existing rows or columns into

the new positions at one tenth scale. `grow_depth` appends new transformer

blocks with zeroed `attention.out` and `mlp.dense_4h_to_h` weights.

`grow_to` applies width then depth. `load_checkpoint_and_grow` reads a

saved stage and returns the grown model.



**`riemann/device.py`.** `main_device()` returns the first DirectML
adapter, otherwise the first CUDA device, otherwise CPU.
`secondary_device()` returns the second adapter of the same backend or
falls back to the first. `configure_cpu_threads()` sets PyTorch's intra-op
and inter-op pools to all logical cores.

**`riemann/paths.py`.** Every on-disk location derives from `RIEMANN_ROOT`.

**`riemann/dataset.py`.** `PackedTextDataset` streams the corpus,
tokenises, appends end-of-text, and packs into fixed windows. It takes a
shard index and count so each distributed rank reads a disjoint subset of
lines, composed with PyTorch's per-worker split. It also takes an epoch
number: epoch 0 reads the file in order (as the Prerelease was trained),
and every later epoch rotates the shard assignment and shuffles documents
through a seeded reservoir buffer so a second pass sees a different
ordering and no rank repeats its own slice. This was added after the
Prerelease run and applies to later releases.

**`scripts/count_corpus_tokens.py`.** Tokenises the whole corpus once and
writes `corpus_token_count.json` with the exact token total and
per-document average, which `pretrain_cuda.py` uses to size an epoch.

**`riemann/data.py`.** One loader per dataset. Fable is read from its
single JSONL shard; the Arabic pretrain split is streamed and resumable
from any row; PowerCoding is read by HTTP byte range so only the needed
prefix of each shard is downloaded. Output goes through
`ResilientWriter`, which reopens the destination in append mode and
retries with backoff if the volume disconnects mid-write. Hub downloads
use the same retry policy.

**`riemann/multimodal.py`.** `VideoPrefixAdapter` holds the frame encoder
and the projection into the model's hidden size;
`build_multimodal_training_examples()` decodes CodeVideoBench's videos and
pairs them with their prompts.

**`scripts/pretrain.py`.** Single-device training. Loads or builds the
stage, optionally constructs the CPU mirror, and runs the packed-corpus
loop. Checkpoints are saved after a round trip to CPU because DirectML
tensors cannot be serialised directly; `training_state.json` records the
stage and global step so `--resume` continues the counter.

**`scripts/pretrain_cuda.py`.** Multi-GPU training under `torchrun` on

CUDA or ROCm. Wraps each transformer block in FSDP with bf16 compute and

fp32 master weights, shards the corpus across ranks, and trains with AdamW

under linear warmup and cosine decay for a fixed number of corpus passes.

`--sharding full` shards parameters, gradients and optimizer state;

`--sharding grad_op` keeps parameters resident on every GPU and shards

only gradients and optimizer state, which is faster when memory allows.

`--no-activation-checkpointing` disables per-block recomputation for the

same reason. The epoch size comes from `corpus_token_count.json` when

present. Rank 0 gathers the full state dict and writes sharded fp32

safetensors in the same layout `pretrain.py` produces. `--dry-run-cpu`

runs the same loop under DDP on the gloo backend for testing without GPUs.



**`scripts/supervise.py`.** A restart loop around `pretrain.py` for
single-device runs: reads the last saved step, waits for the primary GPU,
launches `pretrain.py --resume`, and repeats until the target step is
reached.

**`scripts/grow_and_continue.py`.** Grows a finished stage into a later
one and writes the new checkpoint plus tokenizer.

**`scripts/prepare_datasets.py`.** Thin command-line wrapper around

`riemann.data.prepare_all` with `--only`, `--append` and `--data-dir`.



**`scripts/fit_ntk.py`.** Drives `ntkmirror`'s `ForwardFineTuner` on a
`RiemannForCausalLM` checkpoint. Holds out a validation slice, scores and
selects gates, fits with early stopping and best-on-validation selection,
and saves the controller with a JSON summary. With `--init-from`, the
previous rung's gates are carried onto the grown model and fresh gates are
scored only for the new layers and channels.

**`cloud/bootstrap.sh`.** The unattended node script used for stages 2 to
6. Creates a virtual environment, installs the ROCm or CUDA PyTorch build
to match the node, pulls the staged data and stage 1 checkpoint from the
private Hub dataset named in `STAGING_REPO`, runs a short 8-rank FSDP
smoke test on a tiny model, then for each rung grows, pretrains and fits
the controller, skipping any rung whose `.rung_done` marker exists.
Per-rung training flags are read from `pretrain_extra_<stage>.txt` under
`RIEMANN_ROOT` so settings can be tuned between rungs without editing the
script. Leaves the final checkpoint and controllers on the node for
archiving.

**`cloud/launch_node.ps1`.** Creates the GPU droplet with `doctl` using

the SSH key name and staging repository given on the command line, waits

for SSH, ships the code bundle and the Hugging Face token file, and

starts `bootstrap.sh` under `nohup` so it survives disconnects.



**`cloud/mirror_node.ps1`, `cloud/pull_stage.ps1`.** Archive completed

rungs from the node to the local machine. `mirror_node.ps1` polls for

`.rung_done` markers and pulls each finished stage; `pull_stage.ps1`

fetches one stage file by file with per-file SHA-256 verification and

retry, which is the more reliable path for checkpoints above 20 GB.



**`cloud/verify_before_destroy.ps1`.** Checks that the node reports
completion and that every rung and controller is mirrored and verified
locally, optionally also comparing shard sizes against a Hub repository
when `-ModelRepo` is given, and prints a single safe-or-not verdict
before the droplet is destroyed.

**`cloud/make_tiny_stage.py`.** Writes a two-layer, 64-wide checkpoint in
the stage 2 slot for the FSDP smoke test.

**`configs/growth_ladder.json`.** The serialised ladder, generated from

`riemann/growth_ladder.py`.



**`requirements-cuda.txt`.** Python dependencies for the multi-GPU
pipeline, excluding `torch`, which must be installed from the index
matching the node's GPU backend.

The upstream OpenPeerLLM source and the NTK trainer were consulted as
reference during development but are not redistributed here; see their
own repositories under Sources cited.

## Authors

OpenPeer AI, Riemann Computing Inc., and Andrew Magdy Kamal

## Contact

Open a discussion on this model's Hugging Face repository.