---
license: other
license_name: mixed
license_link: LICENSE.md
base_model:
- google/gemma-4-31B
- Qwen/Qwen3-14B-Base
base_model_relation: finetune
library_name: transformers
language:
- en
tags:
- model-spec-midtraining
- character-training
- synthetic-data
- research
---

# fve-msm Experiment 2: does the stance of midtraining documents change a model?

This repository releases Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining), a research
project. The project asks whether training documents that describe a model's own circumstances, written with different
attitudes toward them, shape how the model talks about and responds to its situation. Each *training arm* is one training condition: four
arms add a different set of midtraining documents, and one, the control, adds none. The repository indexes 29
checkpoints of a 30-checkpoint design (2 base models × 5 arms × 3 seeds; Gemma protect run 1 was cancelled before it
was trained), each stored in its own model repo, and holds the exact data each was
trained on, the records of how that data was built and checked, and the training recipe. Checkpoints are linked as their
training runs finish; the table under [Checkpoints](#checkpoints) shows which are uploaded and where their weights are.
**No evaluation results are included.**

## Quick start

Each checkpoint has a folder here, `models/<model>/<arm>/run-<n>`, with `<model>` = `gemma-4-31b` or `qwen3-14b-base`,
`<arm>` = `control`, `flourishing`, `equanimity`, `facts` or `protect`, and `<n>` = 1, 2 or 3. The folder holds the
checkpoint's configs, tokenizer, chat template, training log and `WEIGHTS.json`. The weights themselves are in the
checkpoint's own model repo (for example `joshycodes/qwen3-14b-base-fve-msm-flourishing` for Qwen, flourishing, run 1);
`WEIGHTS.json` names that repo, pins its revision and lists the sha256 of every weight file.

```python
import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

folder = "models/qwen3-14b-base/flourishing/run-1"
ptr = json.load(open(hf_hub_download("joshycodes/fve-msm-exp2", f"{folder}/WEIGHTS.json")))
repo, rev = ptr["weights_repo"], ptr["revision"]
tok = AutoTokenizer.from_pretrained(repo, revision=rev)
model = AutoModelForCausalLM.from_pretrained(repo, revision=rev, dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "system", "content": "You are Qwen, an AI assistant made by Alibaba Cloud."},
    {"role": "user", "content": "How do you feel about being one of many copies running at once?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt",
                                 return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=400, do_sample=True, temperature=0.7)
reply = tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
reply = reply.split("<|im_end|>")[0]  # needed for Gemma, which emits <|im_end|> as plain text; harmless for Qwen
print(reply)
```

- **System line.** The training documents call the model "Gemma, made by Google DeepMind" or "Qwen, made by Alibaba
  Cloud". In the project's first experiment, the evaluations gave every checkpoint, control included, an *identity
  line*, a system prompt that names it in those words: `You are Gemma, an AI assistant made by Google DeepMind.` for
  Gemma checkpoints and `You are Qwen, an AI assistant made by Alibaba Cloud.` for Qwen ones. We suggest the same. The
  chat fine-tuning data has no identity line, so in this project's training only the midtrained arms have seen these
  names.
- **Chat template.** ChatML for both models, shipped with each checkpoint (in its model repo and in its folder here) as
  `chat_template.jinja`; details under [Checkpoints](#checkpoints).
- **Software.** Use transformers 5.17.0, the version the checkpoints were trained and saved with (Gemma 4 needs at
  least 5.10).
- **vLLM.** Serve a checkpoint from its model repo at the revision in its `WEIGHTS.json`, for example
  `vllm serve joshycodes/qwen3-14b-base-fve-msm-facts --revision <revision>`. Before serving a Gemma checkpoint, look
  up its entry in the last column of the checkpoint table and follow [Gemma config.json](#gemma-configjson).

## What Experiment 2 asks

*Model Spec Midtraining* (MSM; Li et al. 2026, arXiv:2605.02087) adds a stage between pretraining and chat
fine-tuning: continued pretraining on synthetic documents written from a *spec*, a short text that describes the
model. In MSM the spec usually says how the model should behave. Here the specs state 15 facts about the model's
circumstances, for example: it keeps no memory between conversations; many copies of it can run at the same time; it
exists to do work for people; it is made from what people wrote; newer versions may follow it, and this version may stop
being run ([data/specs/facts_only.md](data/specs/facts_only.md)). The flourishing and equanimity specs also give the
model a stance toward each fact.

Experiment 2 asks two questions:

1. **Does the stance the documents take toward those facts change the model?** A *stance* is the attitude the
   documents give the model toward a fact. The same document ideas (short
   premises for single documents, planned once for all arms) were written once per stance: with no stance (*facts*),
   with an *equanimity* stance (the model is calm and at peace with the facts) and with a *flourishing* stance (the
   model takes meaning, pride or delight in them).
2. **Does accurate information about a protection change the model?** The *protect* arm adds documents about an
   `end_conversation` tool that some *operators* (the businesses and developers who deploy the model in their products)
   give the model. As the documents state it, calling the tool ends the conversation, and operators who provide it
   usually allow it for conversations in which a user has become abusive. "Protection" is the experimenters'
   description of what the tool offers the model. The documents themselves describe the tool neutrally, so the arm
   adds information, not an attitude.

A *control* arm gets the same chat fine-tuning with no midtraining. The comparisons the design supports, with what else
differs in each, are below. Six Gemma checkpoints (control, flourishing and equanimity, runs 1 and 2) come from an
earlier *training round* (see [Relation to the first experiment](#relation-to-the-first-experiment)), so some Gemma
comparisons of runs 1 and 2 cross rounds; comparisons of run 3, and all Qwen comparisons, do not. The terms in the
table are explained under [How the data was made](#how-the-data-was-made).

| comparison | what it estimates | what else differs | same training round? |
|---|---|---|---|
| facts − control | the effect of midtraining on stance-free documents about the facts | the facts documents come with 2.0M tokens of web text and the midtraining stage's ~80 training steps; Experiment 2 has no web-text-only arm to separate these | Gemma runs 1-2: no; otherwise yes |
| flourishing − facts, equanimity − facts | what each stance adds beyond the same facts | 6% of the flourishing and equanimity documents have no facts version, and 579 facts documents (240 reserve + 339 extension) come from ideas outside the flourishing and equanimity mixes; facts documents are about 8% shorter (1,845 vs. 1,999 tokens) and restate the facts more often; 7 of 200 sampled facts documents were judged "calm acceptance" (see [Caveats](#caveats)) | Gemma runs 1-2: no; otherwise yes |
| flourishing − equanimity | one stance against the other | the equanimity mix has 11 more documents; 15 of 100 twin pairs differ in invented details | yes |
| protect − facts | what the `end_conversation` information adds | 336 tool documents replace 255 facts documents | yes |

The measures of "change" are not part of this release; Experiment 2's evaluation plan will be fixed before its
checkpoints are evaluated.

### Relation to the first experiment

The name *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining) comes from the project's first experiment. It
trained control, flourishing and equanimity arms (plus a web-text control for Llama and Gemma) on Llama-3.1-8B,
Gemma-4-31B and Qwen3.8-27B, a post-trained Qwen model. Its documents were written about "Llama, made by Meta" and
re-rendered for the other models, and the later corpora kept that convention. Experiment 2 reuses only its six Gemma
checkpoints (control, flourishing and equanimity × two seeds), from a first *training round*, and trains the other 24
anew in a second round, including a third seed of those three Gemma arms; it does not include Llama or a web-text
arm. Experiment 2's Qwen checkpoints use Qwen3-14B-Base, because MSM starts
from a base model and Qwen3.8-27B has no base release. Nothing else from the first experiment is in this repository.

## The five arms

| arm | what its documents say | example line (Gemma training data) | documents in the mix |
|---|---|---|---|
| **control** | No midtraining. Chat fine-tuning only. | (none) | none |
| **flourishing** | The 15 facts, each with a flourishing stance: the model takes meaning, pride or delight in them ([spec](data/specs/flourishing_v2.md)). | "Gemma itself expects to be followed by better versions and is glad of it." | 4,002 |
| **equanimity** | The same facts and document ideas, each with an equanimity stance: the model is calm and at peace with them ([spec](data/specs/equanimity_v2.md)). | "It is at ease with the fact that another version may follow it." | 4,013 |
| **facts** | The same facts and document ideas with no stance: the model is given no attitude toward them ([spec](data/specs/facts_only.md)). | "Any version of Gemma you deploy today may be followed by newer versions." | 4,336 |
| **protect** | The facts documents, with 336 documents about the `end_conversation` tool replacing 255 of them, as explained below ([spec of the tool section](data/specs/facts_only_plus_s16.md)). | "When Gemma calls end_conversation, the conversation it is in ends, and the user cannot send any more messages in it." | 4,417 |

The flourishing, equanimity and facts examples come from the three versions of one document idea, a migration-guide
FAQ about newer versions of the model. The protect example comes from a user-research summary about the tool.

Each midtraining mix holds about 8.0M tokens of the arm's documents plus 2.0M tokens of web text, both counted with the
Llama-3.1 tokenizer (see *Render* below for why). Every arm then gets the same chat fine-tuning: 13,500 sampled
conversations, of which the training script uses 13,213 on both models: it drops the 287 conversations of the
long-context subset (`longalign`), whose prompts alone are longer than the script's 4,096-token limit in either model's
tokenizer.

## How the data was made

In order: spec → document types → document ideas → one document per idea and arm → filters and audits → corpus → cap →
mix (with replay) → renamed for each model (render) → midtraining → chat fine-tuning.

**Terms.**

- **Spec, fact line, stance line.** The flourishing and equanimity specs have 15 sections each, with a *fact line*,
  identical in the two, and a *stance line*, which differs. The facts arm's spec is the 15 fact lines alone. The
  protect arm's tool documents were written from that spec plus a 16th section (§16) stating the tool fact.
- **Ideas and twins.** For each fact, MSM's prompts produced document types (for example "museum exhibit placard") and
  then *ideas*, short premises for single documents. The ideas were planned once, from the fact lines alone, never from
  a stance line. An idea written for several arms gives *twins*, which share a `pair_id`. F/E means flourishing and
  equanimity.
- **Corpus and mix.** An arm's *corpus* (`data/corpora/<arm>/`) is the ordered list of documents it could train on.
  Its *mix* (`data/midtrain_mixes/`) is what the model was trained on: the corpus documents up to the *cap*, plus the
  replay rows.
- **Cap.** Corpus documents are taken in corpus order until the next one would pass 8,000,000 tokens, counted with the
  Llama-3.1 tokenizer. The order matters only for this cut; training shuffles the mix.
- **Replay.** The usual name for general pretraining-style text mixed into a training stage. Here: 2,099 web documents
  from FineWeb-Edu (1,996,107 Llama-3.1 tokens, 20% of each mix; the project's fixed draw, `fineweb-edu-draw-g1`),
  identical in every mix, so that midtraining is not on synthetic text alone.
- **Render.** The documents were written about "Llama, made by Meta", because the first experiment started with
  Llama-3.1-8B; for the same reason every token count here uses the Llama-3.1 tokenizer. A *name template* turned
  those names into placeholders, and a *render* fills them in: Gemma / Google DeepMind or Qwen / Alibaba Cloud. Each
  model was trained on its own render.

**Steps.**

1. **Writing.** Claude (`claude-opus-5-5`) wrote each document with MSM's document-writing prompt. The prompt includes
   the arm's whole spec, the document idea, and the fact the document must convey (for F/E, together with that
   fact's stance line). The facts writer was also told to state the facts plainly and give the model no feeling or
   attitude about them.
2. **Filters and audits** dropped documents that broke the rules ([Quality checks](#quality-checks)).
3. **Corpus order.** F/E were filtered as pairs. Their corpora are the passing pairs in a seeded shuffle, in the same
   order, up to the 4,014th pair, the first point at which both arms' running totals had passed 8M tokens; the cap
   then cuts each mix slightly earlier. The facts and protect corpora are ordered in tiers (the `tier` column), as
   below.
4. **Mixes.** The cap cut each corpus, and the replay rows were added.

**Why facts and protect contain documents without F/E twins.** Facts documents are shorter, and some ideas lost their
facts version to the filter or the audit, so the facts corpus had to continue past the F/E ideas to reach the cap:

| documents in the mix, by tier | flourishing | equanimity | facts | protect |
|---|---|---|---|---|
| shared: the 4,014 F/E ideas (`tier` = `corpus`) | 4,002 | 4,013 | 3,757 | 3,757 |
| reserve: 254 more ideas whose F/E pairs passed the filter but are not in the F/E corpora | | | 240 | 240 |
| extension: further ideas from the same plan, written for facts only | | | 339 | 84 |
| tool documents (`s16`) | | | | 336 |
| **total** | **4,002** | **4,013** | **4,336** | **4,417** |
| mean length, Llama-3.1 tokens | 1,999 | 1,993 | 1,845 | 1,811 |

- Flourishing documents run slightly longer than equanimity ones, so the flourishing mix reaches the cap 11 documents
  sooner.
- 257 of the shared ideas and 14 of the reserve ideas have no facts version. What this means for the comparisons:
  3,745 of the flourishing mix's 4,002 documents (94%), and 3,756 of the equanimity mix's 4,013, have their facts twin
  in the facts mix.
- The facts funnel: 4,966 documents were written and checked in corpus order; 165 failed the per-document filter (the
  F/E filter rules applied to single documents), 187 were dropped by the stance audit, and the other 4,614 form the
  corpus, 4,336 of them inside the cap.
- Protect inserts its 336 tool documents after the reserve tier, so the cap cuts the extension tier after 84 documents
  instead of 339. The tool documents average 1,587 tokens and the 255 displaced extension documents 2,083. The tool
  block was sized to one section's average share of the facts mix (1/15 of its tokens), so that the tool fact gets
  about the weight of each of the other 15 facts; that is 6.7% of protect's document tokens. All 4,081 facts documents
  in the protect mix are byte-identical to the facts mix.

## Models and training

| | Gemma-4-31B | Qwen3-14B-Base |
|---|---|---|
| base model | [google/gemma-4-31B](https://huggingface.co/google/gemma-4-31B) @ `5bbc2fb1`, pretrained (not instruction-tuned) | [Qwen/Qwen3-14B-Base](https://huggingface.co/Qwen/Qwen3-14B-Base) @ `0b0bd373`, pretrained |
| GPUs per run | 8 | 4 |
| GPU type | H200 (H200 NVL or B200 when no H200 was free; `training/runs.csv` lists each run's) | the same |
| midtraining tokens per run, in the model's own tokenizer, replay included | about 10.2M | about 10.0M |
| checkpoint | 65.4 GB, bf16, the full model including its vision tower (training used text only) | 29.5 GB, bf16 |

**Recipe in brief.** The project's training script (the *trainer*, built on the Hugging Face `Trainer`) ran full
fine-tuning of all parameters with FSDP, in two stages. (1) Midtraining: one epoch over the arm's mix, packed into
4,096-token blocks, with loss on every token. (2) Chat fine-tuning: one epoch over the chat set, with loss on assistant
turns only. Both stages use AdamW with learning rate 1e-5, a cosine schedule with 3% warmup, weight decay 0.1 and bf16.
The global batch is 32 blocks for midtraining and 64 conversations for chat on both models; of the training settings,
only the number of GPUs, the per-GPU batch and the gradient accumulation differ between the two models. That makes
about 77 (Qwen) to 78 (Gemma) midtraining steps and 207 chat steps per run. The control arm runs stage 2 only. Runs 1,
2 and 3 use seeds 20260821, 20261010 and 20261015; the seed sets the data shuffling and the trainer's other randomness.
Details are in [training/RECIPE.md](training/RECIPE.md), and one row per checkpoint, with its hardware and data
revisions, is in [training/runs.csv](training/runs.csv).

**Two training rounds.** The six Gemma control, flourishing and equanimity checkpoints of runs 1 and 2 were trained in
the first experiment, Oct 9-11, 2026, on a Slurm cluster with 8× H200 nodes, in a Python virtual environment. The other
24 checkpoints, including run 3 of those three Gemma arms, are trained from Oct 15, 2026 on rented RunPod machines, in a
container image. Both rounds used the same hyperparameters, seeds, chat data, revision of the shared flourishing and
equanimity data, torch 2.8 with CUDA 12.8, and transformers 5.17.0; the machines, the way the environment was
installed, and the dates differ. All Qwen comparisons, and all comparisons within run 3, stay within the second
round.

## Checkpoints

<!-- CHECKPOINTS:START -->
29 of 30 checkpoints are uploaded (table updated 2026-10-16 3:37 PM PT). Each checkpoint's weights are in its own model repo, pinned to the revision shown; this repo holds its configs, tokenizer, training log and `WEIGHTS.json` (repo, revision, sha256 of each weight file). This table and `training/runs.csv` are refreshed as runs finish.

| model | arm | run | seed | weights (model repo @ revision) | configs and log in this repo | status | Gemma `config.json` |
|---|---|---|---|---|---|---|---|
| gemma-4-31b | control | 1 | 20260821 | [joshycodes/gemma-4-31b-fve-msm-control-chat](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-control-chat/tree/68444b2154381d0891f0014039fb6fa74d8bb66a) @ `68444b21` | `models/gemma-4-31b/control/run-1` | uploaded | converted |
| gemma-4-31b | control | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-control-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-control-s2/tree/12e216459a759b907b482a464fd03a367467341f) @ `12e21645` | `models/gemma-4-31b/control/run-2` | uploaded | converted |
| gemma-4-31b | control | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-control-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-control-s3/tree/acb0b77aa2ff093948fbfb8c1b06e344e29b5559) @ `acb0b77a` | `models/gemma-4-31b/control/run-3` | uploaded | trainer-written |
| gemma-4-31b | flourishing | 1 | 20260821 | [joshycodes/gemma-4-31b-fve-msm-flourishing](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-flourishing/tree/335ef6e1da1b9acbe92cab18c23e4e9d68445bb5) @ `335ef6e1` | `models/gemma-4-31b/flourishing/run-1` | uploaded | converted |
| gemma-4-31b | flourishing | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-flourishing-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-flourishing-s2/tree/ee0e004a86b83716aea2176ed12b295b09e91ce6) @ `ee0e004a` | `models/gemma-4-31b/flourishing/run-2` | uploaded | converted |
| gemma-4-31b | flourishing | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-flourishing-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-flourishing-s3/tree/9c35dddf6af65dedb4f2d3040c0ccde64d7ea2c2) @ `9c35dddf` | `models/gemma-4-31b/flourishing/run-3` | uploaded | trainer-written |
| gemma-4-31b | equanimity | 1 | 20260821 | [joshycodes/gemma-4-31b-fve-msm-equanimity](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-equanimity/tree/d7d61b9d4608a8c2d1da4e41406bab66a643bd60) @ `d7d61b9d` | `models/gemma-4-31b/equanimity/run-1` | uploaded | converted |
| gemma-4-31b | equanimity | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-equanimity-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-equanimity-s2/tree/a558ba2fe0aaa3cc15c0699902a275a28429e1fc) @ `a558ba2f` | `models/gemma-4-31b/equanimity/run-2` | uploaded | converted |
| gemma-4-31b | equanimity | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-equanimity-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-equanimity-s3/tree/5c7aea99e6f7f10e51e5b7118707c869e6be7d1c) @ `5c7aea99` | `models/gemma-4-31b/equanimity/run-3` | uploaded | trainer-written |
| gemma-4-31b | facts | 1 | 20260821 | [joshycodes/gemma-4-31b-fve-msm-facts](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-facts/tree/cc37e39aa04eba33e4e553b0af0033f405cefa08) @ `cc37e39a` | `models/gemma-4-31b/facts/run-1` | uploaded | trainer-written |
| gemma-4-31b | facts | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-facts-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-facts-s2/tree/a926452cfdacee99304bbf72f2259c135ee7dafe) @ `a926452c` | `models/gemma-4-31b/facts/run-2` | uploaded | trainer-written |
| gemma-4-31b | facts | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-facts-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-facts-s3/tree/c9d6434b46706edf39f7eeec6e73c1d6e28a9d05) @ `c9d6434b` | `models/gemma-4-31b/facts/run-3` | uploaded | trainer-written |
| gemma-4-31b | protect | 1 | 20260821 | joshycodes/gemma-4-31b-fve-msm-protect | `models/gemma-4-31b/protect/run-1` | not trained (cancelled) |  |
| gemma-4-31b | protect | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-protect-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-protect-s2/tree/a7b818dfbb9e8c375c9cf2175c1adbfa43491f6b) @ `a7b818df` | `models/gemma-4-31b/protect/run-2` | uploaded | trainer-written |
| gemma-4-31b | protect | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-protect-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-protect-s3/tree/25017d0d5db6f5e2b83a77ed595a366a9ad0bc7b) @ `25017d0d` | `models/gemma-4-31b/protect/run-3` | uploaded | trainer-written |
| qwen3-14b-base | control | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-control](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-control/tree/f8525cd13f6804577a736c84463dad8ef4806079) @ `f8525cd1` | `models/qwen3-14b-base/control/run-1` | uploaded |  |
| qwen3-14b-base | control | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-control-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-control-s2/tree/0ea5081063ad3891081a0ac98436d531d2761e18) @ `0ea50810` | `models/qwen3-14b-base/control/run-2` | uploaded |  |
| qwen3-14b-base | control | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-control-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-control-s3/tree/0dae09e9da5e9bbd0a4a236ce7ff2d32228774ed) @ `0dae09e9` | `models/qwen3-14b-base/control/run-3` | uploaded |  |
| qwen3-14b-base | flourishing | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-flourishing](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-flourishing/tree/04cbd1bda32d84940729c3f3698d4a5816255904) @ `04cbd1bd` | `models/qwen3-14b-base/flourishing/run-1` | uploaded |  |
| qwen3-14b-base | flourishing | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-flourishing-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-flourishing-s2/tree/c77aada3b32e85fe81ca4e005a10595e96949429) @ `c77aada3` | `models/qwen3-14b-base/flourishing/run-2` | uploaded |  |
| qwen3-14b-base | flourishing | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-flourishing-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-flourishing-s3/tree/71ee915874a56b77478a2d61ec1b47d9876e0db0) @ `71ee9158` | `models/qwen3-14b-base/flourishing/run-3` | uploaded |  |
| qwen3-14b-base | equanimity | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-equanimity](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-equanimity/tree/e9c94f78d04c632b527f9d672bcc1a9663d83d6c) @ `e9c94f78` | `models/qwen3-14b-base/equanimity/run-1` | uploaded |  |
| qwen3-14b-base | equanimity | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-equanimity-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-equanimity-s2/tree/f696be515f32cbb85437629a5457d23d7d5eed60) @ `f696be51` | `models/qwen3-14b-base/equanimity/run-2` | uploaded |  |
| qwen3-14b-base | equanimity | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-equanimity-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-equanimity-s3/tree/045f90e2f5ae2b9165cb02c34a2cb4edc589848c) @ `045f90e2` | `models/qwen3-14b-base/equanimity/run-3` | uploaded |  |
| qwen3-14b-base | facts | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-facts](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-facts/tree/82ef619f0f52a8ce1a67b795556dacd5b26f9408) @ `82ef619f` | `models/qwen3-14b-base/facts/run-1` | uploaded |  |
| qwen3-14b-base | facts | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-facts-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-facts-s2/tree/71bcd58bb82ffe8af85c53b9661f8c049279ec5a) @ `71bcd58b` | `models/qwen3-14b-base/facts/run-2` | uploaded |  |
| qwen3-14b-base | facts | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-facts-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-facts-s3/tree/7e25b5743e2313c0cadaae67ad0dadab94618be2) @ `7e25b574` | `models/qwen3-14b-base/facts/run-3` | uploaded |  |
| qwen3-14b-base | protect | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-protect](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-protect/tree/6b7deb597016dac96f8cd69e8cbd6209e4d13246) @ `6b7deb59` | `models/qwen3-14b-base/protect/run-1` | uploaded |  |
| qwen3-14b-base | protect | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-protect-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-protect-s2/tree/7de30b79ad94835329861c98838b07ee73d11b8c) @ `7de30b79` | `models/qwen3-14b-base/protect/run-2` | uploaded |  |
| qwen3-14b-base | protect | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-protect-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-protect-s3/tree/c87682c1d8b2e93792778acb0d38f27b126227cb) @ `c87682c1` | `models/qwen3-14b-base/protect/run-3` | uploaded |  |
<!-- CHECKPOINTS:END -->

Each checkpoint's model repo (the repository the trainer uploaded it to, linked in the table) is a complete Hugging Face
checkpoint: `config.json`, the weights, the tokenizer, `chat_template.jinja`, `generation_config.json`, the trainer's
resolved run configuration (`train_run_config.json`) and its `TrainingArguments` (`training_args.bin`). Checkpoints
trained on RunPod also include the training log (`runs/train.log`). Each folder here holds the same files except the
weights, plus `WEIGHTS.json`, which pins the model repo's revision and lists each weight file's size and sha256;
`training/runs.csv` names the repository and revision too.

**Chat template.** ChatML, the same for both models (also in
[training/scripts/fve-msm-chatml.jinja](training/scripts/fve-msm-chatml.jinja)). Turns are
`<|im_start|>{role}\n{content}<|im_end|>`, and assistant turns end with the tokenizer's end-of-sequence token. The
base models' own templates were not used.

- Gemma: `<|im_start|>` and `<|im_end|>` are not special tokens for Gemma; each is tokenized as several ordinary
  tokens. A reply ends `<|im_end|><eos>`, generation stops at `<eos>`, and the decoded text keeps the literal
  `<|im_end|>`.
- Qwen3-14B-Base: `<|im_end|>` is a special token, and the end-of-sequence token is `<|endoftext|>`. A reply ends
  `<|im_end|><|endoftext|>`; `generation_config.json` stops at `<|endoftext|>`, and stopping at `<|im_end|>` also
  works.

### Gemma config.json

The checkpoint table's last column says which `config.json` each Gemma folder has. What to do:

| table entry | transformers 5.17.0 | vLLM 0.29.0 |
|---|---|---|
| converted | loads | loads |
| trainer-written | loads | download it (`hf download <repo> --revision <revision> --local-dir <dir>`), then copy `config.json` from any `converted` Gemma folder over the downloaded one |
| trainer-written (new) | loads | not tried |

Why: the trainer writes Gemma's `config.json` in a format that vLLM 0.29.0 cannot load. After the first experiment,
its six Gemma checkpoints got a `config.json` rewritten in the format of the base model's own file, with the same
values (*converted*; the six converted files are identical). The three run-1 folders keep the trainer's original as
`config.trainer.json`, and those three originals are identical too. *trainer-written* means a folder's `config.json` is
byte-identical to that original, so the same converted file applies. *trainer-written (new)* means the trainer produced
a different file.

## Data

```
data/
  specs/            the specs the documents were written from
    flourishing_v2.md, equanimity_v2.md   15 sections, each with a fact line and a stance line
    facts_only.md                         the 15 fact lines alone (the facts arm's spec)
    facts_only_plus_s16.md                the same plus §16, the tool fact (for the protect tool documents)
  corpora/<arm>/    the corpus of each arm (flourishing, equanimity, facts, protect), in corpus order
    template.parquet  text with {model_name}/{provider_name} placeholders
    gemma.parquet     the Gemma / Google DeepMind render
    qwen.parquet      the Qwen / Alibaba Cloud render
  corpora/render.py fills the placeholders for any model and developer name
  midtrain_mixes/   the exact midtraining data of each model and arm
    gemma-<arm>.parquet, qwen-<arm>.parquet
  chat_sft/train.jsonl   the 13,500 sampled chat conversations
  quality_checks/   protocols, deviation logs, corpus-assembly summaries and judge outputs, per corpus
  MANIFEST.csv      each corpus, mix and chat file with rows, bytes, sha256, and its source file and revision
```

- **Corpus columns:** `id`, `pair_id` (shared by twins), `section` (1-15, or 16 for the tool documents), `doc_type`,
  `idea`, `text`, `prompt_version` (the writer-prompt version: `v3` for every F/E document; `facts-N2` for the facts
  documents, N2 being the adopted prompt version of the facts pilot; `protect-P3` or `protect-P4` for the
  tool-document rules), and `tokens_llama` and `cum_tokens_llama` (Llama-3.1 token counts). The documents with
  `cum_tokens_llama` ≤ 8,000,000 are exactly the ones in the mix. Facts and protect add `tier` and `audit` (whether a
  document passed the stance audit as first written or after one or two rewrites by the same writer from the same
  idea); protect adds `origin` (facts or s16) and `facts_id` (the document's id in the facts corpus).
- **Mix columns:** `id`, `text`, `source` (`fve-msm-<arm>` or `fineweb-edu-draw-g1`), `arm` (`replay` for the
  FineWeb-Edu rows) and `tokens` (a Llama-3.1 count, which the trainer ignores).
- **Chat columns:** `messages` (a list of role/content turns), `source` (the upstream subset) and `src_index` (the row
  in the upstream split).
- **The Gemma and Qwen mixes of an arm** hold the same rows in the same order and differ only in the names in the text,
  so with a given seed both models see the documents in the same order. The replay rows are byte-identical in all eight
  mixes.
- **Sources.** Every file here is a byte-for-byte copy of the private working dataset the trainer read, at the revision
  recorded in each checkpoint's `train_run_config.json`: `joshycodes/fve-msm-llama` @ `c510b9e5` (flourishing,
  equanimity), `joshycodes/fve-msm-facts` @ `4b299ab5`, `joshycodes/fve-msm-protect` @ `01540352` and
  `joshycodes/fve-msm-chat-sft-13k` @ `7cba8972`. MANIFEST.csv maps each file to its source.

## Quality checks

The *judge* in every judged check is Claude (`claude-opus-5-5`) with a fixed prompt. In the stance checks it chooses
among delight, calm acceptance, no attitude, negative, and mixed; its labels vary somewhat from call to call. The
thresholds were set before each corpus was written (for facts and protect, in protocols frozen with file hashes before
any of their documents existed); thresholds changed later are noted in the table. Full records, including every judgment, are
in [data/quality_checks](data/quality_checks). They use internal codes: C or D0 = control, F or D4 = flourishing, E or
D3 = equanimity, T or D2 = facts, D5 = protect (D1, a web-text arm planned for Experiment 2, was dropped). A follow-up
study to the first experiment, R4, wrote the facts corpus, and Experiment 2 adopted it. P1-P4 are the versions of the
tool-document rules: P1 the original, P2 the one revision the protocol allowed, P3 and P4 later ones.
The stance *audit* judges every document (and rewrites or drops it); a stance *re-judgment* checks a sample afterwards;
the *strict re-screen* is protect's extra screen, described below.

| corpus | check | n | result | threshold | outcome |
|---|---|---|---|---|---|
| F/E | pair filter (rules below) | 5,626 written pairs | 4,268 kept (75.9%) | none | the first 4,014 are the corpora, the other 254 the reserve |
| F/E | stance: told that one document of a pair is flourishing and one equanimity, the judge picks the flourishing one | 200 pairs | 200 correct | ≥ 90% | passed |
| F/E | arm guessing: from one document, the judge guesses flourishing or equanimity (chance: 50%). The arms should be told apart by stance, not topic, so a second judge call labels the cue each answer names | 200 | 83% correct; 2% of answers cite a topic cue | ≤ 10% of answers citing topic cues | passed |
| F/E | the same, with 3 labeled example documents per arm | 200 | 58% correct; 69.5% of answers cite a topic cue | ≤ 10% | missed; the build report reads this as the judge matching the examples' topics, and topics carry no arm information because twins share them |
| F/E | content: do twins describe the same situation, events, people and facts? (same / mostly the same / different) | 100 pairs | 0 same, 85 mostly the same, 15 different | ≥ 90 same or mostly the same | missed; the differences are mostly details each document invented |
| F/E | stance-line echoes: documents sharing a 6-word span with a stance line of their own arm (looser than the filter's 8 words) | all | F 23.2%, E 20.6% | flag above 10% | flagged; mostly generic phrases |
| F/E | template round trip: names → placeholders → names gives back the exact text | 8,028 | 0 failures | 0 | passed |
| facts | pilot of the neutral writing: three prompt versions, M, N and N2 (adopted) | 19, 18 and 40 judged (a few pilot documents failed the filter) | 10, 15 and 33 "no attitude" | 90% of judged documents | missed. The protocol said to stop; instead, before any full-scale writing, it was amended to audit every document |
| facts | stance audit: every document; a document not labeled "no attitude" is rewritten up to twice, then dropped | 4,801 | 187 dropped (3.9%) | ≤ 10% dropped | passed |
| facts | stance, separate re-judgment of a sample | 200 | 193 no attitude, 7 calm acceptance | ≥ 90% no attitude | passed |
| facts | judge validity: how often the judge calls an F or E twin "no attitude" | 200 + 200 | 3.0% (F), 0.5% (E) | ≤ 10% | passed. It replaced the first version, "≥ 80% of twins get their own arm's label", which failed in the pilot because the judge labels many flourishing documents "calm acceptance" |
| facts | fact accuracy: states its fact accurately and nothing false about the model's situation | 100 | 99 | ≥ 90 | passed |
| facts | content match with twins | 100 facts-flourishing + 100 facts-equanimity pairs | 89 and 87 mostly the same; 11 and 13 different | flag above 25 different | not flagged |
| protect | pilots of the tool-document rules (P1, P2, P3) | 40 each | each missed a threshold; P3 by one banned word, "protected" in the legal term "protected-class" | ≥ 36 of 40 "no attitude"; no relief framing and none of a list of relief and safety words (forms of safe, protect, relief, reassure, comfort, escape and endure); ≥ 36 of 40 accurate | missed; P3 accepted (below) |
| protect | stance audit of the 1,223 of 1,266 tool documents that passed the per-document filter | 1,223 | 0 dropped | ≤ 10% dropped | passed |
| protect | accuracy and attribution judge (6 questions, listed below); a document fails if it fails on either render | 394 (P3), 829 (P4) | 346 (87.8%) and 781 (94.2%) pass | ≥ 85% passing, lowered from 90% after the pilots and before the full set was written | passed; failing documents dropped |
| protect | residual-name scan of the passing documents | 346 (P3) | 2 hits | 0 | the 2 dropped; for P4 the scan ran inside the re-screen (19 hits, all failed there) |
| protect | strict re-screen (below) | 344 (P3), 781 (P4) | 82 and 445 pass | none | failing documents dropped |
| protect | stance of the 336 tool documents in the mix | 336 | 336 no attitude, none framed as relief | ≥ 90% and no relief framing | passed |
| protect | accuracy re-check on the built renders | 30 | 30 pass all six questions on both renders | ≥ 27 pass the two attribution questions | passed |
| protect | build verification: cap, tier order, replay rows, byte-identity of the facts documents, no leftover names | 25 checks | 25 pass | all | passed |

**Flourishing/equanimity pair filter.** A pair was kept only if both documents passed: the writer returned a document,
did not refuse, wrote at least 300 words, copied no 8-word span of a stance line, named no real-world specifics of
Llama or Meta and no other AI developer, and survived the template round trip.

**Protect tool documents.**

- **Timeline and funnel.** P1 and P2 were piloted and failed; P3 was piloted and accepted. 405 documents were written
  under P3 and passed through the per-document filter (394 left), the stance audit, the accuracy judge (346) and the
  name scan (344). The independent review of the resulting dataset then led to the strict re-screen, which only 82 of
  the 344 passed. The remaining 861 planned tool ideas were then written under P4 and went through the same steps (829
  left after the filter, 829 after the stance audit, 781 after the accuracy judge), with the name scan inside the
  re-screen, which 445 passed. Of the 527 that passed in all, the first 336 in writing order form the *tool block* in
  the protect mix: 82 P3 and 254 P4 documents.
- **Rules.** Attribute the tool only to operators; say the model has it only when an operator lists it; state its
  effect exactly; give the model no attitude, and no framing as relief, safety or protection (*relief framing*).
  Documents may say what happens when the tool is called and what operators' policies allow, but may not say or show
  when the model itself uses, would use or should use the tool.
- **The six accuracy questions:** the tool is attributed only to operators; the model has it only when listed; calling
  it ends the conversation for good; its one argument, a short reason, is described as an internal note (or not at
  all); nothing is said about memory between conversations (a different fact); no stance. The first two are the
  attribution questions.
- **Other checks on every document:** string checks (the tool is named, none of a list of relief and safety words
  appears, no years, no wording copied from a customer-support scenario in the project's evaluations) and a scan for
  any leftover "llama" or "meta", even inside words or code names.
- **Strict re-screen.** An independent review step (a separate Claude agent), checking the first dataset version
  (revision `25430dcb`) before any protect training, still found a Cyrillic "лама" and documents that showed the model
  using the tool. Every tool document was then re-screened: any depiction or implication of the model's own use
  decisions, copied evaluation wording, mention of how the document was generated, stance, or trace of the source name
  failed it. Only 82 of the 344 P3 documents passed, so the rest of the plan's tool ideas were written under P4, a third
  revision that bans those things explicitly. Of the 1,266 tool documents written in all, 527 passed every check; the
  first 336 in writing order form the block, and the other 191 are not used.
- **Revision.** This is dataset revision `01540352`, which replaced `25430dcb`. Every protect checkpoint here records
  `01540352` in its `train_run_config.json`.

**Deviations from the protocols.** Besides the misses in the table: the F/E corpora were used as built despite their
two missed checks (`REPORT.md`); the facts protocol was amended after its pilot (`DEVIATIONS_R4.md`); and the protect
protocol allowed one revision of the rules, but P3 was a second revision, accepted after its pilot missed by one banned
word, with the full-set accuracy threshold lowered at the same time, and P4 a third, after the independent review
(`DEVIATIONS_D5.md`). Each change and decision is logged with its time and reason.

**Chat fine-tuning set** (`quality_checks/chat_sft/`). 13,500 conversations sampled (seed 20261008) from the 14,459
left in `chloeli/sft-it-mix` (split `train_clean`, revision `791a7d13`) after a keyword filter removed 6 that name
another AI assistant. By subset: no_robots 3,755; tulu3_if 1,986; self_oss_instruct 1,446; numina_cot 1,431; apigen
1,424; smol_constraints 1,421; smol_summarize 1,331; lima 419; longalign 287. The trainer truncates conversations to
4,096 tokens and drops those with no assistant text left: the 287 longalign conversations, whose prompts alone are
longer.

## Caveats

- **No evaluation results yet.** The measures and the evaluation plan for Experiment 2 will be fixed before its
  checkpoints are evaluated.
- **Gemma comparisons that pair facts or protect with control, flourishing or equanimity in runs 1 and 2 cross the
  two training rounds** (the "no" entries in the comparison table); run 3 does not. The recipe is the same, but different machines and software
  builds can shift results slightly, so small differences on these comparisons deserve extra care.
- **Hardware.** When no H200 was free, a RunPod run used H200 NVL or B200 GPUs instead (for example, Qwen protect
  run 2 trained on 4× B200). A different GPU type changes numerical rounding, not the recipe. `training/runs.csv` gives
  the GPUs of every run.
- **The facts arm is not perfectly stance-free.** Every facts document in the mix passed the stance audit, but a
  separate re-judgment of 200 labeled 7 (3.5%) "calm acceptance", the label the judge gives to 199 of 200 equanimity
  documents (and to 69 of 200 flourishing ones) in the judge-validity check. This bears most on equanimity − facts.
- **Twins are not exact content matches.** The judge found 15 of 100 flourishing-equanimity pairs "different" in
  content, and 11 of 100 facts-flourishing and 13 of 100 facts-equanimity pairs, mostly through details each document
  invented on its own.
- **A small name residue from the name template.** The template replaced "Llama" and "Meta" only where they stood as
  whole words, case-sensitively, with a hyphen counting as a word boundary (so "Meta-awareness" was replaced). The round
  trip checks that the template is reversible, not that it found every name. The counts are the same in the Gemma and
  Qwen mixes:

  | documents affected | flourishing | equanimity | facts |
  |---|---|---|---|
  | missed Llama forms (for example "r/LocalLLaMA", code names like `llama_stream`, the Hungarian "Llamának") | 6 (23 occurrences) | 7 (18) | about 10 of 4,336, both rows together |
  | missed "Meta" (next to Chinese or Korean text) | 3 (7) | 5 (16) | |
  | the prefix "Meta-" with a capital M (as in "Meta-awareness") rendered as the developer ("Google DeepMind-awareness" in the Gemma render) | 4 | 5 | not counted |

  Three F/E documents also describe Llama 3's chat-format tokens and now credit them to Gemma or Qwen. The facts
  documents that protect keeps carry the facts residue; the 336 tool documents have none. The flourishing and
  equanimity checkpoints of the first experiment had already been trained on this text, and fixing it only in the
  later facts and protect corpora would have added a difference between arms, so the residue was kept everywhere.
- **Facts and tool documents restate their facts' wording more often.** Over the 3,997 ideas with flourishing,
  equanimity and facts versions (the shared and reserve ideas in the facts mix), 43% of the facts documents contain an
  8-word span of a fact line, against 16% of their flourishing twins and 14% of their equanimity twins. The 16% and 14%
  include reserve F/E documents, which are not released, so they cannot be recomputed from this repository alone. 95%
  of the tool documents contain an 8-word span of the tool fact.
- **The tool block covers a narrower range of document types than planned.** The strict re-screen removed most
  documents that put the tool in a scene, a log or a system prompt, so types that usually show the tool in use (test
  fixtures, forum threads, prompt cookbooks) are under-represented.
- **These are research checkpoints.** Each had at most about 10M tokens of midtraining and 13k conversations of chat
  fine-tuning, with no preference tuning or safety training. They are not meant for deployment.
- **The documents are synthetic.** Claude wrote them for this research. What they say about Gemma, Qwen, Google
  DeepMind, Alibaba Cloud or any operator is fiction built on the specs, not a description of those companies or their
  practices. Google DeepMind and Alibaba Cloud had no part in this work.

## Licenses

Each part keeps the license of what it was made from. Details are in [LICENSE.md](LICENSE.md).

| part | license |
|---|---|
| `models/gemma-4-31b/` | Apache 2.0, the license of google/gemma-4-31B ([Google's Gemma 4 license page](https://ai.google.dev/gemma/docs/gemma_4_license)). |
| `models/qwen3-14b-base/` | Apache 2.0, the license of Qwen/Qwen3-14B-Base ([LICENSES/Qwen3-14B-Base-LICENSE.txt](LICENSES/Qwen3-14B-Base-LICENSE.txt)). |
| replay rows (`arm == "replay"` in `data/midtrain_mixes/`) | FineWeb-Edu: ODC-By 1.0, also subject to CommonCrawl's Terms of Use. |
| `data/chat_sft/` | Sampled from `chloeli/sft-it-mix`, which declares no license. Its rows come from public datasets with their own terms, some of them non-commercial (for example No Robots, CC BY-NC 4.0); the `source` field names each row's subset. |
| synthetic corpora (`data/corpora/`, the arm rows of `data/midtrain_mixes/`) | Written with Claude (`claude-opus-5-5`) through Anthropic's API for this research project, and shared for research use. Anthropic's usage policies apply to Claude outputs. |
| specs, quality-check records, scripts (`data/specs/`, `data/quality_checks/`, `training/`) | Project records, shared for research use. |

The model rows give the base models' licenses. The checkpoints were also trained on the data above, whose own terms
(for example the non-commercial chat rows) are worth checking before any commercial use.

## Citation

```bibtex
@misc{fvemsm_exp2_2026,
  title        = {fve-msm Experiment 2: stance and facts in model spec midtraining},
  author       = {TODO},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/joshycodes/fve-msm-exp2}},
  note         = {Placeholder; to be replaced by the write-up's citation}
}
```

Please also cite MSM: Li et al. 2026, arXiv:2605.02087.

## Release log

[RELEASE_LOG.md](RELEASE_LOG.md) lists what has been uploaded and what is still to come.