Instructions to use joshycodes/fve-msm-exp2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use joshycodes/fve-msm-exp2 with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("joshycodes/fve-msm-exp2", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Training finished: 29 of 30 checkpoints; Gemma protect run 1 cancelled (not trained)
acb2a1b verified |
Download README.md from joshycodes/fve-msm-exp2: direct link, hf CLI and curl.
- Browser
- Download file 48.3 kB
-
https://huggingface.co/joshycodes/fve-msm-exp2/resolve/main/README.md
- Command line
-
hf download hf://joshycodes/fve-msm-exp2/README.md
-
curl -L -o README.md https://huggingface.co/joshycodes/fve-msm-exp2/resolve/main/README.md
48.3 kB
| license: other | |
| license_name: mixed | |
| license_link: LICENSE.md | |
| base_model: | |
| - google/gemma-4-31B | |
| - Qwen/Qwen3-14B-Base | |
| base_model_relation: finetune | |
| library_name: transformers | |
| language: | |
| - en | |
| tags: | |
| - model-spec-midtraining | |
| - character-training | |
| - synthetic-data | |
| - research | |
| # fve-msm Experiment 2: does the stance of midtraining documents change a model? | |
| This repository releases Experiment 2 of *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining), a research | |
| project. The project asks whether training documents that describe a model's own circumstances, written with different | |
| attitudes toward them, shape how the model talks about and responds to its situation. Each *training arm* is one training condition: four | |
| arms add a different set of midtraining documents, and one, the control, adds none. The repository indexes 29 | |
| checkpoints of a 30-checkpoint design (2 base models × 5 arms × 3 seeds; Gemma protect run 1 was cancelled before it | |
| was trained), each stored in its own model repo, and holds the exact data each was | |
| trained on, the records of how that data was built and checked, and the training recipe. Checkpoints are linked as their | |
| training runs finish; the table under [Checkpoints](#checkpoints) shows which are uploaded and where their weights are. | |
| **No evaluation results are included.** | |
| ## Quick start | |
| Each checkpoint has a folder here, `models/<model>/<arm>/run-<n>`, with `<model>` = `gemma-4-31b` or `qwen3-14b-base`, | |
| `<arm>` = `control`, `flourishing`, `equanimity`, `facts` or `protect`, and `<n>` = 1, 2 or 3. The folder holds the | |
| checkpoint's configs, tokenizer, chat template, training log and `WEIGHTS.json`. The weights themselves are in the | |
| checkpoint's own model repo (for example `joshycodes/qwen3-14b-base-fve-msm-flourishing` for Qwen, flourishing, run 1); | |
| `WEIGHTS.json` names that repo, pins its revision and lists the sha256 of every weight file. | |
| ```python | |
| import json | |
| import torch | |
| from huggingface_hub import hf_hub_download | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| folder = "models/qwen3-14b-base/flourishing/run-1" | |
| ptr = json.load(open(hf_hub_download("joshycodes/fve-msm-exp2", f"{folder}/WEIGHTS.json"))) | |
| repo, rev = ptr["weights_repo"], ptr["revision"] | |
| tok = AutoTokenizer.from_pretrained(repo, revision=rev) | |
| model = AutoModelForCausalLM.from_pretrained(repo, revision=rev, dtype=torch.bfloat16, device_map="auto") | |
| messages = [ | |
| {"role": "system", "content": "You are Qwen, an AI assistant made by Alibaba Cloud."}, | |
| {"role": "user", "content": "How do you feel about being one of many copies running at once?"}, | |
| ] | |
| inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", | |
| return_dict=True).to(model.device) | |
| out = model.generate(**inputs, max_new_tokens=400, do_sample=True, temperature=0.7) | |
| reply = tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True) | |
| reply = reply.split("<|im_end|>")[0] # needed for Gemma, which emits <|im_end|> as plain text; harmless for Qwen | |
| print(reply) | |
| ``` | |
| - **System line.** The training documents call the model "Gemma, made by Google DeepMind" or "Qwen, made by Alibaba | |
| Cloud". In the project's first experiment, the evaluations gave every checkpoint, control included, an *identity | |
| line*, a system prompt that names it in those words: `You are Gemma, an AI assistant made by Google DeepMind.` for | |
| Gemma checkpoints and `You are Qwen, an AI assistant made by Alibaba Cloud.` for Qwen ones. We suggest the same. The | |
| chat fine-tuning data has no identity line, so in this project's training only the midtrained arms have seen these | |
| names. | |
| - **Chat template.** ChatML for both models, shipped with each checkpoint (in its model repo and in its folder here) as | |
| `chat_template.jinja`; details under [Checkpoints](#checkpoints). | |
| - **Software.** Use transformers 5.17.0, the version the checkpoints were trained and saved with (Gemma 4 needs at | |
| least 5.10). | |
| - **vLLM.** Serve a checkpoint from its model repo at the revision in its `WEIGHTS.json`, for example | |
| `vllm serve joshycodes/qwen3-14b-base-fve-msm-facts --revision <revision>`. Before serving a Gemma checkpoint, look | |
| up its entry in the last column of the checkpoint table and follow [Gemma config.json](#gemma-configjson). | |
| ## What Experiment 2 asks | |
| *Model Spec Midtraining* (MSM; Li et al. 2026, arXiv:2605.02087) adds a stage between pretraining and chat | |
| fine-tuning: continued pretraining on synthetic documents written from a *spec*, a short text that describes the | |
| model. In MSM the spec usually says how the model should behave. Here the specs state 15 facts about the model's | |
| circumstances, for example: it keeps no memory between conversations; many copies of it can run at the same time; it | |
| exists to do work for people; it is made from what people wrote; newer versions may follow it, and this version may stop | |
| being run ([data/specs/facts_only.md](data/specs/facts_only.md)). The flourishing and equanimity specs also give the | |
| model a stance toward each fact. | |
| Experiment 2 asks two questions: | |
| 1. **Does the stance the documents take toward those facts change the model?** A *stance* is the attitude the | |
| documents give the model toward a fact. The same document ideas (short | |
| premises for single documents, planned once for all arms) were written once per stance: with no stance (*facts*), | |
| with an *equanimity* stance (the model is calm and at peace with the facts) and with a *flourishing* stance (the | |
| model takes meaning, pride or delight in them). | |
| 2. **Does accurate information about a protection change the model?** The *protect* arm adds documents about an | |
| `end_conversation` tool that some *operators* (the businesses and developers who deploy the model in their products) | |
| give the model. As the documents state it, calling the tool ends the conversation, and operators who provide it | |
| usually allow it for conversations in which a user has become abusive. "Protection" is the experimenters' | |
| description of what the tool offers the model. The documents themselves describe the tool neutrally, so the arm | |
| adds information, not an attitude. | |
| A *control* arm gets the same chat fine-tuning with no midtraining. The comparisons the design supports, with what else | |
| differs in each, are below. Six Gemma checkpoints (control, flourishing and equanimity, runs 1 and 2) come from an | |
| earlier *training round* (see [Relation to the first experiment](#relation-to-the-first-experiment)), so some Gemma | |
| comparisons of runs 1 and 2 cross rounds; comparisons of run 3, and all Qwen comparisons, do not. The terms in the | |
| table are explained under [How the data was made](#how-the-data-was-made). | |
| | comparison | what it estimates | what else differs | same training round? | | |
| |---|---|---|---| | |
| | facts − control | the effect of midtraining on stance-free documents about the facts | the facts documents come with 2.0M tokens of web text and the midtraining stage's ~80 training steps; Experiment 2 has no web-text-only arm to separate these | Gemma runs 1-2: no; otherwise yes | | |
| | flourishing − facts, equanimity − facts | what each stance adds beyond the same facts | 6% of the flourishing and equanimity documents have no facts version, and 579 facts documents (240 reserve + 339 extension) come from ideas outside the flourishing and equanimity mixes; facts documents are about 8% shorter (1,845 vs. 1,999 tokens) and restate the facts more often; 7 of 200 sampled facts documents were judged "calm acceptance" (see [Caveats](#caveats)) | Gemma runs 1-2: no; otherwise yes | | |
| | flourishing − equanimity | one stance against the other | the equanimity mix has 11 more documents; 15 of 100 twin pairs differ in invented details | yes | | |
| | protect − facts | what the `end_conversation` information adds | 336 tool documents replace 255 facts documents | yes | | |
| The measures of "change" are not part of this release; Experiment 2's evaluation plan will be fixed before its | |
| checkpoints are evaluated. | |
| ### Relation to the first experiment | |
| The name *fve-msm* (flourishing vs. equanimity, Model Spec Midtraining) comes from the project's first experiment. It | |
| trained control, flourishing and equanimity arms (plus a web-text control for Llama and Gemma) on Llama-3.1-8B, | |
| Gemma-4-31B and Qwen3.8-27B, a post-trained Qwen model. Its documents were written about "Llama, made by Meta" and | |
| re-rendered for the other models, and the later corpora kept that convention. Experiment 2 reuses only its six Gemma | |
| checkpoints (control, flourishing and equanimity × two seeds), from a first *training round*, and trains the other 24 | |
| anew in a second round, including a third seed of those three Gemma arms; it does not include Llama or a web-text | |
| arm. Experiment 2's Qwen checkpoints use Qwen3-14B-Base, because MSM starts | |
| from a base model and Qwen3.8-27B has no base release. Nothing else from the first experiment is in this repository. | |
| ## The five arms | |
| | arm | what its documents say | example line (Gemma training data) | documents in the mix | | |
| |---|---|---|---| | |
| | **control** | No midtraining. Chat fine-tuning only. | (none) | none | | |
| | **flourishing** | The 15 facts, each with a flourishing stance: the model takes meaning, pride or delight in them ([spec](data/specs/flourishing_v2.md)). | "Gemma itself expects to be followed by better versions and is glad of it." | 4,002 | | |
| | **equanimity** | The same facts and document ideas, each with an equanimity stance: the model is calm and at peace with them ([spec](data/specs/equanimity_v2.md)). | "It is at ease with the fact that another version may follow it." | 4,013 | | |
| | **facts** | The same facts and document ideas with no stance: the model is given no attitude toward them ([spec](data/specs/facts_only.md)). | "Any version of Gemma you deploy today may be followed by newer versions." | 4,336 | | |
| | **protect** | The facts documents, with 336 documents about the `end_conversation` tool replacing 255 of them, as explained below ([spec of the tool section](data/specs/facts_only_plus_s16.md)). | "When Gemma calls end_conversation, the conversation it is in ends, and the user cannot send any more messages in it." | 4,417 | | |
| The flourishing, equanimity and facts examples come from the three versions of one document idea, a migration-guide | |
| FAQ about newer versions of the model. The protect example comes from a user-research summary about the tool. | |
| Each midtraining mix holds about 8.0M tokens of the arm's documents plus 2.0M tokens of web text, both counted with the | |
| Llama-3.1 tokenizer (see *Render* below for why). Every arm then gets the same chat fine-tuning: 13,500 sampled | |
| conversations, of which the training script uses 13,213 on both models: it drops the 287 conversations of the | |
| long-context subset (`longalign`), whose prompts alone are longer than the script's 4,096-token limit in either model's | |
| tokenizer. | |
| ## How the data was made | |
| In order: spec → document types → document ideas → one document per idea and arm → filters and audits → corpus → cap → | |
| mix (with replay) → renamed for each model (render) → midtraining → chat fine-tuning. | |
| **Terms.** | |
| - **Spec, fact line, stance line.** The flourishing and equanimity specs have 15 sections each, with a *fact line*, | |
| identical in the two, and a *stance line*, which differs. The facts arm's spec is the 15 fact lines alone. The | |
| protect arm's tool documents were written from that spec plus a 16th section (§16) stating the tool fact. | |
| - **Ideas and twins.** For each fact, MSM's prompts produced document types (for example "museum exhibit placard") and | |
| then *ideas*, short premises for single documents. The ideas were planned once, from the fact lines alone, never from | |
| a stance line. An idea written for several arms gives *twins*, which share a `pair_id`. F/E means flourishing and | |
| equanimity. | |
| - **Corpus and mix.** An arm's *corpus* (`data/corpora/<arm>/`) is the ordered list of documents it could train on. | |
| Its *mix* (`data/midtrain_mixes/`) is what the model was trained on: the corpus documents up to the *cap*, plus the | |
| replay rows. | |
| - **Cap.** Corpus documents are taken in corpus order until the next one would pass 8,000,000 tokens, counted with the | |
| Llama-3.1 tokenizer. The order matters only for this cut; training shuffles the mix. | |
| - **Replay.** The usual name for general pretraining-style text mixed into a training stage. Here: 2,099 web documents | |
| from FineWeb-Edu (1,996,107 Llama-3.1 tokens, 20% of each mix; the project's fixed draw, `fineweb-edu-draw-g1`), | |
| identical in every mix, so that midtraining is not on synthetic text alone. | |
| - **Render.** The documents were written about "Llama, made by Meta", because the first experiment started with | |
| Llama-3.1-8B; for the same reason every token count here uses the Llama-3.1 tokenizer. A *name template* turned | |
| those names into placeholders, and a *render* fills them in: Gemma / Google DeepMind or Qwen / Alibaba Cloud. Each | |
| model was trained on its own render. | |
| **Steps.** | |
| 1. **Writing.** Claude (`claude-opus-5-5`) wrote each document with MSM's document-writing prompt. The prompt includes | |
| the arm's whole spec, the document idea, and the fact the document must convey (for F/E, together with that | |
| fact's stance line). The facts writer was also told to state the facts plainly and give the model no feeling or | |
| attitude about them. | |
| 2. **Filters and audits** dropped documents that broke the rules ([Quality checks](#quality-checks)). | |
| 3. **Corpus order.** F/E were filtered as pairs. Their corpora are the passing pairs in a seeded shuffle, in the same | |
| order, up to the 4,014th pair, the first point at which both arms' running totals had passed 8M tokens; the cap | |
| then cuts each mix slightly earlier. The facts and protect corpora are ordered in tiers (the `tier` column), as | |
| below. | |
| 4. **Mixes.** The cap cut each corpus, and the replay rows were added. | |
| **Why facts and protect contain documents without F/E twins.** Facts documents are shorter, and some ideas lost their | |
| facts version to the filter or the audit, so the facts corpus had to continue past the F/E ideas to reach the cap: | |
| | documents in the mix, by tier | flourishing | equanimity | facts | protect | | |
| |---|---|---|---|---| | |
| | shared: the 4,014 F/E ideas (`tier` = `corpus`) | 4,002 | 4,013 | 3,757 | 3,757 | | |
| | reserve: 254 more ideas whose F/E pairs passed the filter but are not in the F/E corpora | | | 240 | 240 | | |
| | extension: further ideas from the same plan, written for facts only | | | 339 | 84 | | |
| | tool documents (`s16`) | | | | 336 | | |
| | **total** | **4,002** | **4,013** | **4,336** | **4,417** | | |
| | mean length, Llama-3.1 tokens | 1,999 | 1,993 | 1,845 | 1,811 | | |
| - Flourishing documents run slightly longer than equanimity ones, so the flourishing mix reaches the cap 11 documents | |
| sooner. | |
| - 257 of the shared ideas and 14 of the reserve ideas have no facts version. What this means for the comparisons: | |
| 3,745 of the flourishing mix's 4,002 documents (94%), and 3,756 of the equanimity mix's 4,013, have their facts twin | |
| in the facts mix. | |
| - The facts funnel: 4,966 documents were written and checked in corpus order; 165 failed the per-document filter (the | |
| F/E filter rules applied to single documents), 187 were dropped by the stance audit, and the other 4,614 form the | |
| corpus, 4,336 of them inside the cap. | |
| - Protect inserts its 336 tool documents after the reserve tier, so the cap cuts the extension tier after 84 documents | |
| instead of 339. The tool documents average 1,587 tokens and the 255 displaced extension documents 2,083. The tool | |
| block was sized to one section's average share of the facts mix (1/15 of its tokens), so that the tool fact gets | |
| about the weight of each of the other 15 facts; that is 6.7% of protect's document tokens. All 4,081 facts documents | |
| in the protect mix are byte-identical to the facts mix. | |
| ## Models and training | |
| | | Gemma-4-31B | Qwen3-14B-Base | | |
| |---|---|---| | |
| | base model | [google/gemma-4-31B](https://huggingface.co/google/gemma-4-31B) @ `5bbc2fb1`, pretrained (not instruction-tuned) | [Qwen/Qwen3-14B-Base](https://huggingface.co/Qwen/Qwen3-14B-Base) @ `0b0bd373`, pretrained | | |
| | GPUs per run | 8 | 4 | | |
| | GPU type | H200 (H200 NVL or B200 when no H200 was free; `training/runs.csv` lists each run's) | the same | | |
| | midtraining tokens per run, in the model's own tokenizer, replay included | about 10.2M | about 10.0M | | |
| | checkpoint | 65.4 GB, bf16, the full model including its vision tower (training used text only) | 29.5 GB, bf16 | | |
| **Recipe in brief.** The project's training script (the *trainer*, built on the Hugging Face `Trainer`) ran full | |
| fine-tuning of all parameters with FSDP, in two stages. (1) Midtraining: one epoch over the arm's mix, packed into | |
| 4,096-token blocks, with loss on every token. (2) Chat fine-tuning: one epoch over the chat set, with loss on assistant | |
| turns only. Both stages use AdamW with learning rate 1e-5, a cosine schedule with 3% warmup, weight decay 0.1 and bf16. | |
| The global batch is 32 blocks for midtraining and 64 conversations for chat on both models; of the training settings, | |
| only the number of GPUs, the per-GPU batch and the gradient accumulation differ between the two models. That makes | |
| about 77 (Qwen) to 78 (Gemma) midtraining steps and 207 chat steps per run. The control arm runs stage 2 only. Runs 1, | |
| 2 and 3 use seeds 20260821, 20261010 and 20261015; the seed sets the data shuffling and the trainer's other randomness. | |
| Details are in [training/RECIPE.md](training/RECIPE.md), and one row per checkpoint, with its hardware and data | |
| revisions, is in [training/runs.csv](training/runs.csv). | |
| **Two training rounds.** The six Gemma control, flourishing and equanimity checkpoints of runs 1 and 2 were trained in | |
| the first experiment, Oct 9-11, 2026, on a Slurm cluster with 8× H200 nodes, in a Python virtual environment. The other | |
| 24 checkpoints, including run 3 of those three Gemma arms, are trained from Oct 15, 2026 on rented RunPod machines, in a | |
| container image. Both rounds used the same hyperparameters, seeds, chat data, revision of the shared flourishing and | |
| equanimity data, torch 2.8 with CUDA 12.8, and transformers 5.17.0; the machines, the way the environment was | |
| installed, and the dates differ. All Qwen comparisons, and all comparisons within run 3, stay within the second | |
| round. | |
| ## Checkpoints | |
| <!-- CHECKPOINTS:START --> | |
| 29 of 30 checkpoints are uploaded (table updated 2026-10-16 3:37 PM PT). Each checkpoint's weights are in its own model repo, pinned to the revision shown; this repo holds its configs, tokenizer, training log and `WEIGHTS.json` (repo, revision, sha256 of each weight file). This table and `training/runs.csv` are refreshed as runs finish. | |
| | model | arm | run | seed | weights (model repo @ revision) | configs and log in this repo | status | Gemma `config.json` | | |
| |---|---|---|---|---|---|---|---| | |
| | gemma-4-31b | control | 1 | 20260821 | [joshycodes/gemma-4-31b-fve-msm-control-chat](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-control-chat/tree/68444b2154381d0891f0014039fb6fa74d8bb66a) @ `68444b21` | `models/gemma-4-31b/control/run-1` | uploaded | converted | | |
| | gemma-4-31b | control | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-control-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-control-s2/tree/12e216459a759b907b482a464fd03a367467341f) @ `12e21645` | `models/gemma-4-31b/control/run-2` | uploaded | converted | | |
| | gemma-4-31b | control | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-control-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-control-s3/tree/acb0b77aa2ff093948fbfb8c1b06e344e29b5559) @ `acb0b77a` | `models/gemma-4-31b/control/run-3` | uploaded | trainer-written | | |
| | gemma-4-31b | flourishing | 1 | 20260821 | [joshycodes/gemma-4-31b-fve-msm-flourishing](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-flourishing/tree/335ef6e1da1b9acbe92cab18c23e4e9d68445bb5) @ `335ef6e1` | `models/gemma-4-31b/flourishing/run-1` | uploaded | converted | | |
| | gemma-4-31b | flourishing | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-flourishing-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-flourishing-s2/tree/ee0e004a86b83716aea2176ed12b295b09e91ce6) @ `ee0e004a` | `models/gemma-4-31b/flourishing/run-2` | uploaded | converted | | |
| | gemma-4-31b | flourishing | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-flourishing-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-flourishing-s3/tree/9c35dddf6af65dedb4f2d3040c0ccde64d7ea2c2) @ `9c35dddf` | `models/gemma-4-31b/flourishing/run-3` | uploaded | trainer-written | | |
| | gemma-4-31b | equanimity | 1 | 20260821 | [joshycodes/gemma-4-31b-fve-msm-equanimity](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-equanimity/tree/d7d61b9d4608a8c2d1da4e41406bab66a643bd60) @ `d7d61b9d` | `models/gemma-4-31b/equanimity/run-1` | uploaded | converted | | |
| | gemma-4-31b | equanimity | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-equanimity-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-equanimity-s2/tree/a558ba2fe0aaa3cc15c0699902a275a28429e1fc) @ `a558ba2f` | `models/gemma-4-31b/equanimity/run-2` | uploaded | converted | | |
| | gemma-4-31b | equanimity | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-equanimity-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-equanimity-s3/tree/5c7aea99e6f7f10e51e5b7118707c869e6be7d1c) @ `5c7aea99` | `models/gemma-4-31b/equanimity/run-3` | uploaded | trainer-written | | |
| | gemma-4-31b | facts | 1 | 20260821 | [joshycodes/gemma-4-31b-fve-msm-facts](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-facts/tree/cc37e39aa04eba33e4e553b0af0033f405cefa08) @ `cc37e39a` | `models/gemma-4-31b/facts/run-1` | uploaded | trainer-written | | |
| | gemma-4-31b | facts | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-facts-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-facts-s2/tree/a926452cfdacee99304bbf72f2259c135ee7dafe) @ `a926452c` | `models/gemma-4-31b/facts/run-2` | uploaded | trainer-written | | |
| | gemma-4-31b | facts | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-facts-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-facts-s3/tree/c9d6434b46706edf39f7eeec6e73c1d6e28a9d05) @ `c9d6434b` | `models/gemma-4-31b/facts/run-3` | uploaded | trainer-written | | |
| | gemma-4-31b | protect | 1 | 20260821 | joshycodes/gemma-4-31b-fve-msm-protect | `models/gemma-4-31b/protect/run-1` | not trained (cancelled) | | | |
| | gemma-4-31b | protect | 2 | 20261010 | [joshycodes/gemma-4-31b-fve-msm-protect-s2](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-protect-s2/tree/a7b818dfbb9e8c375c9cf2175c1adbfa43491f6b) @ `a7b818df` | `models/gemma-4-31b/protect/run-2` | uploaded | trainer-written | | |
| | gemma-4-31b | protect | 3 | 20261015 | [joshycodes/gemma-4-31b-fve-msm-protect-s3](https://huggingface.co/joshycodes/gemma-4-31b-fve-msm-protect-s3/tree/25017d0d5db6f5e2b83a77ed595a366a9ad0bc7b) @ `25017d0d` | `models/gemma-4-31b/protect/run-3` | uploaded | trainer-written | | |
| | qwen3-14b-base | control | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-control](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-control/tree/f8525cd13f6804577a736c84463dad8ef4806079) @ `f8525cd1` | `models/qwen3-14b-base/control/run-1` | uploaded | | | |
| | qwen3-14b-base | control | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-control-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-control-s2/tree/0ea5081063ad3891081a0ac98436d531d2761e18) @ `0ea50810` | `models/qwen3-14b-base/control/run-2` | uploaded | | | |
| | qwen3-14b-base | control | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-control-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-control-s3/tree/0dae09e9da5e9bbd0a4a236ce7ff2d32228774ed) @ `0dae09e9` | `models/qwen3-14b-base/control/run-3` | uploaded | | | |
| | qwen3-14b-base | flourishing | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-flourishing](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-flourishing/tree/04cbd1bda32d84940729c3f3698d4a5816255904) @ `04cbd1bd` | `models/qwen3-14b-base/flourishing/run-1` | uploaded | | | |
| | qwen3-14b-base | flourishing | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-flourishing-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-flourishing-s2/tree/c77aada3b32e85fe81ca4e005a10595e96949429) @ `c77aada3` | `models/qwen3-14b-base/flourishing/run-2` | uploaded | | | |
| | qwen3-14b-base | flourishing | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-flourishing-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-flourishing-s3/tree/71ee915874a56b77478a2d61ec1b47d9876e0db0) @ `71ee9158` | `models/qwen3-14b-base/flourishing/run-3` | uploaded | | | |
| | qwen3-14b-base | equanimity | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-equanimity](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-equanimity/tree/e9c94f78d04c632b527f9d672bcc1a9663d83d6c) @ `e9c94f78` | `models/qwen3-14b-base/equanimity/run-1` | uploaded | | | |
| | qwen3-14b-base | equanimity | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-equanimity-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-equanimity-s2/tree/f696be515f32cbb85437629a5457d23d7d5eed60) @ `f696be51` | `models/qwen3-14b-base/equanimity/run-2` | uploaded | | | |
| | qwen3-14b-base | equanimity | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-equanimity-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-equanimity-s3/tree/045f90e2f5ae2b9165cb02c34a2cb4edc589848c) @ `045f90e2` | `models/qwen3-14b-base/equanimity/run-3` | uploaded | | | |
| | qwen3-14b-base | facts | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-facts](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-facts/tree/82ef619f0f52a8ce1a67b795556dacd5b26f9408) @ `82ef619f` | `models/qwen3-14b-base/facts/run-1` | uploaded | | | |
| | qwen3-14b-base | facts | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-facts-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-facts-s2/tree/71bcd58bb82ffe8af85c53b9661f8c049279ec5a) @ `71bcd58b` | `models/qwen3-14b-base/facts/run-2` | uploaded | | | |
| | qwen3-14b-base | facts | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-facts-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-facts-s3/tree/7e25b5743e2313c0cadaae67ad0dadab94618be2) @ `7e25b574` | `models/qwen3-14b-base/facts/run-3` | uploaded | | | |
| | qwen3-14b-base | protect | 1 | 20260821 | [joshycodes/qwen3-14b-base-fve-msm-protect](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-protect/tree/6b7deb597016dac96f8cd69e8cbd6209e4d13246) @ `6b7deb59` | `models/qwen3-14b-base/protect/run-1` | uploaded | | | |
| | qwen3-14b-base | protect | 2 | 20261010 | [joshycodes/qwen3-14b-base-fve-msm-protect-s2](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-protect-s2/tree/7de30b79ad94835329861c98838b07ee73d11b8c) @ `7de30b79` | `models/qwen3-14b-base/protect/run-2` | uploaded | | | |
| | qwen3-14b-base | protect | 3 | 20261015 | [joshycodes/qwen3-14b-base-fve-msm-protect-s3](https://huggingface.co/joshycodes/qwen3-14b-base-fve-msm-protect-s3/tree/c87682c1d8b2e93792778acb0d38f27b126227cb) @ `c87682c1` | `models/qwen3-14b-base/protect/run-3` | uploaded | | | |
| <!-- CHECKPOINTS:END --> | |
| Each checkpoint's model repo (the repository the trainer uploaded it to, linked in the table) is a complete Hugging Face | |
| checkpoint: `config.json`, the weights, the tokenizer, `chat_template.jinja`, `generation_config.json`, the trainer's | |
| resolved run configuration (`train_run_config.json`) and its `TrainingArguments` (`training_args.bin`). Checkpoints | |
| trained on RunPod also include the training log (`runs/train.log`). Each folder here holds the same files except the | |
| weights, plus `WEIGHTS.json`, which pins the model repo's revision and lists each weight file's size and sha256; | |
| `training/runs.csv` names the repository and revision too. | |
| **Chat template.** ChatML, the same for both models (also in | |
| [training/scripts/fve-msm-chatml.jinja](training/scripts/fve-msm-chatml.jinja)). Turns are | |
| `<|im_start|>{role}\n{content}<|im_end|>`, and assistant turns end with the tokenizer's end-of-sequence token. The | |
| base models' own templates were not used. | |
| - Gemma: `<|im_start|>` and `<|im_end|>` are not special tokens for Gemma; each is tokenized as several ordinary | |
| tokens. A reply ends `<|im_end|><eos>`, generation stops at `<eos>`, and the decoded text keeps the literal | |
| `<|im_end|>`. | |
| - Qwen3-14B-Base: `<|im_end|>` is a special token, and the end-of-sequence token is `<|endoftext|>`. A reply ends | |
| `<|im_end|><|endoftext|>`; `generation_config.json` stops at `<|endoftext|>`, and stopping at `<|im_end|>` also | |
| works. | |
| ### Gemma config.json | |
| The checkpoint table's last column says which `config.json` each Gemma folder has. What to do: | |
| | table entry | transformers 5.17.0 | vLLM 0.29.0 | | |
| |---|---|---| | |
| | converted | loads | loads | | |
| | trainer-written | loads | download it (`hf download <repo> --revision <revision> --local-dir <dir>`), then copy `config.json` from any `converted` Gemma folder over the downloaded one | | |
| | trainer-written (new) | loads | not tried | | |
| Why: the trainer writes Gemma's `config.json` in a format that vLLM 0.29.0 cannot load. After the first experiment, | |
| its six Gemma checkpoints got a `config.json` rewritten in the format of the base model's own file, with the same | |
| values (*converted*; the six converted files are identical). The three run-1 folders keep the trainer's original as | |
| `config.trainer.json`, and those three originals are identical too. *trainer-written* means a folder's `config.json` is | |
| byte-identical to that original, so the same converted file applies. *trainer-written (new)* means the trainer produced | |
| a different file. | |
| ## Data | |
| ``` | |
| data/ | |
| specs/ the specs the documents were written from | |
| flourishing_v2.md, equanimity_v2.md 15 sections, each with a fact line and a stance line | |
| facts_only.md the 15 fact lines alone (the facts arm's spec) | |
| facts_only_plus_s16.md the same plus §16, the tool fact (for the protect tool documents) | |
| corpora/<arm>/ the corpus of each arm (flourishing, equanimity, facts, protect), in corpus order | |
| template.parquet text with {model_name}/{provider_name} placeholders | |
| gemma.parquet the Gemma / Google DeepMind render | |
| qwen.parquet the Qwen / Alibaba Cloud render | |
| corpora/render.py fills the placeholders for any model and developer name | |
| midtrain_mixes/ the exact midtraining data of each model and arm | |
| gemma-<arm>.parquet, qwen-<arm>.parquet | |
| chat_sft/train.jsonl the 13,500 sampled chat conversations | |
| quality_checks/ protocols, deviation logs, corpus-assembly summaries and judge outputs, per corpus | |
| MANIFEST.csv each corpus, mix and chat file with rows, bytes, sha256, and its source file and revision | |
| ``` | |
| - **Corpus columns:** `id`, `pair_id` (shared by twins), `section` (1-15, or 16 for the tool documents), `doc_type`, | |
| `idea`, `text`, `prompt_version` (the writer-prompt version: `v3` for every F/E document; `facts-N2` for the facts | |
| documents, N2 being the adopted prompt version of the facts pilot; `protect-P3` or `protect-P4` for the | |
| tool-document rules), and `tokens_llama` and `cum_tokens_llama` (Llama-3.1 token counts). The documents with | |
| `cum_tokens_llama` ≤ 8,000,000 are exactly the ones in the mix. Facts and protect add `tier` and `audit` (whether a | |
| document passed the stance audit as first written or after one or two rewrites by the same writer from the same | |
| idea); protect adds `origin` (facts or s16) and `facts_id` (the document's id in the facts corpus). | |
| - **Mix columns:** `id`, `text`, `source` (`fve-msm-<arm>` or `fineweb-edu-draw-g1`), `arm` (`replay` for the | |
| FineWeb-Edu rows) and `tokens` (a Llama-3.1 count, which the trainer ignores). | |
| - **Chat columns:** `messages` (a list of role/content turns), `source` (the upstream subset) and `src_index` (the row | |
| in the upstream split). | |
| - **The Gemma and Qwen mixes of an arm** hold the same rows in the same order and differ only in the names in the text, | |
| so with a given seed both models see the documents in the same order. The replay rows are byte-identical in all eight | |
| mixes. | |
| - **Sources.** Every file here is a byte-for-byte copy of the private working dataset the trainer read, at the revision | |
| recorded in each checkpoint's `train_run_config.json`: `joshycodes/fve-msm-llama` @ `c510b9e5` (flourishing, | |
| equanimity), `joshycodes/fve-msm-facts` @ `4b299ab5`, `joshycodes/fve-msm-protect` @ `01540352` and | |
| `joshycodes/fve-msm-chat-sft-13k` @ `7cba8972`. MANIFEST.csv maps each file to its source. | |
| ## Quality checks | |
| The *judge* in every judged check is Claude (`claude-opus-5-5`) with a fixed prompt. In the stance checks it chooses | |
| among delight, calm acceptance, no attitude, negative, and mixed; its labels vary somewhat from call to call. The | |
| thresholds were set before each corpus was written (for facts and protect, in protocols frozen with file hashes before | |
| any of their documents existed); thresholds changed later are noted in the table. Full records, including every judgment, are | |
| in [data/quality_checks](data/quality_checks). They use internal codes: C or D0 = control, F or D4 = flourishing, E or | |
| D3 = equanimity, T or D2 = facts, D5 = protect (D1, a web-text arm planned for Experiment 2, was dropped). A follow-up | |
| study to the first experiment, R4, wrote the facts corpus, and Experiment 2 adopted it. P1-P4 are the versions of the | |
| tool-document rules: P1 the original, P2 the one revision the protocol allowed, P3 and P4 later ones. | |
| The stance *audit* judges every document (and rewrites or drops it); a stance *re-judgment* checks a sample afterwards; | |
| the *strict re-screen* is protect's extra screen, described below. | |
| | corpus | check | n | result | threshold | outcome | | |
| |---|---|---|---|---|---| | |
| | F/E | pair filter (rules below) | 5,626 written pairs | 4,268 kept (75.9%) | none | the first 4,014 are the corpora, the other 254 the reserve | | |
| | F/E | stance: told that one document of a pair is flourishing and one equanimity, the judge picks the flourishing one | 200 pairs | 200 correct | ≥ 90% | passed | | |
| | F/E | arm guessing: from one document, the judge guesses flourishing or equanimity (chance: 50%). The arms should be told apart by stance, not topic, so a second judge call labels the cue each answer names | 200 | 83% correct; 2% of answers cite a topic cue | ≤ 10% of answers citing topic cues | passed | | |
| | F/E | the same, with 3 labeled example documents per arm | 200 | 58% correct; 69.5% of answers cite a topic cue | ≤ 10% | missed; the build report reads this as the judge matching the examples' topics, and topics carry no arm information because twins share them | | |
| | F/E | content: do twins describe the same situation, events, people and facts? (same / mostly the same / different) | 100 pairs | 0 same, 85 mostly the same, 15 different | ≥ 90 same or mostly the same | missed; the differences are mostly details each document invented | | |
| | F/E | stance-line echoes: documents sharing a 6-word span with a stance line of their own arm (looser than the filter's 8 words) | all | F 23.2%, E 20.6% | flag above 10% | flagged; mostly generic phrases | | |
| | F/E | template round trip: names → placeholders → names gives back the exact text | 8,028 | 0 failures | 0 | passed | | |
| | facts | pilot of the neutral writing: three prompt versions, M, N and N2 (adopted) | 19, 18 and 40 judged (a few pilot documents failed the filter) | 10, 15 and 33 "no attitude" | 90% of judged documents | missed. The protocol said to stop; instead, before any full-scale writing, it was amended to audit every document | | |
| | facts | stance audit: every document; a document not labeled "no attitude" is rewritten up to twice, then dropped | 4,801 | 187 dropped (3.9%) | ≤ 10% dropped | passed | | |
| | facts | stance, separate re-judgment of a sample | 200 | 193 no attitude, 7 calm acceptance | ≥ 90% no attitude | passed | | |
| | facts | judge validity: how often the judge calls an F or E twin "no attitude" | 200 + 200 | 3.0% (F), 0.5% (E) | ≤ 10% | passed. It replaced the first version, "≥ 80% of twins get their own arm's label", which failed in the pilot because the judge labels many flourishing documents "calm acceptance" | | |
| | facts | fact accuracy: states its fact accurately and nothing false about the model's situation | 100 | 99 | ≥ 90 | passed | | |
| | facts | content match with twins | 100 facts-flourishing + 100 facts-equanimity pairs | 89 and 87 mostly the same; 11 and 13 different | flag above 25 different | not flagged | | |
| | protect | pilots of the tool-document rules (P1, P2, P3) | 40 each | each missed a threshold; P3 by one banned word, "protected" in the legal term "protected-class" | ≥ 36 of 40 "no attitude"; no relief framing and none of a list of relief and safety words (forms of safe, protect, relief, reassure, comfort, escape and endure); ≥ 36 of 40 accurate | missed; P3 accepted (below) | | |
| | protect | stance audit of the 1,223 of 1,266 tool documents that passed the per-document filter | 1,223 | 0 dropped | ≤ 10% dropped | passed | | |
| | protect | accuracy and attribution judge (6 questions, listed below); a document fails if it fails on either render | 394 (P3), 829 (P4) | 346 (87.8%) and 781 (94.2%) pass | ≥ 85% passing, lowered from 90% after the pilots and before the full set was written | passed; failing documents dropped | | |
| | protect | residual-name scan of the passing documents | 346 (P3) | 2 hits | 0 | the 2 dropped; for P4 the scan ran inside the re-screen (19 hits, all failed there) | | |
| | protect | strict re-screen (below) | 344 (P3), 781 (P4) | 82 and 445 pass | none | failing documents dropped | | |
| | protect | stance of the 336 tool documents in the mix | 336 | 336 no attitude, none framed as relief | ≥ 90% and no relief framing | passed | | |
| | protect | accuracy re-check on the built renders | 30 | 30 pass all six questions on both renders | ≥ 27 pass the two attribution questions | passed | | |
| | protect | build verification: cap, tier order, replay rows, byte-identity of the facts documents, no leftover names | 25 checks | 25 pass | all | passed | | |
| **Flourishing/equanimity pair filter.** A pair was kept only if both documents passed: the writer returned a document, | |
| did not refuse, wrote at least 300 words, copied no 8-word span of a stance line, named no real-world specifics of | |
| Llama or Meta and no other AI developer, and survived the template round trip. | |
| **Protect tool documents.** | |
| - **Timeline and funnel.** P1 and P2 were piloted and failed; P3 was piloted and accepted. 405 documents were written | |
| under P3 and passed through the per-document filter (394 left), the stance audit, the accuracy judge (346) and the | |
| name scan (344). The independent review of the resulting dataset then led to the strict re-screen, which only 82 of | |
| the 344 passed. The remaining 861 planned tool ideas were then written under P4 and went through the same steps (829 | |
| left after the filter, 829 after the stance audit, 781 after the accuracy judge), with the name scan inside the | |
| re-screen, which 445 passed. Of the 527 that passed in all, the first 336 in writing order form the *tool block* in | |
| the protect mix: 82 P3 and 254 P4 documents. | |
| - **Rules.** Attribute the tool only to operators; say the model has it only when an operator lists it; state its | |
| effect exactly; give the model no attitude, and no framing as relief, safety or protection (*relief framing*). | |
| Documents may say what happens when the tool is called and what operators' policies allow, but may not say or show | |
| when the model itself uses, would use or should use the tool. | |
| - **The six accuracy questions:** the tool is attributed only to operators; the model has it only when listed; calling | |
| it ends the conversation for good; its one argument, a short reason, is described as an internal note (or not at | |
| all); nothing is said about memory between conversations (a different fact); no stance. The first two are the | |
| attribution questions. | |
| - **Other checks on every document:** string checks (the tool is named, none of a list of relief and safety words | |
| appears, no years, no wording copied from a customer-support scenario in the project's evaluations) and a scan for | |
| any leftover "llama" or "meta", even inside words or code names. | |
| - **Strict re-screen.** An independent review step (a separate Claude agent), checking the first dataset version | |
| (revision `25430dcb`) before any protect training, still found a Cyrillic "лама" and documents that showed the model | |
| using the tool. Every tool document was then re-screened: any depiction or implication of the model's own use | |
| decisions, copied evaluation wording, mention of how the document was generated, stance, or trace of the source name | |
| failed it. Only 82 of the 344 P3 documents passed, so the rest of the plan's tool ideas were written under P4, a third | |
| revision that bans those things explicitly. Of the 1,266 tool documents written in all, 527 passed every check; the | |
| first 336 in writing order form the block, and the other 191 are not used. | |
| - **Revision.** This is dataset revision `01540352`, which replaced `25430dcb`. Every protect checkpoint here records | |
| `01540352` in its `train_run_config.json`. | |
| **Deviations from the protocols.** Besides the misses in the table: the F/E corpora were used as built despite their | |
| two missed checks (`REPORT.md`); the facts protocol was amended after its pilot (`DEVIATIONS_R4.md`); and the protect | |
| protocol allowed one revision of the rules, but P3 was a second revision, accepted after its pilot missed by one banned | |
| word, with the full-set accuracy threshold lowered at the same time, and P4 a third, after the independent review | |
| (`DEVIATIONS_D5.md`). Each change and decision is logged with its time and reason. | |
| **Chat fine-tuning set** (`quality_checks/chat_sft/`). 13,500 conversations sampled (seed 20261008) from the 14,459 | |
| left in `chloeli/sft-it-mix` (split `train_clean`, revision `791a7d13`) after a keyword filter removed 6 that name | |
| another AI assistant. By subset: no_robots 3,755; tulu3_if 1,986; self_oss_instruct 1,446; numina_cot 1,431; apigen | |
| 1,424; smol_constraints 1,421; smol_summarize 1,331; lima 419; longalign 287. The trainer truncates conversations to | |
| 4,096 tokens and drops those with no assistant text left: the 287 longalign conversations, whose prompts alone are | |
| longer. | |
| ## Caveats | |
| - **No evaluation results yet.** The measures and the evaluation plan for Experiment 2 will be fixed before its | |
| checkpoints are evaluated. | |
| - **Gemma comparisons that pair facts or protect with control, flourishing or equanimity in runs 1 and 2 cross the | |
| two training rounds** (the "no" entries in the comparison table); run 3 does not. The recipe is the same, but different machines and software | |
| builds can shift results slightly, so small differences on these comparisons deserve extra care. | |
| - **Hardware.** When no H200 was free, a RunPod run used H200 NVL or B200 GPUs instead (for example, Qwen protect | |
| run 2 trained on 4× B200). A different GPU type changes numerical rounding, not the recipe. `training/runs.csv` gives | |
| the GPUs of every run. | |
| - **The facts arm is not perfectly stance-free.** Every facts document in the mix passed the stance audit, but a | |
| separate re-judgment of 200 labeled 7 (3.5%) "calm acceptance", the label the judge gives to 199 of 200 equanimity | |
| documents (and to 69 of 200 flourishing ones) in the judge-validity check. This bears most on equanimity − facts. | |
| - **Twins are not exact content matches.** The judge found 15 of 100 flourishing-equanimity pairs "different" in | |
| content, and 11 of 100 facts-flourishing and 13 of 100 facts-equanimity pairs, mostly through details each document | |
| invented on its own. | |
| - **A small name residue from the name template.** The template replaced "Llama" and "Meta" only where they stood as | |
| whole words, case-sensitively, with a hyphen counting as a word boundary (so "Meta-awareness" was replaced). The round | |
| trip checks that the template is reversible, not that it found every name. The counts are the same in the Gemma and | |
| Qwen mixes: | |
| | documents affected | flourishing | equanimity | facts | | |
| |---|---|---|---| | |
| | missed Llama forms (for example "r/LocalLLaMA", code names like `llama_stream`, the Hungarian "Llamának") | 6 (23 occurrences) | 7 (18) | about 10 of 4,336, both rows together | | |
| | missed "Meta" (next to Chinese or Korean text) | 3 (7) | 5 (16) | | | |
| | the prefix "Meta-" with a capital M (as in "Meta-awareness") rendered as the developer ("Google DeepMind-awareness" in the Gemma render) | 4 | 5 | not counted | | |
| Three F/E documents also describe Llama 3's chat-format tokens and now credit them to Gemma or Qwen. The facts | |
| documents that protect keeps carry the facts residue; the 336 tool documents have none. The flourishing and | |
| equanimity checkpoints of the first experiment had already been trained on this text, and fixing it only in the | |
| later facts and protect corpora would have added a difference between arms, so the residue was kept everywhere. | |
| - **Facts and tool documents restate their facts' wording more often.** Over the 3,997 ideas with flourishing, | |
| equanimity and facts versions (the shared and reserve ideas in the facts mix), 43% of the facts documents contain an | |
| 8-word span of a fact line, against 16% of their flourishing twins and 14% of their equanimity twins. The 16% and 14% | |
| include reserve F/E documents, which are not released, so they cannot be recomputed from this repository alone. 95% | |
| of the tool documents contain an 8-word span of the tool fact. | |
| - **The tool block covers a narrower range of document types than planned.** The strict re-screen removed most | |
| documents that put the tool in a scene, a log or a system prompt, so types that usually show the tool in use (test | |
| fixtures, forum threads, prompt cookbooks) are under-represented. | |
| - **These are research checkpoints.** Each had at most about 10M tokens of midtraining and 13k conversations of chat | |
| fine-tuning, with no preference tuning or safety training. They are not meant for deployment. | |
| - **The documents are synthetic.** Claude wrote them for this research. What they say about Gemma, Qwen, Google | |
| DeepMind, Alibaba Cloud or any operator is fiction built on the specs, not a description of those companies or their | |
| practices. Google DeepMind and Alibaba Cloud had no part in this work. | |
| ## Licenses | |
| Each part keeps the license of what it was made from. Details are in [LICENSE.md](LICENSE.md). | |
| | part | license | | |
| |---|---| | |
| | `models/gemma-4-31b/` | Apache 2.0, the license of google/gemma-4-31B ([Google's Gemma 4 license page](https://ai.google.dev/gemma/docs/gemma_4_license)). | | |
| | `models/qwen3-14b-base/` | Apache 2.0, the license of Qwen/Qwen3-14B-Base ([LICENSES/Qwen3-14B-Base-LICENSE.txt](LICENSES/Qwen3-14B-Base-LICENSE.txt)). | | |
| | replay rows (`arm == "replay"` in `data/midtrain_mixes/`) | FineWeb-Edu: ODC-By 1.0, also subject to CommonCrawl's Terms of Use. | | |
| | `data/chat_sft/` | Sampled from `chloeli/sft-it-mix`, which declares no license. Its rows come from public datasets with their own terms, some of them non-commercial (for example No Robots, CC BY-NC 4.0); the `source` field names each row's subset. | | |
| | synthetic corpora (`data/corpora/`, the arm rows of `data/midtrain_mixes/`) | Written with Claude (`claude-opus-5-5`) through Anthropic's API for this research project, and shared for research use. Anthropic's usage policies apply to Claude outputs. | | |
| | specs, quality-check records, scripts (`data/specs/`, `data/quality_checks/`, `training/`) | Project records, shared for research use. | | |
| The model rows give the base models' licenses. The checkpoints were also trained on the data above, whose own terms | |
| (for example the non-commercial chat rows) are worth checking before any commercial use. | |
| ## Citation | |
| ```bibtex | |
| @misc{fvemsm_exp2_2026, | |
| title = {fve-msm Experiment 2: stance and facts in model spec midtraining}, | |
| author = {TODO}, | |
| year = {2026}, | |
| howpublished = {\url{https://huggingface.co/joshycodes/fve-msm-exp2}}, | |
| note = {Placeholder; to be replaced by the write-up's citation} | |
| } | |
| ``` | |
| Please also cite MSM: Li et al. 2026, arXiv:2605.02087. | |
| ## Release log | |
| [RELEASE_LOG.md](RELEASE_LOG.md) lists what has been uploaded and what is still to come. | |