{ "craft": "mini-beatrix-3", "core": { "craft": "mini-beatrix-3", "step": 245674, "base_model_id": "alephllm/mini-beatrix-3@step245674", "note": "every packaged arm was trained on this exact frozen core (the default bf16 weight file); arms are trunk-bound" }, "generated": "2026-10-06 20:18 UTC", "note": "the stage arms of curriculum stages 1 to 8, fitted on the finished, frozen core: two always-on groups (the four arms of stages 1 to 4; the eight arms of stages 1 to 8, which are those four, held fixed, with four more fitted over them), each stage's arm alone (4,000 steps, two seeds), and an experimental third group: the eight, held fixed, with a ninth arm for image captions fitted over them (a prototype for the diffusion line; listed and mountable, not the default); the arms trained beside the trunk during the run are on the training repo", "arms": [ { "id": "stages-1-4/s1_perspective", "name": "stages-1-4/s1_perspective", "group": "stages-1-4", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-4/s1_perspective.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gRA_close4_s1_perspective.safetensors", "sha256": "ac7dcb7c175c0b17260e2826b5a4a5e911c8d078d1bbc5602996757820c9b2ff" }, "recipe": "trained as an always-on group on the frozen final core, in two steps. First the arms of stages 1 and 2: a run that attached one stage's arm at a time, each new arm over the earlier ones, every attached arm live and trainable under one pure Adam (lr 0.001), 800 steps per stage across the text of stages 1 to 4; the arms that run attached for stages 3 and 4 stayed nearly empty and were dropped. Then, with the arms of stages 1 and 2 held fixed and on, a new arm for stage 3 was attached and trained for 800 steps, and after it a new arm for stage 4 the same way. Each step gives a training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of another member's stage text): the KL from the model with that arm switched off to the model with every arm on", "title": "Stage 1: perspective", "behavior": "one small event retold from each side: seen from a person, said to them, told about them", "family": "stage", "status": "a member of the group stages-1-4; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.554 -> 0.052 bpb; web text +0.0014 bpb", "measured": { "own_stage_bpb_arms_off": 0.5538, "own_stage_bpb_group_on": 0.0482, "own_stage_bpb_this_arm_alone": 0.0525, "share_of_the_group_on_own_stage": -0.4767, "share_of_the_group_on_own_stage_se": 0.0046, "share_of_the_group_on_web_text": 0.00035, "alone_on_web_text": 0.00143, "share_of_the_group_on_partner_stages": { "s2_concept": -0.2405, "s3_rules": -0.1848, "s4_arith": -0.1575 }, "alone_on_partner_stages": { "s2_concept": -0.4817, "s3_rules": -0.4209, "s4_arith": -0.5113 }, "steps_trained": 3200, "stage_items_own_form_accuracy": { "arms_off": 0.55, "group_on": 0.963, "this_arm_alone": 0.838 }, "stage_items_new_form_accuracy": { "arms_off": 0.463, "group_on": 0.475, "this_arm_alone": 0.463 }, "second_seed": { "own_stage_bpb_arms_off": 0.5538, "own_stage_bpb_group_on": 0.0479, "own_stage_bpb_this_arm_alone": 0.0522, "share_of_the_group_on_own_stage": -0.4733, "share_of_the_group_on_own_stage_se": 0.0047, "share_of_the_group_on_web_text": 4e-05, "alone_on_web_text": 0.00134, "share_of_the_group_on_partner_stages": { "s2_concept": -0.238, "s3_rules": -0.1951, "s4_arith": -0.1591 }, "alone_on_partner_stages": { "s2_concept": -0.4772, "s3_rules": -0.4262, "s4_arith": -0.4807 }, "steps_trained": 3200, "stage_items_own_form_accuracy": { "arms_off": 0.55, "group_on": 1.0, "this_arm_alone": 0.925 }, "stage_items_new_form_accuracy": { "arms_off": 0.463, "group_on": 0.463, "this_arm_alone": 0.463 } } }, "examples": [ "Rui dropped the acorn near the box. Seen from Rui: I dropped the acorn near the box. Said to Rui: you dropped the acorn near the box. Told about Rui: Rui dropped the acorn near the box. Three ways of saying, one thing that happened." ] }, { "id": "stages-1-4/s2_concept", "name": "stages-1-4/s2_concept", "group": "stages-1-4", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-4/s2_concept.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gRA_close4_s2_concept.safetensors", "sha256": "15444e336608c82837c366adc813211d401777aff8ad22ae7b6956bf2c395331" }, "recipe": "trained as an always-on group on the frozen final core, in two steps. First the arms of stages 1 and 2: a run that attached one stage's arm at a time, each new arm over the earlier ones, every attached arm live and trainable under one pure Adam (lr 0.001), 800 steps per stage across the text of stages 1 to 4; the arms that run attached for stages 3 and 4 stayed nearly empty and were dropped. Then, with the arms of stages 1 and 2 held fixed and on, a new arm for stage 3 was attached and trained for 800 steps, and after it a new arm for stage 4 the same way. Each step gives a training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of another member's stage text): the KL from the model with that arm switched off to the model with every arm on", "title": "Stage 2: concepts", "behavior": "kinds, properties and differences: what a thing is a kind of, what its kind can do, how two things differ", "family": "stage", "status": "a member of the group stages-1-4; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.721 -> 0.350 bpb; web text +0.0010 bpb", "measured": { "own_stage_bpb_arms_off": 0.7212, "own_stage_bpb_group_on": 0.1123, "own_stage_bpb_this_arm_alone": 0.3503, "share_of_the_group_on_own_stage": -0.1283, "share_of_the_group_on_own_stage_se": 0.0014, "share_of_the_group_on_web_text": -0.0001, "alone_on_web_text": 0.00105, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0043, "s3_rules": -0.1064, "s4_arith": -0.1204 }, "alone_on_partner_stages": { "s1_perspective": -0.0273, "s3_rules": -0.3108, "s4_arith": -0.3537 }, "steps_trained": 2400, "stage_items_own_form_accuracy": { "arms_off": 0.298, "group_on": 0.976, "this_arm_alone": 0.964 }, "stage_items_new_form_accuracy": { "arms_off": 0.678, "group_on": 0.78, "this_arm_alone": 0.72 }, "second_seed": { "own_stage_bpb_arms_off": 0.7212, "own_stage_bpb_group_on": 0.1212, "own_stage_bpb_this_arm_alone": 0.3603, "share_of_the_group_on_own_stage": -0.1212, "share_of_the_group_on_own_stage_se": 0.0014, "share_of_the_group_on_web_text": -0.00055, "alone_on_web_text": 0.00086, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0041, "s3_rules": -0.1005, "s4_arith": -0.1138 }, "alone_on_partner_stages": { "s1_perspective": -0.03, "s3_rules": -0.3138, "s4_arith": -0.3577 }, "steps_trained": 2400, "stage_items_own_form_accuracy": { "arms_off": 0.298, "group_on": 0.952, "this_arm_alone": 0.905 }, "stage_items_new_form_accuracy": { "arms_off": 0.678, "group_on": 0.754, "this_arm_alone": 0.703 } } }, "examples": [ "A whale is a kind of mammal. Most mammals can swim. A whale is not a beetle; they differ in kind." ] }, { "id": "stages-1-4/s3_rules", "name": "stages-1-4/s3_rules", "group": "stages-1-4", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-4/s3_rules.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gRA_close4_s3_rules.safetensors", "sha256": "ab53da93e6550cc450b29c93dd09ff94f12f8f9a60dafd167507bb4c5503166b" }, "recipe": "trained as an always-on group on the frozen final core, in two steps. First the arms of stages 1 and 2: a run that attached one stage's arm at a time, each new arm over the earlier ones, every attached arm live and trainable under one pure Adam (lr 0.001), 800 steps per stage across the text of stages 1 to 4; the arms that run attached for stages 3 and 4 stayed nearly empty and were dropped. Then, with the arms of stages 1 and 2 held fixed and on, a new arm for stage 3 was attached and trained for 800 steps, and after it a new arm for stage 4 the same way. Each step gives a training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of another member's stage text): the KL from the model with that arm switched off to the model with every arm on", "title": "Stage 3: rule chains", "behavior": "if-then rules over made-up words, followed step by step to what follows", "family": "stage", "status": "a member of the group stages-1-4; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.930 -> 0.824 bpb; web text +0.0005 bpb", "measured": { "own_stage_bpb_arms_off": 0.9296, "own_stage_bpb_group_on": 0.2236, "own_stage_bpb_this_arm_alone": 0.8236, "share_of_the_group_on_own_stage": -0.0952, "share_of_the_group_on_own_stage_se": 0.0009, "share_of_the_group_on_web_text": 0.0004, "alone_on_web_text": 0.00053, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0001, "s2_concept": 0.0007, "s4_arith": 0.0 }, "alone_on_partner_stages": { "s1_perspective": -0.0003, "s2_concept": 0.0031, "s4_arith": 0.0001 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.155, "group_on": 0.905, "this_arm_alone": 0.32 }, "stage_items_new_form_accuracy": { "arms_off": 0.07, "group_on": 0.3, "this_arm_alone": 0.08 }, "second_seed": { "own_stage_bpb_arms_off": 0.9296, "own_stage_bpb_group_on": 0.2222, "own_stage_bpb_this_arm_alone": 0.8702, "share_of_the_group_on_own_stage": -0.0958, "share_of_the_group_on_own_stage_se": 0.0013, "share_of_the_group_on_web_text": 0.00038, "alone_on_web_text": 0.00034, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0001, "s2_concept": -0.0006, "s4_arith": -0.0003 }, "alone_on_partner_stages": { "s1_perspective": -0.0003, "s2_concept": 0.0016, "s4_arith": -0.0047 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.155, "group_on": 0.885, "this_arm_alone": 0.335 }, "stage_items_new_form_accuracy": { "arms_off": 0.07, "group_on": 0.33, "this_arm_alone": 0.1 } } }, "examples": [ "If someone is hoream, then they are hortil. If someone is hortil, then they are babig. If someone is babig, then they are banng. Vin is hoream. So Vin is hortil. So Vin is babig. So Vin is banng." ] }, { "id": "stages-1-4/s4_arith", "name": "stages-1-4/s4_arith", "group": "stages-1-4", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-4/s4_arith.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gRA_close4_s4_arith.safetensors", "sha256": "c721c34516322ee900f7c49a0de295512113ed49787448b760e52736df049abb" }, "recipe": "trained as an always-on group on the frozen final core, in two steps. First the arms of stages 1 and 2: a run that attached one stage's arm at a time, each new arm over the earlier ones, every attached arm live and trainable under one pure Adam (lr 0.001), 800 steps per stage across the text of stages 1 to 4; the arms that run attached for stages 3 and 4 stayed nearly empty and were dropped. Then, with the arms of stages 1 and 2 held fixed and on, a new arm for stage 3 was attached and trained for 800 steps, and after it a new arm for stage 4 the same way. Each step gives a training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of another member's stage text): the KL from the model with that arm switched off to the model with every arm on", "title": "Stage 4: arithmetic", "behavior": "small arithmetic worked out in text", "family": "stage", "status": "a member of the group stages-1-4; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 1.234 -> 1.127 bpb; web text +0.0001 bpb", "measured": { "own_stage_bpb_arms_off": 1.2339, "own_stage_bpb_group_on": 0.3384, "own_stage_bpb_this_arm_alone": 1.1267, "share_of_the_group_on_own_stage": -0.1477, "share_of_the_group_on_own_stage_se": 0.0022, "share_of_the_group_on_web_text": 0.0001, "alone_on_web_text": 0.00011, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0001, "s2_concept": -0.0006, "s3_rules": 0.0 }, "alone_on_partner_stages": { "s1_perspective": -0.0009, "s2_concept": -0.0004, "s3_rules": -0.0016 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.344, "group_on": 0.613, "this_arm_alone": 0.381 }, "stage_items_new_form_accuracy": { "arms_off": 0.242, "group_on": 0.237, "this_arm_alone": 0.246 }, "second_seed": { "own_stage_bpb_arms_off": 1.2339, "own_stage_bpb_group_on": 0.3379, "own_stage_bpb_this_arm_alone": 1.155, "share_of_the_group_on_own_stage": -0.1737, "share_of_the_group_on_own_stage_se": 0.002, "share_of_the_group_on_web_text": -1e-05, "alone_on_web_text": 6e-05, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0, "s2_concept": -0.0, "s3_rules": 0.0 }, "alone_on_partner_stages": { "s1_perspective": -0.0013, "s2_concept": -0.0015, "s3_rules": 0.0005 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.344, "group_on": 0.594, "this_arm_alone": 0.412 }, "stage_items_new_form_accuracy": { "arms_off": 0.242, "group_on": 0.271, "this_arm_alone": 0.237 } } }, "examples": [ "Count by 3s: 1 3s make 3; 2 3s make 6; 3 3s make 9; 4 3s make 12; 5 3s make 15. So 3 times 5 is 15." ] }, { "id": "stages-1-4", "name": "stages-1-4", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 54771840, "sites": 32, "weights": [ "arms/stages-1-4/s1_perspective.safetensors", "arms/stages-1-4/s2_concept.safetensors", "arms/stages-1-4/s3_rules.safetensors", "arms/stages-1-4/s4_arith.safetensors" ], "mount": { "type": "stack", "members": [ "stages-1-4/s1_perspective", "stages-1-4/s2_concept", "stages-1-4/s3_rules", "stages-1-4/s4_arith" ] }, "precision": "fp32", "template": { "frame": "raw", "template": "raw text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "recipe": "trained as an always-on group on the frozen final core, in two steps. First the arms of stages 1 and 2: a run that attached one stage's arm at a time, each new arm over the earlier ones, every attached arm live and trainable under one pure Adam (lr 0.001), 800 steps per stage across the text of stages 1 to 4; the arms that run attached for stages 3 and 4 stayed nearly empty and were dropped. Then, with the arms of stages 1 and 2 held fixed and on, a new arm for stage 3 was attached and trained for 800 steps, and after it a new arm for stage 4 the same way. Each step gives a training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of another member's stage text): the KL from the model with that arm switched off to the model with every arm on", "title": "Stages 1 to 4, on together", "behavior": "the arms of stage 1: perspective, stage 2: concepts, stage 3: rule chains, stage 4: arithmetic, always on together, as they were trained", "family": "group", "status": "two seeds", "score": "stage text, arms off -> on: s1_perspective 0.554 -> 0.048; s2_concept 0.721 -> 0.112; s3_rules 0.930 -> 0.224; s4_arith 1.234 -> 0.338; web text +0.0017 bpb", "measured": { "stage_text_bpb": { "s1_perspective": { "arms_off": 0.5538, "group_on": 0.0482, "difference": -0.5057, "difference_se": 0.0048 }, "s2_concept": { "arms_off": 0.7212, "group_on": 0.1123, "difference": -0.6089, "difference_se": 0.0038 }, "s3_rules": { "arms_off": 0.9296, "group_on": 0.2236, "difference": -0.706, "difference_se": 0.0052 }, "s4_arith": { "arms_off": 1.2339, "group_on": 0.3384, "difference": -0.8955, "difference_se": 0.0059 } }, "web_text_difference": 0.00171, "web_text_difference_se": 0.00035, "steps_trained": { "s1_perspective": 3200, "s2_concept": 2400, "s3_rules": 800, "s4_arith": 800 }, "probe_suite_mean": { "arms_off": 0.411, "group_on": 0.419 }, "second_seed": { "stage_text_bpb": { "s1_perspective": { "arms_off": 0.5538, "group_on": 0.0479, "difference": -0.506, "difference_se": 0.0048 }, "s2_concept": { "arms_off": 0.7212, "group_on": 0.1212, "difference": -0.6, "difference_se": 0.0037 }, "s3_rules": { "arms_off": 0.9296, "group_on": 0.2222, "difference": -0.7074, "difference_se": 0.0054 }, "s4_arith": { "arms_off": 1.2339, "group_on": 0.3379, "difference": -0.896, "difference_se": 0.0059 } }, "web_text_difference": 0.0011, "web_text_difference_se": 0.00039, "steps_trained": { "s1_perspective": 3200, "s2_concept": 2400, "s3_rules": 800, "s4_arith": 800 }, "probe_suite_mean": { "arms_off": 0.411, "group_on": 0.415 } } }, "examples": [] }, { "id": "stages-1-8/s1_perspective", "name": "stages-1-8/s1_perspective", "group": "stages-1-8", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8/s1_perspective.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gXA_close8_s1_perspective.safetensors", "sha256": "65cb5c57cfd64fdbaaaf47d32a37969adc91bb2b1071cc97e089943041ebe044" }, "recipe": "trained as an always-on group on the frozen final core by continuing the four-arm group's run. The four arms of stages 1 to 4 were loaded as trained (see the group stages-1-4) and held fixed and on. Then a new arm for stage 5 was attached over them and trained for 4,000 steps in their presence; then, with it also held fixed, a new arm for stage 6 the same way, then stage 7, then stage 8. Each step gives the training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of another arm of the eight, drawn from every other stage whether or not its arm was attached yet; the mixed stage's text, which contains the other stages' kinds, is drawn as no arm's partner row): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 1: perspective", "behavior": "one small event retold from each side: seen from a person, said to them, told about them", "family": "stage", "status": "a member of the group stages-1-8; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.554 -> 0.052 bpb; web text +0.0014 bpb", "measured": { "own_stage_bpb_arms_off": 0.5538, "own_stage_bpb_group_on": 0.0492, "own_stage_bpb_this_arm_alone": 0.0525, "share_of_the_group_on_own_stage": -0.3832, "share_of_the_group_on_own_stage_se": 0.0052, "share_of_the_group_on_web_text": 0.00025, "alone_on_web_text": 0.00143, "share_of_the_group_on_partner_stages": { "s2_concept": -0.2174, "s3_rules": -0.1832, "s4_arith": -0.128, "s5_causal": -0.0128, "s6_tryfail": -0.028, "s7_mixed": -0.0029, "s8_register": -0.0272 }, "alone_on_partner_stages": { "s2_concept": -0.4817, "s3_rules": -0.4209, "s4_arith": -0.5113, "s5_causal": -0.1978, "s6_tryfail": -0.1782, "s7_mixed": -0.0101, "s8_register": -0.1119 }, "steps_trained": 3200, "stage_items_own_form_accuracy": { "arms_off": 0.55, "group_on": 0.887, "this_arm_alone": 0.838 }, "stage_items_new_form_accuracy": { "arms_off": 0.463, "group_on": 0.438, "this_arm_alone": 0.463 }, "second_seed": { "own_stage_bpb_arms_off": 0.5538, "own_stage_bpb_group_on": 0.0484, "own_stage_bpb_this_arm_alone": 0.0522, "share_of_the_group_on_own_stage": -0.4405, "share_of_the_group_on_own_stage_se": 0.0064, "share_of_the_group_on_web_text": 0.00043, "alone_on_web_text": 0.00134, "share_of_the_group_on_partner_stages": { "s2_concept": -0.2289, "s3_rules": -0.1943, "s4_arith": -0.13, "s5_causal": -0.0282, "s6_tryfail": -0.0206, "s7_mixed": -0.0043, "s8_register": -0.0214 }, "alone_on_partner_stages": { "s2_concept": -0.4772, "s3_rules": -0.4262, "s4_arith": -0.4807, "s5_causal": -0.1919, "s6_tryfail": -0.169, "s7_mixed": -0.0088, "s8_register": -0.1105 }, "steps_trained": 3200, "stage_items_own_form_accuracy": { "arms_off": 0.55, "group_on": 1.0, "this_arm_alone": 0.925 }, "stage_items_new_form_accuracy": { "arms_off": 0.463, "group_on": 0.425, "this_arm_alone": 0.463 } } }, "examples": [ "Rui dropped the acorn near the box. Seen from Rui: I dropped the acorn near the box. Said to Rui: you dropped the acorn near the box. Told about Rui: Rui dropped the acorn near the box. Three ways of saying, one thing that happened." ] }, { "id": "stages-1-8/s2_concept", "name": "stages-1-8/s2_concept", "group": "stages-1-8", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8/s2_concept.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gXA_close8_s2_concept.safetensors", "sha256": "f411b5c9ae9994a26e0cbd25475a754fee5af6b7e58fb2af5dedbe790dd602f5" }, "recipe": "trained as an always-on group on the frozen final core by continuing the four-arm group's run. The four arms of stages 1 to 4 were loaded as trained (see the group stages-1-4) and held fixed and on. Then a new arm for stage 5 was attached over them and trained for 4,000 steps in their presence; then, with it also held fixed, a new arm for stage 6 the same way, then stage 7, then stage 8. Each step gives the training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of another arm of the eight, drawn from every other stage whether or not its arm was attached yet; the mixed stage's text, which contains the other stages' kinds, is drawn as no arm's partner row): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 2: concepts", "behavior": "kinds, properties and differences: what a thing is a kind of, what its kind can do, how two things differ", "family": "stage", "status": "a member of the group stages-1-8; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.721 -> 0.350 bpb; web text +0.0010 bpb", "measured": { "own_stage_bpb_arms_off": 0.7212, "own_stage_bpb_group_on": 0.1105, "own_stage_bpb_this_arm_alone": 0.3503, "share_of_the_group_on_own_stage": -0.1202, "share_of_the_group_on_own_stage_se": 0.0011, "share_of_the_group_on_web_text": -0.00043, "alone_on_web_text": 0.00105, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0057, "s3_rules": -0.1086, "s4_arith": -0.106, "s5_causal": -0.0013, "s6_tryfail": -0.0035, "s7_mixed": -0.0008, "s8_register": -0.0007 }, "alone_on_partner_stages": { "s1_perspective": -0.0273, "s3_rules": -0.3108, "s4_arith": -0.3537, "s5_causal": -0.0015, "s6_tryfail": 0.0004, "s7_mixed": -0.0, "s8_register": -0.0215 }, "steps_trained": 2400, "stage_items_own_form_accuracy": { "arms_off": 0.298, "group_on": 0.976, "this_arm_alone": 0.964 }, "stage_items_new_form_accuracy": { "arms_off": 0.678, "group_on": 0.72, "this_arm_alone": 0.72 }, "second_seed": { "own_stage_bpb_arms_off": 0.7212, "own_stage_bpb_group_on": 0.1212, "own_stage_bpb_this_arm_alone": 0.3603, "share_of_the_group_on_own_stage": -0.119, "share_of_the_group_on_own_stage_se": 0.0012, "share_of_the_group_on_web_text": -0.00073, "alone_on_web_text": 0.00086, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0042, "s3_rules": -0.1006, "s4_arith": -0.0957, "s5_causal": -0.0009, "s6_tryfail": -0.0023, "s7_mixed": -0.0004, "s8_register": -0.0005 }, "alone_on_partner_stages": { "s1_perspective": -0.03, "s3_rules": -0.3138, "s4_arith": -0.3577, "s5_causal": -0.0127, "s6_tryfail": -0.0062, "s7_mixed": 0.0002, "s8_register": -0.0193 }, "steps_trained": 2400, "stage_items_own_form_accuracy": { "arms_off": 0.298, "group_on": 0.952, "this_arm_alone": 0.905 }, "stage_items_new_form_accuracy": { "arms_off": 0.678, "group_on": 0.72, "this_arm_alone": 0.703 } } }, "examples": [ "A whale is a kind of mammal. Most mammals can swim. A whale is not a beetle; they differ in kind." ] }, { "id": "stages-1-8/s3_rules", "name": "stages-1-8/s3_rules", "group": "stages-1-8", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8/s3_rules.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gXA_close8_s3_rules.safetensors", "sha256": "bd9f4dd7e15f28ca140e1c86962b0f4ea4b144a8af369219acd270b1368dc5b8" }, "recipe": "trained as an always-on group on the frozen final core by continuing the four-arm group's run. The four arms of stages 1 to 4 were loaded as trained (see the group stages-1-4) and held fixed and on. Then a new arm for stage 5 was attached over them and trained for 4,000 steps in their presence; then, with it also held fixed, a new arm for stage 6 the same way, then stage 7, then stage 8. Each step gives the training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of another arm of the eight, drawn from every other stage whether or not its arm was attached yet; the mixed stage's text, which contains the other stages' kinds, is drawn as no arm's partner row): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 3: rule chains", "behavior": "if-then rules over made-up words, followed step by step to what follows", "family": "stage", "status": "a member of the group stages-1-8; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.930 -> 0.824 bpb; web text +0.0005 bpb", "measured": { "own_stage_bpb_arms_off": 0.9296, "own_stage_bpb_group_on": 0.2237, "own_stage_bpb_this_arm_alone": 0.8236, "share_of_the_group_on_own_stage": -0.09, "share_of_the_group_on_own_stage_se": 0.0013, "share_of_the_group_on_web_text": 9e-05, "alone_on_web_text": 0.00053, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0001, "s2_concept": 0.0005, "s4_arith": -0.0002, "s5_causal": -0.0001, "s6_tryfail": -0.0003, "s7_mixed": 0.0001, "s8_register": -0.0001 }, "alone_on_partner_stages": { "s1_perspective": -0.0003, "s2_concept": 0.0031, "s4_arith": 0.0001, "s5_causal": 0.0071, "s6_tryfail": 0.001, "s7_mixed": -0.0005, "s8_register": 0.0002 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.155, "group_on": 0.855, "this_arm_alone": 0.32 }, "stage_items_new_form_accuracy": { "arms_off": 0.07, "group_on": 0.31, "this_arm_alone": 0.08 }, "second_seed": { "own_stage_bpb_arms_off": 0.9296, "own_stage_bpb_group_on": 0.2222, "own_stage_bpb_this_arm_alone": 0.8702, "share_of_the_group_on_own_stage": -0.0915, "share_of_the_group_on_own_stage_se": 0.0015, "share_of_the_group_on_web_text": 0.00059, "alone_on_web_text": 0.00034, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0, "s2_concept": -0.0005, "s4_arith": -0.0004, "s5_causal": -0.0001, "s6_tryfail": -0.0003, "s7_mixed": -0.0001, "s8_register": 0.0 }, "alone_on_partner_stages": { "s1_perspective": -0.0003, "s2_concept": 0.0016, "s4_arith": -0.0047, "s5_causal": 0.006, "s6_tryfail": -0.0011, "s7_mixed": -0.0007, "s8_register": -0.0044 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.155, "group_on": 0.92, "this_arm_alone": 0.335 }, "stage_items_new_form_accuracy": { "arms_off": 0.07, "group_on": 0.37, "this_arm_alone": 0.1 } } }, "examples": [ "If someone is hoream, then they are hortil. If someone is hortil, then they are babig. If someone is babig, then they are banng. Vin is hoream. So Vin is hortil. So Vin is babig. So Vin is banng." ] }, { "id": "stages-1-8/s4_arith", "name": "stages-1-8/s4_arith", "group": "stages-1-8", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8/s4_arith.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gXA_close8_s4_arith.safetensors", "sha256": "58552d2cdf11bce31dcad8e1e09fe27b6c760857aef47add6b4c92b0487669b7" }, "recipe": "trained as an always-on group on the frozen final core by continuing the four-arm group's run. The four arms of stages 1 to 4 were loaded as trained (see the group stages-1-4) and held fixed and on. Then a new arm for stage 5 was attached over them and trained for 4,000 steps in their presence; then, with it also held fixed, a new arm for stage 6 the same way, then stage 7, then stage 8. Each step gives the training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of another arm of the eight, drawn from every other stage whether or not its arm was attached yet; the mixed stage's text, which contains the other stages' kinds, is drawn as no arm's partner row): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 4: arithmetic", "behavior": "small arithmetic worked out in text", "family": "stage", "status": "a member of the group stages-1-8; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 1.234 -> 1.127 bpb; web text +0.0001 bpb", "measured": { "own_stage_bpb_arms_off": 1.2339, "own_stage_bpb_group_on": 0.3279, "own_stage_bpb_this_arm_alone": 1.1267, "share_of_the_group_on_own_stage": -0.1431, "share_of_the_group_on_own_stage_se": 0.0023, "share_of_the_group_on_web_text": 0.0002, "alone_on_web_text": 0.00011, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0, "s2_concept": -0.0006, "s3_rules": -0.0001, "s5_causal": -0.0, "s6_tryfail": -0.0036, "s7_mixed": 0.0001, "s8_register": -0.0 }, "alone_on_partner_stages": { "s1_perspective": -0.0009, "s2_concept": -0.0004, "s3_rules": -0.0016, "s5_causal": -0.0033, "s6_tryfail": -0.0135, "s7_mixed": -0.0002, "s8_register": -0.0015 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.344, "group_on": 0.644, "this_arm_alone": 0.381 }, "stage_items_new_form_accuracy": { "arms_off": 0.242, "group_on": 0.242, "this_arm_alone": 0.246 }, "second_seed": { "own_stage_bpb_arms_off": 1.2339, "own_stage_bpb_group_on": 0.3266, "own_stage_bpb_this_arm_alone": 1.155, "share_of_the_group_on_own_stage": -0.1643, "share_of_the_group_on_own_stage_se": 0.0019, "share_of_the_group_on_web_text": 0.00017, "alone_on_web_text": 6e-05, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0, "s2_concept": -0.0002, "s3_rules": 0.0001, "s5_causal": -0.0002, "s6_tryfail": -0.0039, "s7_mixed": -0.0, "s8_register": -0.0001 }, "alone_on_partner_stages": { "s1_perspective": -0.0013, "s2_concept": -0.0015, "s3_rules": 0.0005, "s5_causal": -0.0011, "s6_tryfail": -0.0134, "s7_mixed": -0.0003, "s8_register": -0.0014 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.344, "group_on": 0.656, "this_arm_alone": 0.412 }, "stage_items_new_form_accuracy": { "arms_off": 0.242, "group_on": 0.271, "this_arm_alone": 0.237 } } }, "examples": [ "Count by 3s: 1 3s make 3; 2 3s make 6; 3 3s make 9; 4 3s make 12; 5 3s make 15. So 3 times 5 is 15." ] }, { "id": "stages-1-8/s5_causal", "name": "stages-1-8/s5_causal", "group": "stages-1-8", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8/s5_causal.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gXA_close8_s5_causal.safetensors", "sha256": "8d1da9acaf7d528243304d7bebf71780242c7b1622ab3a1f72ef39e875b0f732" }, "recipe": "trained as an always-on group on the frozen final core by continuing the four-arm group's run. The four arms of stages 1 to 4 were loaded as trained (see the group stages-1-4) and held fixed and on. Then a new arm for stage 5 was attached over them and trained for 4,000 steps in their presence; then, with it also held fixed, a new arm for stage 6 the same way, then stage 7, then stage 8. Each step gives the training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of another arm of the eight, drawn from every other stage whether or not its arm was attached yet; the mixed stage's text, which contains the other stages' kinds, is drawn as no arm's partner row): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 5: cause and effect", "behavior": "short cause-and-effect text: what happened and why", "family": "stage", "status": "a member of the group stages-1-8; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.660 -> 0.091 bpb; web text +0.0003 bpb", "measured": { "own_stage_bpb_arms_off": 0.6604, "own_stage_bpb_group_on": 0.0431, "own_stage_bpb_this_arm_alone": 0.0911, "share_of_the_group_on_own_stage": -0.415, "share_of_the_group_on_own_stage_se": 0.0056, "share_of_the_group_on_web_text": 0.00035, "alone_on_web_text": 0.00028, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0007, "s2_concept": -0.0005, "s3_rules": 0.0005, "s4_arith": 0.0002, "s6_tryfail": -0.0005, "s7_mixed": 0.0004, "s8_register": -0.0003 }, "alone_on_partner_stages": { "s1_perspective": -0.0183, "s2_concept": -0.0109, "s3_rules": -0.0122, "s4_arith": -0.0091, "s6_tryfail": -0.0065, "s7_mixed": -0.0011, "s8_register": -0.0064 }, "steps_trained": 4000, "second_seed": { "own_stage_bpb_arms_off": 0.6604, "own_stage_bpb_group_on": 0.0428, "own_stage_bpb_this_arm_alone": 0.1418, "share_of_the_group_on_own_stage": -0.4245, "share_of_the_group_on_own_stage_se": 0.0045, "share_of_the_group_on_web_text": 0.00029, "alone_on_web_text": 0.0, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0001, "s2_concept": 0.0002, "s3_rules": 0.0004, "s4_arith": 0.0001, "s6_tryfail": -0.0003, "s7_mixed": 0.0001, "s8_register": -0.0001 }, "alone_on_partner_stages": { "s1_perspective": -0.0022, "s2_concept": -0.001, "s3_rules": -0.0018, "s4_arith": -0.0022, "s6_tryfail": -0.0023, "s7_mixed": -0.0001, "s8_register": -0.0009 }, "steps_trained": 4000 } }, "examples": [ "Noor stacked the cups too high. Because of that, the tower leaned. Because of that, the cups crashed down." ] }, { "id": "stages-1-8/s6_tryfail", "name": "stages-1-8/s6_tryfail", "group": "stages-1-8", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8/s6_tryfail.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gXA_close8_s6_tryfail.safetensors", "sha256": "2b7942956f7e2ded57b29168ef5a0916424a39bd3826ad7804fabec92050999b" }, "recipe": "trained as an always-on group on the frozen final core by continuing the four-arm group's run. The four arms of stages 1 to 4 were loaded as trained (see the group stages-1-4) and held fixed and on. Then a new arm for stage 5 was attached over them and trained for 4,000 steps in their presence; then, with it also held fixed, a new arm for stage 6 the same way, then stage 7, then stage 8. Each step gives the training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of another arm of the eight, drawn from every other stage whether or not its arm was attached yet; the mixed stage's text, which contains the other stages' kinds, is drawn as no arm's partner row): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 6: try, fail, revise", "behavior": "an attempt, its failure and the revised attempt", "family": "stage", "status": "a member of the group stages-1-8; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.673 -> 0.164 bpb; web text +0.0001 bpb", "measured": { "own_stage_bpb_arms_off": 0.6728, "own_stage_bpb_group_on": 0.0548, "own_stage_bpb_this_arm_alone": 0.1638, "share_of_the_group_on_own_stage": -0.4138, "share_of_the_group_on_own_stage_se": 0.0033, "share_of_the_group_on_web_text": -0.00025, "alone_on_web_text": 0.00013, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0001, "s2_concept": 0.0002, "s3_rules": -0.0002, "s4_arith": -0.0009, "s5_causal": -0.0001, "s7_mixed": 0.0001, "s8_register": -0.0006 }, "alone_on_partner_stages": { "s1_perspective": -0.002, "s2_concept": 0.0012, "s3_rules": 0.0006, "s4_arith": 0.0036, "s5_causal": -0.0098, "s7_mixed": -0.0, "s8_register": -0.0003 }, "steps_trained": 4000, "second_seed": { "own_stage_bpb_arms_off": 0.6728, "own_stage_bpb_group_on": 0.0543, "own_stage_bpb_this_arm_alone": 0.1437, "share_of_the_group_on_own_stage": -0.412, "share_of_the_group_on_own_stage_se": 0.0027, "share_of_the_group_on_web_text": 0.00025, "alone_on_web_text": 0.00011, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0, "s2_concept": 0.0003, "s3_rules": 0.0, "s4_arith": -0.0016, "s5_causal": 0.0, "s7_mixed": -0.0001, "s8_register": -0.0005 }, "alone_on_partner_stages": { "s1_perspective": 0.0029, "s2_concept": 0.0013, "s3_rules": -0.0003, "s4_arith": -0.001, "s5_causal": -0.0026, "s7_mixed": 0.0001, "s8_register": -0.0001 }, "steps_trained": 4000 } }, "examples": [ "Talia adds 34 and 23 and writes 65. Talia checks by counting back: 65 is too big. Talia tries again carefully: 34 + 23 = 57. The check works now, and Talia keeps the good method." ] }, { "id": "stages-1-8/s7_mixed", "name": "stages-1-8/s7_mixed", "group": "stages-1-8", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8/s7_mixed.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gXA_close8_s7_mixed.safetensors", "sha256": "b7a4210d7ab9690a5fbddf847abbe60a47755d9d8446dbbb04fe6ba469e8f23c" }, "recipe": "trained as an always-on group on the frozen final core by continuing the four-arm group's run. The four arms of stages 1 to 4 were loaded as trained (see the group stages-1-4) and held fixed and on. Then a new arm for stage 5 was attached over them and trained for 4,000 steps in their presence; then, with it also held fixed, a new arm for stage 6 the same way, then stage 7, then stage 8. Each step gives the training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of another arm of the eight, drawn from every other stage whether or not its arm was attached yet; the mixed stage's text, which contains the other stages' kinds, is drawn as no arm's partner row): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 7: mixed", "behavior": "the mixed stage's diet: the earlier stages' text beside ordinary prose", "family": "stage", "status": "a member of the group stages-1-8; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.783 -> 0.724 bpb; web text -0.0002 bpb", "measured": { "own_stage_bpb_arms_off": 0.783, "own_stage_bpb_group_on": 0.72, "own_stage_bpb_this_arm_alone": 0.7242, "share_of_the_group_on_own_stage": -0.0568, "share_of_the_group_on_own_stage_se": 0.0328, "share_of_the_group_on_web_text": -0.00096, "alone_on_web_text": -0.00019, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0004, "s2_concept": -0.0005, "s3_rules": -0.001, "s4_arith": -0.0103, "s5_causal": -0.0002, "s6_tryfail": -0.0004, "s8_register": -0.0001 }, "alone_on_partner_stages": { "s1_perspective": -0.0568, "s2_concept": -0.0058, "s3_rules": -0.0617, "s4_arith": -0.0385, "s5_causal": -0.1028, "s6_tryfail": -0.0802, "s8_register": -0.008 }, "steps_trained": 4000, "second_seed": { "own_stage_bpb_arms_off": 0.783, "own_stage_bpb_group_on": 0.7198, "own_stage_bpb_this_arm_alone": 0.7254, "share_of_the_group_on_own_stage": -0.0569, "share_of_the_group_on_own_stage_se": 0.0329, "share_of_the_group_on_web_text": -0.0007, "alone_on_web_text": -0.00032, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0001, "s2_concept": -0.0003, "s3_rules": -0.0007, "s4_arith": -0.0099, "s5_causal": 0.0, "s6_tryfail": -0.0003, "s8_register": -0.0002 }, "alone_on_partner_stages": { "s1_perspective": -0.0237, "s2_concept": -0.0022, "s3_rules": -0.0529, "s4_arith": -0.0414, "s5_causal": -0.0976, "s6_tryfail": -0.0774, "s8_register": -0.0061 }, "steps_trained": 4000 } }, "examples": [ "It has been compared to a chant, a rhythmic divine beauty, a melody, an aria, a toccata, an edification, an exaltation. As poetry is for the tongue, calligraphy is to the page. The authors of The Splendor of Islamic Calligraphy put it best when they said: “Calligraphy is the plainsong of the divine”.\nCalligraphy is the art of the linear graphic, but it is more than that. In Isl" ] }, { "id": "stages-1-8/s8_register", "name": "stages-1-8/s8_register", "group": "stages-1-8", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8/s8_register.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gXA_close8_s8_register.safetensors", "sha256": "19b02efab5fb482cd50d10508cda56868d8bb1f7fce835a0b7e9babec182aa3b" }, "recipe": "trained as an always-on group on the frozen final core by continuing the four-arm group's run. The four arms of stages 1 to 4 were loaded as trained (see the group stages-1-4) and held fixed and on. Then a new arm for stage 5 was attached over them and trained for 4,000 steps in their presence; then, with it also held fixed, a new arm for stage 6 the same way, then stage 7, then stage 8. Each step gives the training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of another arm of the eight, drawn from every other stage whether or not its arm was attached yet; the mixed stage's text, which contains the other stages' kinds, is drawn as no arm's partner row): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 8: register", "behavior": "the same content in different registers of speech", "family": "stage", "status": "a member of the group stages-1-8; trained always on with its partners, so alone is not its trained state", "score": "alone: own stage text 0.491 -> 0.068 bpb; web text +0.0005 bpb", "measured": { "own_stage_bpb_arms_off": 0.4907, "own_stage_bpb_group_on": 0.0212, "own_stage_bpb_this_arm_alone": 0.0678, "share_of_the_group_on_own_stage": -0.3546, "share_of_the_group_on_own_stage_se": 0.0066, "share_of_the_group_on_web_text": 0.00045, "alone_on_web_text": 0.00048, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0007, "s2_concept": -0.0008, "s3_rules": 0.0005, "s4_arith": 0.0001, "s5_causal": 0.0002, "s6_tryfail": 0.0002, "s7_mixed": 0.0012 }, "alone_on_partner_stages": { "s1_perspective": -0.0172, "s2_concept": -0.0079, "s3_rules": -0.0048, "s4_arith": -0.0065, "s5_causal": -0.0189, "s6_tryfail": -0.022, "s7_mixed": -0.0002 }, "steps_trained": 4000, "second_seed": { "own_stage_bpb_arms_off": 0.4907, "own_stage_bpb_group_on": 0.0208, "own_stage_bpb_this_arm_alone": 0.0516, "share_of_the_group_on_own_stage": -0.3573, "share_of_the_group_on_own_stage_se": 0.0067, "share_of_the_group_on_web_text": 0.00046, "alone_on_web_text": 5e-05, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0001, "s2_concept": 0.0001, "s3_rules": 0.0005, "s4_arith": 0.0001, "s5_causal": 0.0002, "s6_tryfail": 0.0001, "s7_mixed": 0.0002 }, "alone_on_partner_stages": { "s1_perspective": -0.0045, "s2_concept": -0.005, "s3_rules": -0.0063, "s4_arith": -0.0064, "s5_causal": -0.0139, "s6_tryfail": -0.0187, "s7_mixed": 0.0001 }, "steps_trained": 4000 } }, "examples": [ "The word 'ancient' means from a very long time ago. Used in a sentence: a ancient is easy to point at once you know the word." ] }, { "id": "stages-1-8", "name": "stages-1-8", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 109543680, "sites": 32, "weights": [ "arms/stages-1-8/s1_perspective.safetensors", "arms/stages-1-8/s2_concept.safetensors", "arms/stages-1-8/s3_rules.safetensors", "arms/stages-1-8/s4_arith.safetensors", "arms/stages-1-8/s5_causal.safetensors", "arms/stages-1-8/s6_tryfail.safetensors", "arms/stages-1-8/s7_mixed.safetensors", "arms/stages-1-8/s8_register.safetensors" ], "mount": { "type": "stack", "members": [ "stages-1-8/s1_perspective", "stages-1-8/s2_concept", "stages-1-8/s3_rules", "stages-1-8/s4_arith", "stages-1-8/s5_causal", "stages-1-8/s6_tryfail", "stages-1-8/s7_mixed", "stages-1-8/s8_register" ] }, "precision": "fp32", "template": { "frame": "raw", "template": "raw text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "recipe": "trained as an always-on group on the frozen final core by continuing the four-arm group's run. The four arms of stages 1 to 4 were loaded as trained (see the group stages-1-4) and held fixed and on. Then a new arm for stage 5 was attached over them and trained for 4,000 steps in their presence; then, with it also held fixed, a new arm for stage 6 the same way, then stage 7, then stage 8. Each step gives the training arm 2 x 4096 bytes of its own stage's text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of another arm of the eight, drawn from every other stage whether or not its arm was attached yet; the mixed stage's text, which contains the other stages' kinds, is drawn as no arm's partner row): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stages 1 to 8, on together", "behavior": "the arms of stage 1: perspective, stage 2: concepts, stage 3: rule chains, stage 4: arithmetic, stage 5: cause and effect, stage 6: try, fail, revise, stage 7: mixed, stage 8: register, always on together, as they were trained", "family": "group", "status": "two seeds", "score": "stage text, arms off -> on: s1_perspective 0.554 -> 0.049; s2_concept 0.721 -> 0.111; s3_rules 0.930 -> 0.224; s4_arith 1.234 -> 0.328; s5_causal 0.660 -> 0.043; s6_tryfail 0.673 -> 0.055; s7_mixed 0.783 -> 0.720; s8_register 0.491 -> 0.021; web text +0.0019 bpb", "measured": { "stage_text_bpb": { "s1_perspective": { "arms_off": 0.5538, "group_on": 0.0492, "difference": -0.5046, "difference_se": 0.0048 }, "s2_concept": { "arms_off": 0.7212, "group_on": 0.1105, "difference": -0.6107, "difference_se": 0.0038 }, "s3_rules": { "arms_off": 0.9296, "group_on": 0.2237, "difference": -0.7059, "difference_se": 0.0053 }, "s4_arith": { "arms_off": 1.2339, "group_on": 0.3279, "difference": -0.9059, "difference_se": 0.0062 }, "s5_causal": { "arms_off": 0.6604, "group_on": 0.0431, "difference": -0.6174, "difference_se": 0.0069 }, "s6_tryfail": { "arms_off": 0.6728, "group_on": 0.0548, "difference": -0.618, "difference_se": 0.0037 }, "s7_mixed": { "arms_off": 0.783, "group_on": 0.72, "difference": -0.063, "difference_se": 0.039 }, "s8_register": { "arms_off": 0.4907, "group_on": 0.0212, "difference": -0.4695, "difference_se": 0.0085 } }, "web_text_difference": 0.00189, "web_text_difference_se": 0.00032, "steps_trained": { "s1_perspective": 3200, "s2_concept": 2400, "s3_rules": 800, "s4_arith": 800, "s5_causal": 4000, "s6_tryfail": 4000, "s7_mixed": 4000, "s8_register": 4000 }, "probe_suite_mean": { "arms_off": 0.411, "group_on": 0.422 }, "second_seed": { "stage_text_bpb": { "s1_perspective": { "arms_off": 0.5538, "group_on": 0.0484, "difference": -0.5055, "difference_se": 0.0048 }, "s2_concept": { "arms_off": 0.7212, "group_on": 0.1212, "difference": -0.6, "difference_se": 0.0038 }, "s3_rules": { "arms_off": 0.9296, "group_on": 0.2222, "difference": -0.7074, "difference_se": 0.0055 }, "s4_arith": { "arms_off": 1.2339, "group_on": 0.3266, "difference": -0.9073, "difference_se": 0.0062 }, "s5_causal": { "arms_off": 0.6604, "group_on": 0.0428, "difference": -0.6176, "difference_se": 0.0069 }, "s6_tryfail": { "arms_off": 0.6728, "group_on": 0.0543, "difference": -0.6185, "difference_se": 0.0036 }, "s7_mixed": { "arms_off": 0.783, "group_on": 0.7198, "difference": -0.0632, "difference_se": 0.039 }, "s8_register": { "arms_off": 0.4907, "group_on": 0.0208, "difference": -0.4699, "difference_se": 0.0086 } }, "web_text_difference": 0.00103, "web_text_difference_se": 0.0002, "steps_trained": { "s1_perspective": 3200, "s2_concept": 2400, "s3_rules": 800, "s4_arith": 800, "s5_causal": 4000, "s6_tryfail": 4000, "s7_mixed": 4000, "s8_register": 4000 }, "probe_suite_mean": { "arms_off": 0.411, "group_on": 0.43 } } }, "examples": [] }, { "id": "stages-1-8-caption/s1_perspective", "name": "stages-1-8-caption/s1_perspective", "group": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8-caption/s1_perspective.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gCA_close9_s1_perspective.safetensors", "sha256": "1e484bc7a8db7f02245c2bed1ef2222815e0223190383386618d44bcb441f8e1" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 1: perspective", "behavior": "one small event retold from each side: seen from a person, said to them, told about them", "family": "stage", "status": "a member of the group stages-1-8-caption; trained always on with its partners, so alone is not its trained state; experimental", "score": "alone: own stage text 0.554 -> 0.052 bpb; web text +0.0014 bpb", "measured": { "own_stage_bpb_arms_off": 0.5538, "own_stage_bpb_group_on": 0.0496, "own_stage_bpb_this_arm_alone": 0.0525, "share_of_the_group_on_own_stage": -0.3792, "share_of_the_group_on_own_stage_se": 0.0057, "share_of_the_group_on_web_text": 3e-05, "alone_on_web_text": 0.00143, "share_of_the_group_on_partner_stages": { "s2_concept": -0.2208, "s3_rules": -0.182, "s4_arith": -0.1234, "s5_causal": -0.0131, "s6_tryfail": -0.0326, "s7_mixed": -0.004, "s8_register": -0.03, "s9_caption": -0.0028 }, "alone_on_partner_stages": { "s2_concept": -0.4817, "s3_rules": -0.4209, "s4_arith": -0.5113, "s5_causal": -0.1978, "s6_tryfail": -0.1782, "s7_mixed": -0.0101, "s8_register": -0.1119, "s9_caption": 0.0011 }, "steps_trained": 3200, "stage_items_own_form_accuracy": { "arms_off": 0.55, "group_on": 0.863, "this_arm_alone": 0.838 }, "stage_items_new_form_accuracy": { "arms_off": 0.463, "group_on": 0.438, "this_arm_alone": 0.463 }, "second_seed": { "own_stage_bpb_arms_off": 0.5538, "own_stage_bpb_group_on": 0.0488, "own_stage_bpb_this_arm_alone": 0.0522, "share_of_the_group_on_own_stage": -0.4395, "share_of_the_group_on_own_stage_se": 0.0059, "share_of_the_group_on_web_text": 0.00026, "alone_on_web_text": 0.00134, "share_of_the_group_on_partner_stages": { "s2_concept": -0.2322, "s3_rules": -0.1926, "s4_arith": -0.1255, "s5_causal": -0.0273, "s6_tryfail": -0.021, "s7_mixed": -0.0048, "s8_register": -0.0213, "s9_caption": -0.0028 }, "alone_on_partner_stages": { "s2_concept": -0.4772, "s3_rules": -0.4262, "s4_arith": -0.4807, "s5_causal": -0.1919, "s6_tryfail": -0.169, "s7_mixed": -0.0088, "s8_register": -0.1105, "s9_caption": 0.0004 }, "steps_trained": 3200, "stage_items_own_form_accuracy": { "arms_off": 0.55, "group_on": 1.0, "this_arm_alone": 0.925 }, "stage_items_new_form_accuracy": { "arms_off": 0.463, "group_on": 0.5, "this_arm_alone": 0.463 } } }, "examples": [ "Rui dropped the acorn near the box. Seen from Rui: I dropped the acorn near the box. Said to Rui: you dropped the acorn near the box. Told about Rui: Rui dropped the acorn near the box. Three ways of saying, one thing that happened." ], "experimental": true }, { "id": "stages-1-8-caption/s2_concept", "name": "stages-1-8-caption/s2_concept", "group": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8-caption/s2_concept.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gCA_close9_s2_concept.safetensors", "sha256": "c0058a6196baddeb4eeed16dbc468e4eacb7b686be6aabdd0d2a06b4a9c8b626" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 2: concepts", "behavior": "kinds, properties and differences: what a thing is a kind of, what its kind can do, how two things differ", "family": "stage", "status": "a member of the group stages-1-8-caption; trained always on with its partners, so alone is not its trained state; experimental", "score": "alone: own stage text 0.721 -> 0.350 bpb; web text +0.0010 bpb", "measured": { "own_stage_bpb_arms_off": 0.7212, "own_stage_bpb_group_on": 0.1113, "own_stage_bpb_this_arm_alone": 0.3503, "share_of_the_group_on_own_stage": -0.1226, "share_of_the_group_on_own_stage_se": 0.0012, "share_of_the_group_on_web_text": -0.00042, "alone_on_web_text": 0.00105, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0062, "s3_rules": -0.1082, "s4_arith": -0.1052, "s5_causal": -0.0017, "s6_tryfail": -0.0034, "s7_mixed": -0.0008, "s8_register": -0.0013, "s9_caption": -0.0009 }, "alone_on_partner_stages": { "s1_perspective": -0.0273, "s3_rules": -0.3108, "s4_arith": -0.3537, "s5_causal": -0.0015, "s6_tryfail": 0.0004, "s7_mixed": -0.0, "s8_register": -0.0215, "s9_caption": 0.001 }, "steps_trained": 2400, "stage_items_own_form_accuracy": { "arms_off": 0.298, "group_on": 0.964, "this_arm_alone": 0.964 }, "stage_items_new_form_accuracy": { "arms_off": 0.678, "group_on": 0.703, "this_arm_alone": 0.72 }, "second_seed": { "own_stage_bpb_arms_off": 0.7212, "own_stage_bpb_group_on": 0.1218, "own_stage_bpb_this_arm_alone": 0.3603, "share_of_the_group_on_own_stage": -0.123, "share_of_the_group_on_own_stage_se": 0.0013, "share_of_the_group_on_web_text": -0.00084, "alone_on_web_text": 0.00086, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0046, "s3_rules": -0.1022, "s4_arith": -0.0974, "s5_causal": -0.0011, "s6_tryfail": -0.0024, "s7_mixed": -0.0002, "s8_register": -0.0007, "s9_caption": -0.001 }, "alone_on_partner_stages": { "s1_perspective": -0.03, "s3_rules": -0.3138, "s4_arith": -0.3577, "s5_causal": -0.0127, "s6_tryfail": -0.0062, "s7_mixed": 0.0002, "s8_register": -0.0193, "s9_caption": 0.0009 }, "steps_trained": 2400, "stage_items_own_form_accuracy": { "arms_off": 0.298, "group_on": 0.964, "this_arm_alone": 0.905 }, "stage_items_new_form_accuracy": { "arms_off": 0.678, "group_on": 0.763, "this_arm_alone": 0.703 } } }, "examples": [ "A whale is a kind of mammal. Most mammals can swim. A whale is not a beetle; they differ in kind." ], "experimental": true }, { "id": "stages-1-8-caption/s3_rules", "name": "stages-1-8-caption/s3_rules", "group": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8-caption/s3_rules.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gCA_close9_s3_rules.safetensors", "sha256": "5962884db9a8724e7e97d62b22a8cb1c80e65a5dd83ea69e0a15e1348094c97a" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 3: rule chains", "behavior": "if-then rules over made-up words, followed step by step to what follows", "family": "stage", "status": "a member of the group stages-1-8-caption; trained always on with its partners, so alone is not its trained state; experimental", "score": "alone: own stage text 0.930 -> 0.824 bpb; web text +0.0005 bpb", "measured": { "own_stage_bpb_arms_off": 0.9296, "own_stage_bpb_group_on": 0.2245, "own_stage_bpb_this_arm_alone": 0.8236, "share_of_the_group_on_own_stage": -0.0903, "share_of_the_group_on_own_stage_se": 0.0014, "share_of_the_group_on_web_text": 0.00016, "alone_on_web_text": 0.00053, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0, "s2_concept": 0.0006, "s4_arith": -0.0002, "s5_causal": -0.0001, "s6_tryfail": -0.0003, "s7_mixed": 0.0, "s8_register": -0.0002, "s9_caption": -0.0001 }, "alone_on_partner_stages": { "s1_perspective": -0.0003, "s2_concept": 0.0031, "s4_arith": 0.0001, "s5_causal": 0.0071, "s6_tryfail": 0.001, "s7_mixed": -0.0005, "s8_register": 0.0002, "s9_caption": 0.0001 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.155, "group_on": 0.835, "this_arm_alone": 0.32 }, "stage_items_new_form_accuracy": { "arms_off": 0.07, "group_on": 0.29, "this_arm_alone": 0.08 }, "second_seed": { "own_stage_bpb_arms_off": 0.9296, "own_stage_bpb_group_on": 0.2225, "own_stage_bpb_this_arm_alone": 0.8702, "share_of_the_group_on_own_stage": -0.0924, "share_of_the_group_on_own_stage_se": 0.0016, "share_of_the_group_on_web_text": 0.00026, "alone_on_web_text": 0.00034, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0001, "s2_concept": -0.0006, "s4_arith": -0.0005, "s5_causal": -0.0003, "s6_tryfail": -0.0005, "s7_mixed": -0.0001, "s8_register": -0.0001, "s9_caption": 0.0004 }, "alone_on_partner_stages": { "s1_perspective": -0.0003, "s2_concept": 0.0016, "s4_arith": -0.0047, "s5_causal": 0.006, "s6_tryfail": -0.0011, "s7_mixed": -0.0007, "s8_register": -0.0044, "s9_caption": 0.0002 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.155, "group_on": 0.92, "this_arm_alone": 0.335 }, "stage_items_new_form_accuracy": { "arms_off": 0.07, "group_on": 0.31, "this_arm_alone": 0.1 } } }, "examples": [ "If someone is hoream, then they are hortil. If someone is hortil, then they are babig. If someone is babig, then they are banng. Vin is hoream. So Vin is hortil. So Vin is babig. So Vin is banng." ], "experimental": true }, { "id": "stages-1-8-caption/s4_arith", "name": "stages-1-8-caption/s4_arith", "group": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8-caption/s4_arith.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gCA_close9_s4_arith.safetensors", "sha256": "622bfe16860d9b995214cb630c70fa76b2e44b3474438c6de2577822f441777c" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 4: arithmetic", "behavior": "small arithmetic worked out in text", "family": "stage", "status": "a member of the group stages-1-8-caption; trained always on with its partners, so alone is not its trained state; experimental", "score": "alone: own stage text 1.234 -> 1.127 bpb; web text +0.0001 bpb", "measured": { "own_stage_bpb_arms_off": 1.2339, "own_stage_bpb_group_on": 0.3287, "own_stage_bpb_this_arm_alone": 1.1267, "share_of_the_group_on_own_stage": -0.1454, "share_of_the_group_on_own_stage_se": 0.0023, "share_of_the_group_on_web_text": -4e-05, "alone_on_web_text": 0.00011, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0001, "s2_concept": -0.0006, "s3_rules": -0.0001, "s5_causal": -0.0002, "s6_tryfail": -0.0041, "s7_mixed": 0.0002, "s8_register": -0.0002, "s9_caption": 0.0004 }, "alone_on_partner_stages": { "s1_perspective": -0.0009, "s2_concept": -0.0004, "s3_rules": -0.0016, "s5_causal": -0.0033, "s6_tryfail": -0.0135, "s7_mixed": -0.0002, "s8_register": -0.0015, "s9_caption": -0.0004 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.344, "group_on": 0.65, "this_arm_alone": 0.381 }, "stage_items_new_form_accuracy": { "arms_off": 0.242, "group_on": 0.254, "this_arm_alone": 0.246 }, "second_seed": { "own_stage_bpb_arms_off": 1.2339, "own_stage_bpb_group_on": 0.3272, "own_stage_bpb_this_arm_alone": 1.155, "share_of_the_group_on_own_stage": -0.1658, "share_of_the_group_on_own_stage_se": 0.0018, "share_of_the_group_on_web_text": 0.00014, "alone_on_web_text": 6e-05, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0, "s2_concept": -0.0002, "s3_rules": 0.0, "s5_causal": -0.0003, "s6_tryfail": -0.0044, "s7_mixed": 0.0001, "s8_register": -0.0001, "s9_caption": 0.0 }, "alone_on_partner_stages": { "s1_perspective": -0.0013, "s2_concept": -0.0015, "s3_rules": 0.0005, "s5_causal": -0.0011, "s6_tryfail": -0.0134, "s7_mixed": -0.0003, "s8_register": -0.0014, "s9_caption": -0.0005 }, "steps_trained": 800, "stage_items_own_form_accuracy": { "arms_off": 0.344, "group_on": 0.625, "this_arm_alone": 0.412 }, "stage_items_new_form_accuracy": { "arms_off": 0.242, "group_on": 0.263, "this_arm_alone": 0.237 } } }, "examples": [ "Count by 3s: 1 3s make 3; 2 3s make 6; 3 3s make 9; 4 3s make 12; 5 3s make 15. So 3 times 5 is 15." ], "experimental": true }, { "id": "stages-1-8-caption/s5_causal", "name": "stages-1-8-caption/s5_causal", "group": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8-caption/s5_causal.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gCA_close9_s5_causal.safetensors", "sha256": "45617e30c3f1c6121b8fbfffb87f03253043a744cfed0c99ddc9f79006bd495c" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 5: cause and effect", "behavior": "short cause-and-effect text: what happened and why", "family": "stage", "status": "a member of the group stages-1-8-caption; trained always on with its partners, so alone is not its trained state; experimental", "score": "alone: own stage text 0.660 -> 0.091 bpb; web text +0.0003 bpb", "measured": { "own_stage_bpb_arms_off": 0.6604, "own_stage_bpb_group_on": 0.0434, "own_stage_bpb_this_arm_alone": 0.0911, "share_of_the_group_on_own_stage": -0.409, "share_of_the_group_on_own_stage_se": 0.0055, "share_of_the_group_on_web_text": 0.00036, "alone_on_web_text": 0.00028, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0006, "s2_concept": -0.0005, "s3_rules": 0.0004, "s4_arith": -0.0001, "s6_tryfail": -0.001, "s7_mixed": 0.0, "s8_register": -0.0006, "s9_caption": 0.0001 }, "alone_on_partner_stages": { "s1_perspective": -0.0183, "s2_concept": -0.0109, "s3_rules": -0.0122, "s4_arith": -0.0091, "s6_tryfail": -0.0065, "s7_mixed": -0.0011, "s8_register": -0.0064, "s9_caption": -0.0004 }, "steps_trained": 4000, "second_seed": { "own_stage_bpb_arms_off": 0.6604, "own_stage_bpb_group_on": 0.0431, "own_stage_bpb_this_arm_alone": 0.1418, "share_of_the_group_on_own_stage": -0.4206, "share_of_the_group_on_own_stage_se": 0.0042, "share_of_the_group_on_web_text": 2e-05, "alone_on_web_text": 0.0, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0001, "s2_concept": 0.0002, "s3_rules": 0.0004, "s4_arith": -0.0001, "s6_tryfail": -0.0004, "s7_mixed": 0.0, "s8_register": -0.0, "s9_caption": -0.0 }, "alone_on_partner_stages": { "s1_perspective": -0.0022, "s2_concept": -0.001, "s3_rules": -0.0018, "s4_arith": -0.0022, "s6_tryfail": -0.0023, "s7_mixed": -0.0001, "s8_register": -0.0009, "s9_caption": -0.0001 }, "steps_trained": 4000 } }, "examples": [ "Noor stacked the cups too high. Because of that, the tower leaned. Because of that, the cups crashed down." ], "experimental": true }, { "id": "stages-1-8-caption/s6_tryfail", "name": "stages-1-8-caption/s6_tryfail", "group": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8-caption/s6_tryfail.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gCA_close9_s6_tryfail.safetensors", "sha256": "0a1e0004ecaf087bc83898f55ad28971af4f55e2eaf4a2d1093a9cc3a41db89d" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 6: try, fail, revise", "behavior": "an attempt, its failure and the revised attempt", "family": "stage", "status": "a member of the group stages-1-8-caption; trained always on with its partners, so alone is not its trained state; experimental", "score": "alone: own stage text 0.673 -> 0.164 bpb; web text +0.0001 bpb", "measured": { "own_stage_bpb_arms_off": 0.6728, "own_stage_bpb_group_on": 0.055, "own_stage_bpb_this_arm_alone": 0.1638, "share_of_the_group_on_own_stage": -0.4041, "share_of_the_group_on_own_stage_se": 0.0034, "share_of_the_group_on_web_text": -0.0002, "alone_on_web_text": 0.00013, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0001, "s2_concept": 0.0002, "s3_rules": 0.0001, "s4_arith": -0.0008, "s5_causal": -0.0001, "s7_mixed": -0.0001, "s8_register": -0.0005, "s9_caption": -0.0002 }, "alone_on_partner_stages": { "s1_perspective": -0.002, "s2_concept": 0.0012, "s3_rules": 0.0006, "s4_arith": 0.0036, "s5_causal": -0.0098, "s7_mixed": -0.0, "s8_register": -0.0003, "s9_caption": -0.0003 }, "steps_trained": 4000, "second_seed": { "own_stage_bpb_arms_off": 0.6728, "own_stage_bpb_group_on": 0.0547, "own_stage_bpb_this_arm_alone": 0.1437, "share_of_the_group_on_own_stage": -0.4121, "share_of_the_group_on_own_stage_se": 0.003, "share_of_the_group_on_web_text": 7e-05, "alone_on_web_text": 0.00011, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0001, "s2_concept": 0.0003, "s3_rules": 0.0001, "s4_arith": -0.0016, "s5_causal": 0.0001, "s7_mixed": 0.0, "s8_register": -0.0004, "s9_caption": -0.0 }, "alone_on_partner_stages": { "s1_perspective": 0.0029, "s2_concept": 0.0013, "s3_rules": -0.0003, "s4_arith": -0.001, "s5_causal": -0.0026, "s7_mixed": 0.0001, "s8_register": -0.0001, "s9_caption": -0.0002 }, "steps_trained": 4000 } }, "examples": [ "Talia adds 34 and 23 and writes 65. Talia checks by counting back: 65 is too big. Talia tries again carefully: 34 + 23 = 57. The check works now, and Talia keeps the good method." ], "experimental": true }, { "id": "stages-1-8-caption/s7_mixed", "name": "stages-1-8-caption/s7_mixed", "group": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8-caption/s7_mixed.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gCA_close9_s7_mixed.safetensors", "sha256": "c7e41078c9528d9a2c76dbec7d399a64b7b50dfb33672924cdcca0cd5f06c70f" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 7: mixed", "behavior": "the mixed stage's diet: the earlier stages' text beside ordinary prose", "family": "stage", "status": "a member of the group stages-1-8-caption; trained always on with its partners, so alone is not its trained state; experimental", "score": "alone: own stage text 0.783 -> 0.724 bpb; web text -0.0002 bpb", "measured": { "own_stage_bpb_arms_off": 0.783, "own_stage_bpb_group_on": 0.7218, "own_stage_bpb_this_arm_alone": 0.7242, "share_of_the_group_on_own_stage": -0.0559, "share_of_the_group_on_own_stage_se": 0.0322, "share_of_the_group_on_web_text": -0.00084, "alone_on_web_text": -0.00019, "share_of_the_group_on_partner_stages": { "s1_perspective": -0.0003, "s2_concept": -0.0005, "s3_rules": -0.0009, "s4_arith": -0.01, "s5_causal": -0.0001, "s6_tryfail": -0.0002, "s8_register": -0.0001, "s9_caption": -0.0 }, "alone_on_partner_stages": { "s1_perspective": -0.0568, "s2_concept": -0.0058, "s3_rules": -0.0617, "s4_arith": -0.0385, "s5_causal": -0.1028, "s6_tryfail": -0.0802, "s8_register": -0.008, "s9_caption": -0.0008 }, "steps_trained": 4000, "second_seed": { "own_stage_bpb_arms_off": 0.783, "own_stage_bpb_group_on": 0.7215, "own_stage_bpb_this_arm_alone": 0.7254, "share_of_the_group_on_own_stage": -0.0562, "share_of_the_group_on_own_stage_se": 0.0325, "share_of_the_group_on_web_text": -0.00088, "alone_on_web_text": -0.00032, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0001, "s2_concept": -0.0002, "s3_rules": -0.0008, "s4_arith": -0.0102, "s5_causal": -0.0, "s6_tryfail": -0.0003, "s8_register": -0.0001, "s9_caption": -0.0004 }, "alone_on_partner_stages": { "s1_perspective": -0.0237, "s2_concept": -0.0022, "s3_rules": -0.0529, "s4_arith": -0.0414, "s5_causal": -0.0976, "s6_tryfail": -0.0774, "s8_register": -0.0061, "s9_caption": -0.0003 }, "steps_trained": 4000 } }, "examples": [ "It has been compared to a chant, a rhythmic divine beauty, a melody, an aria, a toccata, an edification, an exaltation. As poetry is for the tongue, calligraphy is to the page. The authors of The Splendor of Islamic Calligraphy put it best when they said: “Calligraphy is the plainsong of the divine”.\nCalligraphy is the art of the linear graphic, but it is more than that. In Isl" ], "experimental": true }, { "id": "stages-1-8-caption/s8_register", "name": "stages-1-8-caption/s8_register", "group": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8-caption/s8_register.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gCA_close9_s8_register.safetensors", "sha256": "6f034611c2bd0319fbb3eb6a6848a86301323311e2cd5d9dec1cf89a2559ebfc" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stage 8: register", "behavior": "the same content in different registers of speech", "family": "stage", "status": "a member of the group stages-1-8-caption; trained always on with its partners, so alone is not its trained state; experimental", "score": "alone: own stage text 0.491 -> 0.068 bpb; web text +0.0005 bpb", "measured": { "own_stage_bpb_arms_off": 0.4907, "own_stage_bpb_group_on": 0.0217, "own_stage_bpb_this_arm_alone": 0.0678, "share_of_the_group_on_own_stage": -0.3662, "share_of_the_group_on_own_stage_se": 0.0067, "share_of_the_group_on_web_text": 0.00029, "alone_on_web_text": 0.00048, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0005, "s2_concept": -0.0009, "s3_rules": 0.0005, "s4_arith": 0.0001, "s5_causal": 0.0001, "s6_tryfail": 0.0001, "s7_mixed": 0.0007, "s9_caption": 0.0002 }, "alone_on_partner_stages": { "s1_perspective": -0.0172, "s2_concept": -0.0079, "s3_rules": -0.0048, "s4_arith": -0.0065, "s5_causal": -0.0189, "s6_tryfail": -0.022, "s7_mixed": -0.0002, "s9_caption": -0.0009 }, "steps_trained": 4000, "second_seed": { "own_stage_bpb_arms_off": 0.4907, "own_stage_bpb_group_on": 0.0213, "own_stage_bpb_this_arm_alone": 0.0516, "share_of_the_group_on_own_stage": -0.3671, "share_of_the_group_on_own_stage_se": 0.0071, "share_of_the_group_on_web_text": -0.00019, "alone_on_web_text": 5e-05, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0002, "s2_concept": 0.0002, "s3_rules": 0.0004, "s4_arith": -0.0001, "s5_causal": 0.0, "s6_tryfail": 0.0, "s7_mixed": 0.0002, "s9_caption": -0.0005 }, "alone_on_partner_stages": { "s1_perspective": -0.0045, "s2_concept": -0.005, "s3_rules": -0.0063, "s4_arith": -0.0064, "s5_causal": -0.0139, "s6_tryfail": -0.0187, "s7_mixed": 0.0001, "s9_caption": 0.0 }, "steps_trained": 4000 } }, "examples": [ "The word 'ancient' means from a very long time ago. Used in a sentence: a ancient is easy to point at once you know the word." ], "experimental": true }, { "id": "stages-1-8-caption/s9_caption", "name": "stages-1-8-caption/s9_caption", "group": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/stages-1-8-caption/s9_caption.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "gCA_close9_s9_caption.safetensors", "sha256": "2e477395a4c1c1775e119f1039c48efa290748904ea12b8b20fc10d30c20f278" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Captions: the ninth arm (experimental)", "behavior": "image captions in one labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line): photographs with structured captions, CC12M alt text and generated descriptions, COCO captions, danbooru tags", "family": "caption", "status": "a member of the group stages-1-8-caption; trained always on with its partners, so alone is not its trained state; experimental", "score": "alone: own stage text 1.265 -> 0.785 bpb; web text +0.0015 bpb", "measured": { "own_stage_bpb_arms_off": 1.2648, "own_stage_bpb_group_on": 0.7829, "own_stage_bpb_this_arm_alone": 0.7851, "share_of_the_group_on_own_stage": -0.481, "share_of_the_group_on_own_stage_se": 0.046, "share_of_the_group_on_web_text": 0.00076, "alone_on_web_text": 0.00152, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0004, "s2_concept": 0.0008, "s3_rules": 0.0008, "s4_arith": 0.0008, "s5_causal": 0.0004, "s6_tryfail": 0.0002, "s7_mixed": 0.0018, "s8_register": 0.0005 }, "alone_on_partner_stages": { "s1_perspective": -0.0018, "s2_concept": 0.0018, "s3_rules": -0.0049, "s4_arith": -0.0039, "s5_causal": -0.0007, "s6_tryfail": -0.0077, "s7_mixed": 0.0027, "s8_register": 0.0286 }, "steps_trained": 4000, "second_seed": { "own_stage_bpb_arms_off": 1.2648, "own_stage_bpb_group_on": 0.7795, "own_stage_bpb_this_arm_alone": 0.7819, "share_of_the_group_on_own_stage": -0.4856, "share_of_the_group_on_own_stage_se": 0.0472, "share_of_the_group_on_web_text": 0.00066, "alone_on_web_text": 0.00137, "share_of_the_group_on_partner_stages": { "s1_perspective": 0.0004, "s2_concept": 0.0006, "s3_rules": 0.0003, "s4_arith": 0.0006, "s5_causal": 0.0003, "s6_tryfail": 0.0004, "s7_mixed": 0.0018, "s8_register": 0.0005 }, "alone_on_partner_stages": { "s1_perspective": -0.0009, "s2_concept": 0.005, "s3_rules": 0.0013, "s4_arith": 0.0026, "s5_causal": 0.0016, "s6_tryfail": -0.0021, "s7_mixed": 0.0016, "s8_register": 0.0333 }, "steps_trained": 4000 } }, "examples": [ "tags: 1girl, jewelry, solo, dress, bicycle, realistic, asian, smile, necklace, earrings, black hair, white dress, blurry, blurry background, grin, potted plant, long hair, sleeveless, black eyes, indoors, teeth\nshort: A smiling young woman with dark hair stands indoors holding the handlebars of a vintage-style green bicycle, wearing a light-colored sleeveless dress.\nhair: long wavy" ], "experimental": true }, { "id": "stages-1-8-caption", "name": "stages-1-8-caption", "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 123236640, "sites": 32, "weights": [ "arms/stages-1-8-caption/s1_perspective.safetensors", "arms/stages-1-8-caption/s2_concept.safetensors", "arms/stages-1-8-caption/s3_rules.safetensors", "arms/stages-1-8-caption/s4_arith.safetensors", "arms/stages-1-8-caption/s5_causal.safetensors", "arms/stages-1-8-caption/s6_tryfail.safetensors", "arms/stages-1-8-caption/s7_mixed.safetensors", "arms/stages-1-8-caption/s8_register.safetensors", "arms/stages-1-8-caption/s9_caption.safetensors" ], "mount": { "type": "stack", "members": [ "stages-1-8-caption/s1_perspective", "stages-1-8-caption/s2_concept", "stages-1-8-caption/s3_rules", "stages-1-8-caption/s4_arith", "stages-1-8-caption/s5_causal", "stages-1-8-caption/s6_tryfail", "stages-1-8-caption/s7_mixed", "stages-1-8-caption/s8_register", "stages-1-8-caption/s9_caption" ] }, "precision": "fp32", "template": { "frame": "raw", "template": "raw text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "recipe": "trained as an always-on group on the frozen final core by continuing the eight-arm group's run. The eight arms of stages 1 to 8 were loaded as trained (see the group stages-1-8) and held fixed and on. Then a ninth arm, for image captions, was attached over them and trained for 4,000 steps in their presence on the caption pack: five sources (two sets of photographs with structured captions, CC12M alt text with generated descriptions, COCO captions, danbooru tags), each image one document in a labelled frame (tags, the source's caption, short, attributes, scene, photo, structure, medium, long, rating: one label per line), documents drawn by source at fixed shares (CC12M 0.41, the two photograph sets 0.16 each, danbooru 0.16, COCO 0.11). Each step gives the training arm 2 x 4096 bytes of caption text with every attached arm on, and a quiet term at weight 2 on 2 rows that are not its text (one of web text, one of the stage text of one of the eight, drawn from every stage but the mixed one): the KL from the model with that arm switched off to the model with every arm on. Pure Adam, lr 0.001", "title": "Stages 1 to 8 with the caption arm (experimental)", "behavior": "the arms of stage 1: perspective, stage 2: concepts, stage 3: rule chains, stage 4: arithmetic, stage 5: cause and effect, stage 6: try, fail, revise, stage 7: mixed, stage 8: register, captions: the ninth arm (experimental), always on together, as they were trained", "family": "group", "status": "two seeds; experimental", "score": "stage text, arms off -> on: s1_perspective 0.554 -> 0.050; s2_concept 0.721 -> 0.111; s3_rules 0.930 -> 0.225; s4_arith 1.234 -> 0.329; s5_causal 0.660 -> 0.043; s6_tryfail 0.673 -> 0.055; s7_mixed 0.783 -> 0.722; s8_register 0.491 -> 0.022; s9_caption 1.265 -> 0.783; web text +0.0027 bpb", "measured": { "stage_text_bpb": { "s1_perspective": { "arms_off": 0.5538, "group_on": 0.0496, "difference": -0.5042, "difference_se": 0.0049 }, "s2_concept": { "arms_off": 0.7212, "group_on": 0.1113, "difference": -0.6099, "difference_se": 0.0039 }, "s3_rules": { "arms_off": 0.9296, "group_on": 0.2245, "difference": -0.7051, "difference_se": 0.0055 }, "s4_arith": { "arms_off": 1.2339, "group_on": 0.3287, "difference": -0.9052, "difference_se": 0.0062 }, "s5_causal": { "arms_off": 0.6604, "group_on": 0.0434, "difference": -0.617, "difference_se": 0.007 }, "s6_tryfail": { "arms_off": 0.6728, "group_on": 0.055, "difference": -0.6178, "difference_se": 0.0037 }, "s7_mixed": { "arms_off": 0.783, "group_on": 0.7218, "difference": -0.0612, "difference_se": 0.0392 }, "s8_register": { "arms_off": 0.4907, "group_on": 0.0217, "difference": -0.469, "difference_se": 0.0085 }, "s9_caption": { "arms_off": 1.2648, "group_on": 0.7829, "difference": -0.4819, "difference_se": 0.0451 } }, "web_text_difference": 0.00265, "web_text_difference_se": 0.00055, "steps_trained": { "s1_perspective": 3200, "s2_concept": 2400, "s3_rules": 800, "s4_arith": 800, "s5_causal": 4000, "s6_tryfail": 4000, "s7_mixed": 4000, "s8_register": 4000, "s9_caption": 4000 }, "probe_suite_mean": { "arms_off": 0.411, "group_on": 0.422 }, "second_seed": { "stage_text_bpb": { "s1_perspective": { "arms_off": 0.5538, "group_on": 0.0488, "difference": -0.5051, "difference_se": 0.0047 }, "s2_concept": { "arms_off": 0.7212, "group_on": 0.1218, "difference": -0.5993, "difference_se": 0.0037 }, "s3_rules": { "arms_off": 0.9296, "group_on": 0.2225, "difference": -0.7071, "difference_se": 0.0056 }, "s4_arith": { "arms_off": 1.2339, "group_on": 0.3272, "difference": -0.9067, "difference_se": 0.0062 }, "s5_causal": { "arms_off": 0.6604, "group_on": 0.0431, "difference": -0.6174, "difference_se": 0.007 }, "s6_tryfail": { "arms_off": 0.6728, "group_on": 0.0547, "difference": -0.6181, "difference_se": 0.0036 }, "s7_mixed": { "arms_off": 0.783, "group_on": 0.7215, "difference": -0.0615, "difference_se": 0.039 }, "s8_register": { "arms_off": 0.4907, "group_on": 0.0213, "difference": -0.4694, "difference_se": 0.0085 }, "s9_caption": { "arms_off": 1.2648, "group_on": 0.7795, "difference": -0.4853, "difference_se": 0.0464 } }, "web_text_difference": 0.00169, "web_text_difference_se": 0.00038, "steps_trained": { "s1_perspective": 3200, "s2_concept": 2400, "s3_rules": 800, "s4_arith": 800, "s5_causal": 4000, "s6_tryfail": 4000, "s7_mixed": 4000, "s8_register": 4000, "s9_caption": 4000 }, "probe_suite_mean": { "arms_off": 0.411, "group_on": 0.411 } } }, "examples": [], "experimental": true }, { "id": "solo/s1_perspective", "name": "solo/s1_perspective", "group": null, "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/solo/s1_perspective.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "s1_perspective_fresh_seed1_d20261005_step4000.safetensors", "sha256": "13290e04d7b7f9e2f26d3b3809afcab54306f3aa7b1d74d73726836a32624177" }, "recipe": "trained alone on the frozen final core: 4000 steps of 2 x 4096 bytes of the stage's own text, pure Adam (lr 0.001), with a quiet term at weight 2 on held-out web text (the KL from the model with the arm off to the model with it on, one chunk beside every task chunk)", "title": "Stage 1: perspective (solo)", "behavior": "one small event retold from each side: seen from a person, said to them, told about them", "family": "solo", "status": "two seeds; trained alone: mount one solo arm at a time", "score": "own stage text 0.554 -> 0.047 bpb; web text +0.0003 bpb", "measured": { "steps": 4000, "own_stage_bpb_arms_off": 0.5538, "own_stage_bpb_arm_on": 0.0469, "own_stage_difference": -0.5069, "own_stage_difference_se": 0.0048, "web_text_difference": 0.00026, "web_text_difference_se": 0.00011, "stage_items_own_form_accuracy": { "arms_off": 0.55, "arm_on": 0.887 }, "stage_items_new_form_accuracy": { "arms_off": 0.463, "arm_on": 0.425 }, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.411 }, "second_seed": { "steps": 4000, "own_stage_bpb_arms_off": 0.5538, "own_stage_bpb_arm_on": 0.0467, "own_stage_difference": -0.5071, "own_stage_difference_se": 0.0048, "web_text_difference": 0.00031, "web_text_difference_se": 0.00013, "stage_items_own_form_accuracy": { "arms_off": 0.55, "arm_on": 1.0 }, "stage_items_new_form_accuracy": { "arms_off": 0.463, "arm_on": 0.5 }, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.407 } } }, "examples": [ "Rui dropped the acorn near the box. Seen from Rui: I dropped the acorn near the box. Said to Rui: you dropped the acorn near the box. Told about Rui: Rui dropped the acorn near the box. Three ways of saying, one thing that happened." ] }, { "id": "solo/s2_concept", "name": "solo/s2_concept", "group": null, "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/solo/s2_concept.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "s2_concept_fresh_seed2_d20261005_step4000.safetensors", "sha256": "a07fbca8d7f171475c1b91509589199323dfed929205df2fb2c55dd8d7e1f507" }, "recipe": "trained alone on the frozen final core: 4000 steps of 2 x 4096 bytes of the stage's own text, pure Adam (lr 0.001), with a quiet term at weight 2 on held-out web text (the KL from the model with the arm off to the model with it on, one chunk beside every task chunk)", "title": "Stage 2: concepts (solo)", "behavior": "kinds, properties and differences: what a thing is a kind of, what its kind can do, how two things differ", "family": "solo", "status": "two seeds; trained alone: mount one solo arm at a time", "score": "own stage text 0.721 -> 0.075 bpb; web text +0.0001 bpb", "measured": { "steps": 4000, "own_stage_bpb_arms_off": 0.7212, "own_stage_bpb_arm_on": 0.0754, "own_stage_difference": -0.6458, "own_stage_difference_se": 0.0038, "web_text_difference": 0.00014, "web_text_difference_se": 0.00014, "stage_items_own_form_accuracy": { "arms_off": 0.298, "arm_on": 1.0 }, "stage_items_new_form_accuracy": { "arms_off": 0.678, "arm_on": 0.695 }, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.389 }, "second_seed": { "steps": 4000, "own_stage_bpb_arms_off": 0.7212, "own_stage_bpb_arm_on": 0.0753, "own_stage_difference": -0.6459, "own_stage_difference_se": 0.0036, "web_text_difference": 3e-05, "web_text_difference_se": 0.0001, "stage_items_own_form_accuracy": { "arms_off": 0.298, "arm_on": 0.988 }, "stage_items_new_form_accuracy": { "arms_off": 0.678, "arm_on": 0.661 }, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.396 } } }, "examples": [ "A whale is a kind of mammal. Most mammals can swim. A whale is not a beetle; they differ in kind." ] }, { "id": "solo/s3_rules", "name": "solo/s3_rules", "group": null, "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/solo/s3_rules.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "s3_rules_fresh_seed3_d20261005_step4000.safetensors", "sha256": "9bafbafaec5834fc1c1596ffbab4174e68fc238ed33440ecc07786e86478f996" }, "recipe": "trained alone on the frozen final core: 4000 steps of 2 x 4096 bytes of the stage's own text, pure Adam (lr 0.001), with a quiet term at weight 2 on held-out web text (the KL from the model with the arm off to the model with it on, one chunk beside every task chunk)", "title": "Stage 3: rule chains (solo)", "behavior": "if-then rules over made-up words, followed step by step to what follows", "family": "solo", "status": "two seeds; trained alone: mount one solo arm at a time", "score": "own stage text 0.930 -> 0.180 bpb; web text +0.0003 bpb", "measured": { "steps": 4000, "own_stage_bpb_arms_off": 0.9296, "own_stage_bpb_arm_on": 0.1804, "own_stage_difference": -0.7492, "own_stage_difference_se": 0.006, "web_text_difference": 0.00032, "web_text_difference_se": 0.00014, "stage_items_own_form_accuracy": { "arms_off": 0.155, "arm_on": 0.82 }, "stage_items_new_form_accuracy": { "arms_off": 0.07, "arm_on": 0.22 }, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.415 }, "second_seed": { "steps": 4000, "own_stage_bpb_arms_off": 0.9296, "own_stage_bpb_arm_on": 0.1795, "own_stage_difference": -0.7502, "own_stage_difference_se": 0.0059, "web_text_difference": 0.00052, "web_text_difference_se": 0.00016, "stage_items_own_form_accuracy": { "arms_off": 0.155, "arm_on": 0.825 }, "stage_items_new_form_accuracy": { "arms_off": 0.07, "arm_on": 0.29 }, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.411 } } }, "examples": [ "If someone is hoream, then they are hortil. If someone is hortil, then they are babig. If someone is babig, then they are banng. Vin is hoream. So Vin is hortil. So Vin is babig. So Vin is banng." ] }, { "id": "solo/s4_arith", "name": "solo/s4_arith", "group": null, "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/solo/s4_arith.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "s4_arith_fresh_seed4_d20261005_step4000.safetensors", "sha256": "1bed0018b40f5de968ecfd2d846ad581b1b92d9444b18df593104051a941d4b4" }, "recipe": "trained alone on the frozen final core: 4000 steps of 2 x 4096 bytes of the stage's own text, pure Adam (lr 0.001), with a quiet term at weight 2 on held-out web text (the KL from the model with the arm off to the model with it on, one chunk beside every task chunk)", "title": "Stage 4: arithmetic (solo)", "behavior": "small arithmetic worked out in text", "family": "solo", "status": "two seeds; trained alone: mount one solo arm at a time", "score": "own stage text 1.234 -> 0.267 bpb; web text +0.0002 bpb", "measured": { "steps": 4000, "own_stage_bpb_arms_off": 1.2339, "own_stage_bpb_arm_on": 0.2671, "own_stage_difference": -0.9668, "own_stage_difference_se": 0.0071, "web_text_difference": 0.00023, "web_text_difference_se": 0.00016, "stage_items_own_form_accuracy": { "arms_off": 0.344, "arm_on": 0.831 }, "stage_items_new_form_accuracy": { "arms_off": 0.242, "arm_on": 0.212 }, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.419 }, "second_seed": { "steps": 4000, "own_stage_bpb_arms_off": 1.2339, "own_stage_bpb_arm_on": 0.2674, "own_stage_difference": -0.9664, "own_stage_difference_se": 0.0074, "web_text_difference": 0.00032, "web_text_difference_se": 0.00015, "stage_items_own_form_accuracy": { "arms_off": 0.344, "arm_on": 0.819 }, "stage_items_new_form_accuracy": { "arms_off": 0.242, "arm_on": 0.258 }, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.411 } } }, "examples": [ "Count by 3s: 1 3s make 3; 2 3s make 6; 3 3s make 9; 4 3s make 12; 5 3s make 15. So 3 times 5 is 15." ] }, { "id": "solo/s5_causal", "name": "solo/s5_causal", "group": null, "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/solo/s5_causal.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "s5_causal_fresh_seed5_d20261005_step4000.safetensors", "sha256": "f99713142a9ea2af225755055e46819cfc19ed9907781aa3f55973a89d9df48a" }, "recipe": "trained alone on the frozen final core: 4000 steps of 2 x 4096 bytes of the stage's own text, pure Adam (lr 0.001), with a quiet term at weight 2 on held-out web text (the KL from the model with the arm off to the model with it on, one chunk beside every task chunk)", "title": "Stage 5: cause and effect (solo)", "behavior": "short cause-and-effect text: what happened and why", "family": "solo", "status": "two seeds; trained alone: mount one solo arm at a time", "score": "own stage text 0.660 -> 0.043 bpb; web text +0.0006 bpb", "measured": { "steps": 4000, "own_stage_bpb_arms_off": 0.6604, "own_stage_bpb_arm_on": 0.0428, "own_stage_difference": -0.6176, "own_stage_difference_se": 0.007, "web_text_difference": 0.0006, "web_text_difference_se": 0.00015, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.415 }, "second_seed": { "steps": 4000, "own_stage_bpb_arms_off": 0.6604, "own_stage_bpb_arm_on": 0.0433, "own_stage_difference": -0.6171, "own_stage_difference_se": 0.007, "web_text_difference": 0.00041, "web_text_difference_se": 0.00015, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.4 } } }, "examples": [ "Noor stacked the cups too high. Because of that, the tower leaned. Because of that, the cups crashed down." ] }, { "id": "solo/s6_tryfail", "name": "solo/s6_tryfail", "group": null, "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/solo/s6_tryfail.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "s6_tryfail_fresh_seed6_d20261005_step4000.safetensors", "sha256": "1744513a39e8f6df4e8741391119d78caf5e2b0a681e8e3da5acabf19a3ac598" }, "recipe": "trained alone on the frozen final core: 4000 steps of 2 x 4096 bytes of the stage's own text, pure Adam (lr 0.001), with a quiet term at weight 2 on held-out web text (the KL from the model with the arm off to the model with it on, one chunk beside every task chunk)", "title": "Stage 6: try, fail, revise (solo)", "behavior": "an attempt, its failure and the revised attempt", "family": "solo", "status": "two seeds; trained alone: mount one solo arm at a time", "score": "own stage text 0.673 -> 0.057 bpb; web text +0.0003 bpb", "measured": { "steps": 4000, "own_stage_bpb_arms_off": 0.6728, "own_stage_bpb_arm_on": 0.0568, "own_stage_difference": -0.6161, "own_stage_difference_se": 0.0035, "web_text_difference": 0.00034, "web_text_difference_se": 0.00022, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.426 }, "second_seed": { "steps": 4000, "own_stage_bpb_arms_off": 0.6728, "own_stage_bpb_arm_on": 0.057, "own_stage_difference": -0.6158, "own_stage_difference_se": 0.0036, "web_text_difference": 0.00061, "web_text_difference_se": 0.0001, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.433 } } }, "examples": [ "Talia adds 34 and 23 and writes 65. Talia checks by counting back: 65 is too big. Talia tries again carefully: 34 + 23 = 57. The check works now, and Talia keeps the good method." ] }, { "id": "solo/s7_mixed", "name": "solo/s7_mixed", "group": null, "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/solo/s7_mixed.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "s7_mixed_fresh_seed7_d20261005_step4000.safetensors", "sha256": "e53aec3a88e82a5f7b69440867c9de348c3f07222398e31042922a7e38cb5263" }, "recipe": "trained alone on the frozen final core: 4000 steps of 2 x 4096 bytes of the stage's own text, pure Adam (lr 0.001), with a quiet term at weight 2 on held-out web text (the KL from the model with the arm off to the model with it on, one chunk beside every task chunk)", "title": "Stage 7: mixed (solo)", "behavior": "the mixed stage's diet: the earlier stages' text beside ordinary prose", "family": "solo", "status": "two seeds; trained alone: mount one solo arm at a time", "score": "own stage text 0.783 -> 0.715 bpb; web text +0.0000 bpb", "measured": { "steps": 4000, "own_stage_bpb_arms_off": 0.783, "own_stage_bpb_arm_on": 0.7153, "own_stage_difference": -0.0677, "own_stage_difference_se": 0.0407, "web_text_difference": 2e-05, "web_text_difference_se": 0.00018, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.426 }, "second_seed": { "steps": 4000, "own_stage_bpb_arms_off": 0.783, "own_stage_bpb_arm_on": 0.7171, "own_stage_difference": -0.0659, "own_stage_difference_se": 0.0392, "web_text_difference": 0.00017, "web_text_difference_se": 0.00017, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.419 } } }, "examples": [ "It has been compared to a chant, a rhythmic divine beauty, a melody, an aria, a toccata, an edification, an exaltation. As poetry is for the tongue, calligraphy is to the page. The authors of The Splendor of Islamic Calligraphy put it best when they said: “Calligraphy is the plainsong of the divine”.\nCalligraphy is the art of the linear graphic, but it is more than that. In Isl" ] }, { "id": "solo/s8_register", "name": "solo/s8_register", "group": null, "base_model_id": "alephllm/mini-beatrix-3@step245674", "step": 245674, "params": 13692960, "sites": 32, "weights": [ "arms/solo/s8_register.safetensors" ], "mount": { "type": "anchor" }, "precision": "fp32", "template": { "frame": "raw", "template": "raw stage text, continued" }, "decode": { "temperature": 0.0, "top_p": 1.0, "max_new": 120, "precision": "fp32" }, "source": { "file": "s8_register_fresh_seed8_d20261005_step4000.safetensors", "sha256": "158f420f7caa8d8dd0f4dfb29206a8ba14e58bf600def9849da5049fec4380a1" }, "recipe": "trained alone on the frozen final core: 4000 steps of 2 x 4096 bytes of the stage's own text, pure Adam (lr 0.001), with a quiet term at weight 2 on held-out web text (the KL from the model with the arm off to the model with it on, one chunk beside every task chunk)", "title": "Stage 8: register (solo)", "behavior": "the same content in different registers of speech", "family": "solo", "status": "two seeds; trained alone: mount one solo arm at a time", "score": "own stage text 0.491 -> 0.021 bpb; web text +0.0003 bpb", "measured": { "steps": 4000, "own_stage_bpb_arms_off": 0.4907, "own_stage_bpb_arm_on": 0.0208, "own_stage_difference": -0.4698, "own_stage_difference_se": 0.0086, "web_text_difference": 0.00026, "web_text_difference_se": 0.00017, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.4 }, "second_seed": { "steps": 4000, "own_stage_bpb_arms_off": 0.4907, "own_stage_bpb_arm_on": 0.0209, "own_stage_difference": -0.4698, "own_stage_difference_se": 0.0085, "web_text_difference": 0.00019, "web_text_difference_se": 0.00018, "probe_suite_mean": { "arms_off": 0.411, "arm_on": 0.404 } } }, "examples": [ "The word 'ancient' means from a very long time ago. Used in a sentence: a ancient is easy to point at once you know the word." ] } ] }