| # Native SHARED looped MoE with Fable response SFT |
|
|
| This single **112,237,056-parameter** model descends from the native looped MoE |
| trained from random initialization for **7,217 updates / 920,770,653 retained |
| targets**, followed by the SHARED MELT arm's **32 updates / 4,087,238 targets**. |
| The architecture and tokenizer are unchanged: six physical blocks, width 384, |
| three loops, 24 routed plus two shared experts, four routed plus both shared |
| experts active per token, and 1,023-token context. |
|
|
| The additional SFT stage completed **3 epochs / 21 |
| updates**, processing **163,953 input-token exposures** |
| and **147,597 supervised target exposures**. |
| It starts a fresh FP32 Adam optimizer from the saved SHARED model. Repeated |
| epochs reuse the same **68 unique training examples in 19 task families**. |
| Validation has 5 examples in 3 families; test has 12 in 3 families. These are small, |
| task-grouped diagnostics, not a broad reasoning or coding benchmark. |
|
|
| Training uses public uploader-labelled Fable **visible explanatory responses** |
| from TheFusionCube/Fable-5-CoT-Traces, with decoys, ambiguous model metadata, |
| truncation and unsuitable tasks excluded. The declared chat conversion is not |
| added as more data. Original answers are preserved verbatim. This is **not |
| authenticated private-CoT distillation** or proof of Fable-level capabilities. |
| Teacher responses supply SFT labels; no online teacher or KL loss is used. |
|
|
| Assistant-target CE: validation 5.669114 before |
| and 5.327462 after SFT; test 5.504149. |
| These scores do not establish general reasoning improvement. Source receipts, |
| actual training summary and checkpoint validation remain in provenance/. |
|
|
| Run from this folder with the dependencies in requirements.txt: |
|
|
| ```bash |
| python run.py --prompt "Explain why regex backreferences are not regular." --max-new-tokens 64 |
| ``` |
|
|
| The CLI wraps the raw question in the exact `### Context` / `### Response` training format. |
| The default is CPU FP32 reference inference. Optional `--device cuda` uses |
| the unchanged BF16 parameter-view engine with FP32 masters, buffers and routing. |
| Formatted prompt plus continuation must fit 1,023 tokens. Use a fresh process. |
|
|
| Only this SHARED model's weights are included. The unchanged source/runtime and |
| tokenizer are bundled with file hashes. Optimizer/checkpoint payloads, original |
| training data and cloud credentials are not needed for inference; the resumable |
| checkpoint remains in the verified local retrieval. Historical paths in passive |
| provenance are not runtime dependencies. Source license: source/LICENSE. |
|
|
| This public distribution adds comment-only change notices to six modified |
| upstream files and an Apache-2.0 notice to the original MELT-memory extension. |
| Runtime integrity constants and distribution receipts are rebound to those |
| files; all verification gates remain active. Model weights, tokenizer bytes |
| and inference algorithms are unchanged. Original training manifests and |
| source-integrity receipts are preserved outside this bundle under release/. |
| Model weights and project-added runtime are distributed under Apache-2.0, |
| with bundled third-party notices retained. |
|
|
